May 25 12:09:50.771178 ip-10-0-134-183 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
May 25 12:09:50.771190 ip-10-0-134-183 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
May 25 12:09:50.771200 ip-10-0-134-183 systemd[1]: kubelet.service: Failed with result 'resources'.
May 25 12:09:50.771681 ip-10-0-134-183 systemd[1]: Failed to start Kubernetes Kubelet.
May 25 12:10:01.002887 ip-10-0-134-183 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
May 25 12:10:01.002913 ip-10-0-134-183 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 7f95aaa047d148c8bf679fac42f71f09 --
May 25 12:12:13.895917 ip-10-0-134-183 systemd[1]: Starting Kubernetes Kubelet...
May 25 12:12:14.395573 ip-10-0-134-183 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 12:12:14.395573 ip-10-0-134-183 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 25 12:12:14.395573 ip-10-0-134-183 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 12:12:14.395573 ip-10-0-134-183 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 25 12:12:14.395573 ip-10-0-134-183 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 12:12:14.397321 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.397240    2563 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 25 12:12:14.402150 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402126    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 12:12:14.402150 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402147    2563 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 12:12:14.402150 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402151    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 12:12:14.402150 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402155    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 12:12:14.402150 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402158    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 12:12:14.402336 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402163    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 12:12:14.402336 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402168    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 12:12:14.402336 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402171    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 12:12:14.402336 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402174    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 12:12:14.402336 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402177    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 12:12:14.402336 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402179    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 12:12:14.402336 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402182    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 12:12:14.402336 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402185    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 12:12:14.402336 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402188    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 12:12:14.402336 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402190    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 12:12:14.402336 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402193    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 12:12:14.402336 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402196    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 12:12:14.402336 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402198    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 12:12:14.402336 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402201    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 12:12:14.402336 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402203    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 12:12:14.402336 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402206    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 12:12:14.402336 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402209    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 12:12:14.402336 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402211    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 12:12:14.402336 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402215    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 12:12:14.402336 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402220    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 12:12:14.402848 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402223    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 12:12:14.402848 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402227    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 12:12:14.402848 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402229    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 12:12:14.402848 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402232    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 12:12:14.402848 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402234    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 12:12:14.402848 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402237    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 12:12:14.402848 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402239    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 12:12:14.402848 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402242    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 12:12:14.402848 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402245    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 12:12:14.402848 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402247    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 12:12:14.402848 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402250    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 12:12:14.402848 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402253    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 12:12:14.402848 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402255    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 12:12:14.402848 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402258    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 12:12:14.402848 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402261    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 12:12:14.402848 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402264    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 12:12:14.402848 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402266    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 12:12:14.402848 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402269    2563 feature_gate.go:328] unrecognized feature gate: Example2
May 25 12:12:14.402848 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402272    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 12:12:14.402848 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402274    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 12:12:14.403332 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402277    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 12:12:14.403332 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402280    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 12:12:14.403332 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402283    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 12:12:14.403332 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402285    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 12:12:14.403332 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402288    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 12:12:14.403332 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402290    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 12:12:14.403332 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402293    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 12:12:14.403332 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402296    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 12:12:14.403332 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402298    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 12:12:14.403332 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402301    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 12:12:14.403332 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402304    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 12:12:14.403332 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402306    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 12:12:14.403332 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402309    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 12:12:14.403332 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402313    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 12:12:14.403332 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402317    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 12:12:14.403332 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402320    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 12:12:14.403332 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402323    2563 feature_gate.go:328] unrecognized feature gate: Example
May 25 12:12:14.403332 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402326    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 12:12:14.403332 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402328    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 12:12:14.403332 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402331    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 12:12:14.403839 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402334    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 12:12:14.403839 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402337    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 12:12:14.403839 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402340    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 12:12:14.403839 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402343    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 12:12:14.403839 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402346    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 12:12:14.403839 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402350    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 12:12:14.403839 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402353    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 12:12:14.403839 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402356    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 12:12:14.403839 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402359    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 12:12:14.403839 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402362    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 12:12:14.403839 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402364    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 12:12:14.403839 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402367    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 12:12:14.403839 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402370    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 12:12:14.403839 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402374    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 12:12:14.403839 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402377    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 12:12:14.403839 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402379    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 12:12:14.403839 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402382    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 12:12:14.403839 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402384    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 12:12:14.403839 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402387    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 12:12:14.403839 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402389    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 12:12:14.404322 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402392    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 12:12:14.404322 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402782    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 12:12:14.404322 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402788    2563 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 12:12:14.404322 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402791    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 12:12:14.404322 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402794    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 12:12:14.404322 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402797    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 12:12:14.404322 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402800    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 12:12:14.404322 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402802    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 12:12:14.404322 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402805    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 12:12:14.404322 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402808    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 12:12:14.404322 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402810    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 12:12:14.404322 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402813    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 12:12:14.404322 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402816    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 12:12:14.404322 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402819    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 12:12:14.404322 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402821    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 12:12:14.404322 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402824    2563 feature_gate.go:328] unrecognized feature gate: Example
May 25 12:12:14.404322 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402826    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 12:12:14.404322 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402829    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 12:12:14.404322 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402832    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 12:12:14.404322 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402835    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 12:12:14.404814 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402838    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 12:12:14.404814 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402841    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 12:12:14.404814 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402843    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 12:12:14.404814 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402846    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 12:12:14.404814 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402849    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 12:12:14.404814 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402852    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 12:12:14.404814 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402855    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 12:12:14.404814 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402858    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 12:12:14.404814 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402862    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 12:12:14.404814 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402865    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 12:12:14.404814 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402868    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 12:12:14.404814 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402871    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 12:12:14.404814 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402873    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 12:12:14.404814 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402876    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 12:12:14.404814 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402879    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 12:12:14.404814 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402881    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 12:12:14.404814 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402884    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 12:12:14.404814 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402887    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 12:12:14.404814 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402889    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 12:12:14.404814 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402891    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 12:12:14.405302 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402894    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 12:12:14.405302 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402897    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 12:12:14.405302 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402900    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 12:12:14.405302 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402902    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 12:12:14.405302 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402905    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 12:12:14.405302 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402907    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 12:12:14.405302 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402910    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 12:12:14.405302 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402912    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 12:12:14.405302 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402915    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 12:12:14.405302 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402917    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 12:12:14.405302 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402920    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 12:12:14.405302 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402924    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 12:12:14.405302 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402927    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 12:12:14.405302 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402930    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 12:12:14.405302 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402933    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 12:12:14.405302 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402936    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 12:12:14.405302 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402938    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 12:12:14.405302 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402941    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 12:12:14.405302 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402944    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 12:12:14.405778 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402947    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 12:12:14.405778 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402950    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 12:12:14.405778 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402952    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 12:12:14.405778 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402955    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 12:12:14.405778 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402957    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 12:12:14.405778 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402960    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 12:12:14.405778 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402963    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 12:12:14.405778 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402966    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 12:12:14.405778 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402969    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 12:12:14.405778 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402971    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 12:12:14.405778 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402974    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 12:12:14.405778 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402976    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 12:12:14.405778 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402979    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 12:12:14.405778 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402982    2563 feature_gate.go:328] unrecognized feature gate: Example2
May 25 12:12:14.405778 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402985    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 12:12:14.405778 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402988    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 12:12:14.405778 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402990    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 12:12:14.405778 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402993    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 12:12:14.405778 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402995    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 12:12:14.406263 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.402998    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 12:12:14.406263 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403000    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 12:12:14.406263 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403003    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 12:12:14.406263 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403005    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 12:12:14.406263 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403008    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 12:12:14.406263 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403011    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 12:12:14.406263 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403013    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 12:12:14.406263 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403015    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 12:12:14.406263 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403018    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 12:12:14.406263 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403092    2563 flags.go:64] FLAG: --address="0.0.0.0"
May 25 12:12:14.406263 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403099    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 25 12:12:14.406263 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403106    2563 flags.go:64] FLAG: --anonymous-auth="true"
May 25 12:12:14.406263 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403111    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 25 12:12:14.406263 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403116    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
May 25 12:12:14.406263 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403119    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 25 12:12:14.406263 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403124    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 25 12:12:14.406263 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403128    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 25 12:12:14.406263 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403131    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 25 12:12:14.406263 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403134    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 25 12:12:14.406263 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403138    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 25 12:12:14.406263 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403141    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 25 12:12:14.406263 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403144    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 25 12:12:14.406802 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403147    2563 flags.go:64] FLAG: --cgroup-root=""
May 25 12:12:14.406802 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403150    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
May 25 12:12:14.406802 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403153    2563 flags.go:64] FLAG: --client-ca-file=""
May 25 12:12:14.406802 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403156    2563 flags.go:64] FLAG: --cloud-config=""
May 25 12:12:14.406802 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403159    2563 flags.go:64] FLAG: --cloud-provider="external"
May 25 12:12:14.406802 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403163    2563 flags.go:64] FLAG: --cluster-dns="[]"
May 25 12:12:14.406802 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403168    2563 flags.go:64] FLAG: --cluster-domain=""
May 25 12:12:14.406802 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403170    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 25 12:12:14.406802 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403174    2563 flags.go:64] FLAG: --config-dir=""
May 25 12:12:14.406802 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403177    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 25 12:12:14.406802 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403180    2563 flags.go:64] FLAG: --container-log-max-files="5"
May 25 12:12:14.406802 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403189    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 25 12:12:14.406802 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403192    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 25 12:12:14.406802 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403195    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 25 12:12:14.406802 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403198    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 25 12:12:14.406802 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403202    2563 flags.go:64] FLAG: --contention-profiling="false"
May 25 12:12:14.406802 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403205    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 25 12:12:14.406802 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403207    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 25 12:12:14.406802 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403211    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
May 25 12:12:14.406802 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403214    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 25 12:12:14.406802 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403218    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 25 12:12:14.406802 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403221    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 25 12:12:14.406802 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403224    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 25 12:12:14.406802 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403227    2563 flags.go:64] FLAG: --enable-load-reader="false"
May 25 12:12:14.406802 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403230    2563 flags.go:64] FLAG: --enable-server="true"
May 25 12:12:14.407427 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403233    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 25 12:12:14.407427 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403237    2563 flags.go:64] FLAG: --event-burst="100"
May 25 12:12:14.407427 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403241    2563 flags.go:64] FLAG: --event-qps="50"
May 25 12:12:14.407427 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403243    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 25 12:12:14.407427 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403247    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 25 12:12:14.407427 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403250    2563 flags.go:64] FLAG: --eviction-hard=""
May 25 12:12:14.407427 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403253    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 25 12:12:14.407427 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403256    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 25 12:12:14.407427 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403260    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 25 12:12:14.407427 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403263    2563 flags.go:64] FLAG: --eviction-soft=""
May 25 12:12:14.407427 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403266    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 25 12:12:14.407427 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403269    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 25 12:12:14.407427 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403274    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 25 12:12:14.407427 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403277    2563 flags.go:64] FLAG: --experimental-mounter-path=""
May 25 12:12:14.407427 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403280    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
May 25 12:12:14.407427 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403284    2563 flags.go:64] FLAG: --fail-swap-on="true"
May 25 12:12:14.407427 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403287    2563 flags.go:64] FLAG: --feature-gates=""
May 25 12:12:14.407427 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403291    2563 flags.go:64] FLAG: --file-check-frequency="20s"
May 25 12:12:14.407427 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403293    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 25 12:12:14.407427 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403297    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 25 12:12:14.407427 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403300    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 25 12:12:14.407427 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403303    2563 flags.go:64] FLAG: --healthz-port="10248"
May 25 12:12:14.407427 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403307    2563 flags.go:64] FLAG: --help="false"
May 25 12:12:14.407427 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403309    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-134-183.ec2.internal"
May 25 12:12:14.407427 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403313    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
May 25 12:12:14.408084 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403316    2563 flags.go:64] FLAG: --http-check-frequency="20s"
May 25 12:12:14.408084 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403319    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 25 12:12:14.408084 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403322    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 25 12:12:14.408084 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403325    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 25 12:12:14.408084 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403328    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 25 12:12:14.408084 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403331    2563 flags.go:64] FLAG: --image-service-endpoint=""
May 25 12:12:14.408084 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403334    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 25 12:12:14.408084 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403338    2563 flags.go:64] FLAG: --kube-api-burst="100"
May 25 12:12:14.408084 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403341    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 25 12:12:14.408084 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403344    2563 flags.go:64] FLAG: --kube-api-qps="50"
May 25 12:12:14.408084 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403347    2563 flags.go:64] FLAG: --kube-reserved=""
May 25 12:12:14.408084 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403350    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 25 12:12:14.408084 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403352    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 25 12:12:14.408084 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403356    2563 flags.go:64] FLAG: --kubelet-cgroups=""
May 25 12:12:14.408084 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403359    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 25 12:12:14.408084 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403362    2563 flags.go:64] FLAG: --lock-file=""
May 25 12:12:14.408084 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403365    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 25 12:12:14.408084 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403368    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
May 25 12:12:14.408084 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403370    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 25 12:12:14.408084 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403386    2563 flags.go:64] FLAG: --log-json-split-stream="false"
May 25 12:12:14.408084 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403389    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 25 12:12:14.408084 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403392    2563 flags.go:64] FLAG: --log-text-split-stream="false"
May 25 12:12:14.408084 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403395    2563 flags.go:64] FLAG: --logging-format="text"
May 25 12:12:14.408642 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403398    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 25 12:12:14.408642 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403401    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 25 12:12:14.408642 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403404    2563 flags.go:64] FLAG: --manifest-url=""
May 25 12:12:14.408642 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403407    2563 flags.go:64] FLAG: --manifest-url-header=""
May 25 12:12:14.408642 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403411    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 25 12:12:14.408642 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403414    2563 flags.go:64] FLAG: --max-open-files="1000000"
May 25 12:12:14.408642 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403418    2563 flags.go:64] FLAG: --max-pods="110"
May 25 12:12:14.408642 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403422    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 25 12:12:14.408642 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403424    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 25 12:12:14.408642 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403427    2563 flags.go:64] FLAG: --memory-manager-policy="None"
May 25 12:12:14.408642 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403430    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 25 12:12:14.408642 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403433    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 25 12:12:14.408642 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403436    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 25 12:12:14.408642 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403439    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 25 12:12:14.408642 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403446    2563 flags.go:64] FLAG: --node-status-max-images="50"
May 25 12:12:14.408642 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403449    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 25 12:12:14.408642 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403452    2563 flags.go:64] FLAG: --oom-score-adj="-999"
May 25 12:12:14.408642 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403456    2563 flags.go:64] FLAG: --pod-cidr=""
May 25 12:12:14.408642 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403459    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 25 12:12:14.408642 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403464    2563 flags.go:64] FLAG: --pod-manifest-path=""
May 25 12:12:14.408642 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403467    2563 flags.go:64] FLAG: --pod-max-pids="-1"
May 25 12:12:14.408642 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403470    2563 flags.go:64] FLAG: --pods-per-core="0"
May 25 12:12:14.408642 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403473    2563 flags.go:64] FLAG: --port="10250"
May 25 12:12:14.408642 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403476    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 25 12:12:14.409246 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403479    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-055e33907190b7856"
May 25 12:12:14.409246 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403482    2563 flags.go:64] FLAG: --qos-reserved=""
May 25 12:12:14.409246 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403485    2563 flags.go:64] FLAG: --read-only-port="10255"
May 25 12:12:14.409246 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403488    2563 flags.go:64] FLAG: --register-node="true"
May 25 12:12:14.409246 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403496    2563 flags.go:64] FLAG: --register-schedulable="true"
May 25 12:12:14.409246 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403499    2563 flags.go:64] FLAG: --register-with-taints=""
May 25 12:12:14.409246 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403502    2563 flags.go:64] FLAG: --registry-burst="10"
May 25 12:12:14.409246 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403505    2563 flags.go:64] FLAG: --registry-qps="5"
May 25 12:12:14.409246 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403508    2563 flags.go:64] FLAG: --reserved-cpus=""
May 25 12:12:14.409246 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403511    2563 flags.go:64] FLAG: --reserved-memory=""
May 25 12:12:14.409246 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403515    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 25 12:12:14.409246 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403518    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 25 12:12:14.409246 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403535    2563 flags.go:64] FLAG: --rotate-certificates="false"
May 25 12:12:14.409246 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403538    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
May 25 12:12:14.409246 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403541    2563 flags.go:64] FLAG: --runonce="false"
May 25 12:12:14.409246 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403544    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 25 12:12:14.409246 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403547    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 25 12:12:14.409246 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403550    2563 flags.go:64] FLAG: --seccomp-default="false"
May 25 12:12:14.409246 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403553    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
May 25 12:12:14.409246 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403556    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 25 12:12:14.409246 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403559    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 25 12:12:14.409246 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403562    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 25 12:12:14.409246 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403565    2563 flags.go:64] FLAG: --storage-driver-password="root"
May 25 12:12:14.409246 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403568    2563 flags.go:64] FLAG: --storage-driver-secure="false"
May 25 12:12:14.409246 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403571    2563 flags.go:64] FLAG: --storage-driver-table="stats"
May 25 12:12:14.409246 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403574    2563 flags.go:64] FLAG: --storage-driver-user="root"
May 25 12:12:14.409889 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403578    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 25 12:12:14.409889 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403581    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
May 25 12:12:14.409889 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403584    2563 flags.go:64] FLAG: --system-cgroups=""
May 25 12:12:14.409889 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403587    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 25 12:12:14.409889 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403592    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
May 25 12:12:14.409889 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403595    2563 flags.go:64] FLAG: --tls-cert-file=""
May 25 12:12:14.409889 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403598    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 25 12:12:14.409889 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403602    2563 flags.go:64] FLAG: --tls-min-version=""
May 25 12:12:14.409889 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403605    2563 flags.go:64] FLAG: --tls-private-key-file=""
May 25 12:12:14.409889 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403608    2563 flags.go:64] FLAG: --topology-manager-policy="none"
May 25 12:12:14.409889 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403613    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
May 25 12:12:14.409889 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403616    2563 flags.go:64] FLAG: --topology-manager-scope="container"
May 25 12:12:14.409889 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403619    2563 flags.go:64] FLAG: --v="2"
May 25 12:12:14.409889 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403623    2563 flags.go:64] FLAG: --version="false"
May 25 12:12:14.409889 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403627    2563 flags.go:64] FLAG: --vmodule=""
May 25 12:12:14.409889 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403632    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 25 12:12:14.409889 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.403635    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 25 12:12:14.409889 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403733    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 12:12:14.409889 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403737    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 12:12:14.409889 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403740    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 12:12:14.409889 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403743    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 12:12:14.409889 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403746    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 12:12:14.409889 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403749    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 12:12:14.410448 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403752    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 12:12:14.410448 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403755    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 12:12:14.410448 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403758    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 12:12:14.410448 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403762    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 12:12:14.410448 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403765    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 12:12:14.410448 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403768    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 12:12:14.410448 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403770    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 12:12:14.410448 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403773    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 12:12:14.410448 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403776    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 12:12:14.410448 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403778    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 12:12:14.410448 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403781    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 12:12:14.410448 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403784    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 12:12:14.410448 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403787    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 12:12:14.410448 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403789    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 12:12:14.410448 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403795    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 12:12:14.410448 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403798    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 12:12:14.410448 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403800    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 12:12:14.410448 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403803    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 12:12:14.410448 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403806    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 12:12:14.410448 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403809    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 12:12:14.410957 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403812    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 12:12:14.410957 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403815    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 12:12:14.410957 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403818    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 12:12:14.410957 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403820    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 12:12:14.410957 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403823    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 12:12:14.410957 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403826    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 12:12:14.410957 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403828    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 12:12:14.410957 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403831    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 12:12:14.410957 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403833    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 12:12:14.410957 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403836    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 12:12:14.410957 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403838    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 12:12:14.410957 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403841    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 12:12:14.410957 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403844    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 12:12:14.410957 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403847    2563 feature_gate.go:328] unrecognized feature gate: Example2
May 25 12:12:14.410957 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403849    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 12:12:14.410957 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403852    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 12:12:14.410957 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403855    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 12:12:14.410957 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403857    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 12:12:14.410957 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403860    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 12:12:14.410957 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403862    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 12:12:14.411452 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403865    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 12:12:14.411452 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403867    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 12:12:14.411452 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403870    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 12:12:14.411452 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403873    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 12:12:14.411452 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403875    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 12:12:14.411452 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403878    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 12:12:14.411452 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403882    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 12:12:14.411452 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403885    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 12:12:14.411452 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403887    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 12:12:14.411452 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403890    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 12:12:14.411452 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403893    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 12:12:14.411452 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403897    2563 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 12:12:14.411452 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403899    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 12:12:14.411452 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403902    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 12:12:14.411452 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403904    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 12:12:14.411452 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403907    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 12:12:14.411452 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403910    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 12:12:14.411452 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403912    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 12:12:14.411452 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403915    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 12:12:14.411452 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403917    2563 feature_gate.go:328] unrecognized feature gate: Example
May 25 12:12:14.411995 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403920    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 12:12:14.411995 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403923    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 12:12:14.411995 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403925    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 12:12:14.411995 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403927    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 12:12:14.411995 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403930    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 12:12:14.411995 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403933    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 12:12:14.411995 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403935    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 12:12:14.411995 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403938    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 12:12:14.411995 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403941    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 12:12:14.411995 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403945    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 12:12:14.411995 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403948    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 12:12:14.411995 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403951    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 12:12:14.411995 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403953    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 12:12:14.411995 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403957    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 12:12:14.411995 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403960    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 12:12:14.411995 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403963    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 12:12:14.411995 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403967    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 12:12:14.411995 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403969    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 12:12:14.411995 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403973    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 12:12:14.412510 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.403976    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 12:12:14.412510 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.404805    2563 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 12:12:14.412510 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.411108    2563 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 25 12:12:14.412510 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.411123    2563 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 25 12:12:14.412510 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411172    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 12:12:14.412510 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411177    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 12:12:14.412510 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411181    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 12:12:14.412510 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411184    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 12:12:14.412510 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411187    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 12:12:14.412510 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411190    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 12:12:14.412510 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411193    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 12:12:14.412510 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411196    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 12:12:14.412510 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411199    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 12:12:14.412510 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411202    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 12:12:14.412510 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411205    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 12:12:14.412510 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411208    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 12:12:14.412975 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411211    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 12:12:14.412975 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411214    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 12:12:14.412975 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411216    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 12:12:14.412975 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411219    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 12:12:14.412975 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411223    2563 feature_gate.go:328] unrecognized feature gate: Example
May 25 12:12:14.412975 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411225    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 12:12:14.412975 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411228    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 12:12:14.412975 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411230    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 12:12:14.412975 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411233    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 12:12:14.412975 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411236    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 12:12:14.412975 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411238    2563 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 12:12:14.412975 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411241    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 12:12:14.412975 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411244    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 12:12:14.412975 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411248    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 12:12:14.412975 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411250    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 12:12:14.412975 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411253    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 12:12:14.412975 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411256    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 12:12:14.412975 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411258    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 12:12:14.412975 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411261    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 12:12:14.413453 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411264    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 12:12:14.413453 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411267    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 12:12:14.413453 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411269    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 12:12:14.413453 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411272    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 12:12:14.413453 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411275    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 12:12:14.413453 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411277    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 12:12:14.413453 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411280    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 12:12:14.413453 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411282    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 12:12:14.413453 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411285    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 12:12:14.413453 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411287    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 12:12:14.413453 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411290    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 12:12:14.413453 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411293    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 12:12:14.413453 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411295    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 12:12:14.413453 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411298    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 12:12:14.413453 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411300    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 12:12:14.413453 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411303    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 12:12:14.413453 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411305    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 12:12:14.413453 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411308    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 12:12:14.413453 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411311    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 12:12:14.413453 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411313    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 12:12:14.413964 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411316    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 12:12:14.413964 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411318    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 12:12:14.413964 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411321    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 12:12:14.413964 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411323    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 12:12:14.413964 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411326    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 12:12:14.413964 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411329    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 12:12:14.413964 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411331    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 12:12:14.413964 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411334    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 12:12:14.413964 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411337    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 12:12:14.413964 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411339    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 12:12:14.413964 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411342    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 12:12:14.413964 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411344    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 12:12:14.413964 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411347    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 12:12:14.413964 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411350    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 12:12:14.413964 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411352    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 12:12:14.413964 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411355    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 12:12:14.413964 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411358    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 12:12:14.413964 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411361    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 12:12:14.413964 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411363    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 12:12:14.413964 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411366    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 12:12:14.414449 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411370    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 12:12:14.414449 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411374    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 12:12:14.414449 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411376    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 12:12:14.414449 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411379    2563 feature_gate.go:328] unrecognized feature gate: Example2
May 25 12:12:14.414449 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411382    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 12:12:14.414449 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411384    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 12:12:14.414449 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411386    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 12:12:14.414449 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411389    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 12:12:14.414449 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411392    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 12:12:14.414449 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411394    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 12:12:14.414449 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411396    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 12:12:14.414449 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411399    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 12:12:14.414449 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411403    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 12:12:14.414449 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411407    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 12:12:14.414449 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411410    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 12:12:14.414838 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.411422    2563 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 12:12:14.414838 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411549    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 12:12:14.414838 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411555    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 12:12:14.414838 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411558    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 12:12:14.414838 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411562    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 12:12:14.414838 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411567    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 12:12:14.414838 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411570    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 12:12:14.414838 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411573    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 12:12:14.414838 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411576    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 12:12:14.414838 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411579    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 12:12:14.414838 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411582    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 12:12:14.414838 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411584    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 12:12:14.414838 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411587    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 12:12:14.414838 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411590    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 12:12:14.414838 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411592    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 12:12:14.415222 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411595    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 12:12:14.415222 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411597    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 12:12:14.415222 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411600    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 12:12:14.415222 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411603    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 12:12:14.415222 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411605    2563 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 12:12:14.415222 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411608    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 12:12:14.415222 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411610    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 12:12:14.415222 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411613    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 12:12:14.415222 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411616    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 12:12:14.415222 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411618    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 12:12:14.415222 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411621    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 12:12:14.415222 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411623    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 12:12:14.415222 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411625    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 12:12:14.415222 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411628    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 12:12:14.415222 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411630    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 12:12:14.415222 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411633    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 12:12:14.415222 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411635    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 12:12:14.415222 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411638    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 12:12:14.415222 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411641    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 12:12:14.415222 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411644    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 12:12:14.415743 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411647    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 12:12:14.415743 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411649    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 12:12:14.415743 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411652    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 12:12:14.415743 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411655    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 12:12:14.415743 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411658    2563 feature_gate.go:328] unrecognized feature gate: Example
May 25 12:12:14.415743 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411661    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 12:12:14.415743 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411663    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 12:12:14.415743 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411666    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 12:12:14.415743 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411668    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 12:12:14.415743 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411671    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 12:12:14.415743 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411673    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 12:12:14.415743 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411676    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 12:12:14.415743 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411678    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 12:12:14.415743 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411680    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 12:12:14.415743 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411683    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 12:12:14.415743 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411685    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 12:12:14.415743 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411688    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 12:12:14.415743 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411690    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 12:12:14.415743 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411693    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 12:12:14.415743 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411695    2563 feature_gate.go:328] unrecognized feature gate: Example2
May 25 12:12:14.416278 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411698    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 12:12:14.416278 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411702    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 12:12:14.416278 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411706    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 12:12:14.416278 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411709    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 12:12:14.416278 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411712    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 12:12:14.416278 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411715    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 12:12:14.416278 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411717    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 12:12:14.416278 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411720    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 12:12:14.416278 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411722    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 12:12:14.416278 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411725    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 12:12:14.416278 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411727    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 12:12:14.416278 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411730    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 12:12:14.416278 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411732    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 12:12:14.416278 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411735    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 12:12:14.416278 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411738    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 12:12:14.416278 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411740    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 12:12:14.416278 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411743    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 12:12:14.416278 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411746    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 12:12:14.416278 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411749    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 12:12:14.416753 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411751    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 12:12:14.416753 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411754    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 12:12:14.416753 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411756    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 12:12:14.416753 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411759    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 12:12:14.416753 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411761    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 12:12:14.416753 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411764    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 12:12:14.416753 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411766    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 12:12:14.416753 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411769    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 12:12:14.416753 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411772    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 12:12:14.416753 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411775    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 12:12:14.416753 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411777    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 12:12:14.416753 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411780    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 12:12:14.416753 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:14.411782    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 12:12:14.416753 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.411787    2563 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 12:12:14.416753 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.412438    2563 server.go:962] "Client rotation is on, will bootstrap in background"
May 25 12:12:14.417124 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.414932    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 25 12:12:14.417124 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.415886    2563 server.go:1019] "Starting client certificate rotation"
May 25 12:12:14.417124 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.415985    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 25 12:12:14.417124 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.416865    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 25 12:12:14.443230 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.443214    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 25 12:12:14.445055 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.445035    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 25 12:12:14.461931 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.461913    2563 log.go:25] "Validated CRI v1 runtime API"
May 25 12:12:14.467120 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.467104    2563 log.go:25] "Validated CRI v1 image API"
May 25 12:12:14.468322 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.468306    2563 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 25 12:12:14.472076 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.472055    2563 fs.go:135] Filesystem UUIDs: map[26a7771d-a118-44aa-b683-dfd19b83b8e1:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d9337150-9ca8-4bce-aded-2313b010bc43:/dev/nvme0n1p4]
May 25 12:12:14.472142 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.472074    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 25 12:12:14.474978 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.474962    2563 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 25 12:12:14.476453 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.476340    2563 manager.go:217] Machine: {Timestamp:2026-05-25 12:12:14.475835153 +0000 UTC m=+0.443226785 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099987 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec273231c734fce389b915ade5b0f5ea SystemUUID:ec273231-c734-fce3-89b9-15ade5b0f5ea BootID:7f95aaa0-47d1-48c8-bf67-9fac42f71f09 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a0:09:c6:c8:27 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a0:09:c6:c8:27 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9a:8d:8b:aa:98:04 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 25 12:12:14.477053 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.477039    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 25 12:12:14.477144 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.477131    2563 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 25 12:12:14.480277 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.480253    2563 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 25 12:12:14.480443 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.480279    2563 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-183.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 25 12:12:14.480490 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.480450    2563 topology_manager.go:138] "Creating topology manager with none policy"
May 25 12:12:14.480490 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.480459    2563 container_manager_linux.go:306] "Creating device plugin manager"
May 25 12:12:14.480490 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.480467    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 25 12:12:14.481407 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.481396    2563 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 25 12:12:14.482848 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.482838    2563 state_mem.go:36] "Initialized new in-memory state store"
May 25 12:12:14.482953 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.482945    2563 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 25 12:12:14.485678 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.485667    2563 kubelet.go:491] "Attempting to sync node with API server"
May 25 12:12:14.485727 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.485687    2563 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 25 12:12:14.485727 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.485699    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 25 12:12:14.485727 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.485707    2563 kubelet.go:397] "Adding apiserver pod source"
May 25 12:12:14.485727 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.485717    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 25 12:12:14.486899 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.486887    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 25 12:12:14.486942 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.486906    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 25 12:12:14.490025 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.490009    2563 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 25 12:12:14.491904 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.491890    2563 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 25 12:12:14.492733 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.492721    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 25 12:12:14.492775 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.492738    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 25 12:12:14.492775 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.492745    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 25 12:12:14.492775 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.492750    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 25 12:12:14.492775 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.492761    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 25 12:12:14.492775 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.492768    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 25 12:12:14.492775 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.492777    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 25 12:12:14.492951 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.492786    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 25 12:12:14.492951 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.492795    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 25 12:12:14.492951 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.492801    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 25 12:12:14.492951 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.492815    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 25 12:12:14.493172 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.493158    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 25 12:12:14.493207 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.493181    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w6l9h"
May 25 12:12:14.494075 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.494062    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 25 12:12:14.494110 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.494076    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 25 12:12:14.497285 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.497271    2563 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 25 12:12:14.497348 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.497306    2563 server.go:1295] "Started kubelet"
May 25 12:12:14.497404 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.497374    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 25 12:12:14.497475 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.497428    2563 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 25 12:12:14.497541 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.497490    2563 server_v1.go:47] "podresources" method="list" useActivePods=true
May 25 12:12:14.498128 ip-10-0-134-183 systemd[1]: Started Kubernetes Kubelet.
May 25 12:12:14.498820 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.498799    2563 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 25 12:12:14.499736 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.499720    2563 server.go:317] "Adding debug handlers to kubelet server"
May 25 12:12:14.501139 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.501086    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w6l9h"
May 25 12:12:14.503493 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:14.503462    2563 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-183.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 25 12:12:14.503593 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:14.503463    2563 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 25 12:12:14.503642 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.503596    2563 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-183.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 25 12:12:14.506641 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.506623    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 25 12:12:14.506641 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.506635    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 25 12:12:14.506773 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.506682    2563 volume_manager.go:295] "The desired_state_of_world populator starts"
May 25 12:12:14.506773 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.506695    2563 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 25 12:12:14.506866 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.506811    2563 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 25 12:12:14.506866 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.506845    2563 reconstruct.go:97] "Volume reconstruction finished"
May 25 12:12:14.506866 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.506850    2563 reconciler.go:26] "Reconciler: start to sync state"
May 25 12:12:14.507639 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:14.507619    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-183.ec2.internal\" not found"
May 25 12:12:14.508138 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.508121    2563 factory.go:153] Registering CRI-O factory
May 25 12:12:14.508206 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.508176    2563 factory.go:223] Registration of the crio container factory successfully
May 25 12:12:14.508255 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.508227    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 25 12:12:14.508255 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.508237    2563 factory.go:55] Registering systemd factory
May 25 12:12:14.508255 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.508244    2563 factory.go:223] Registration of the systemd container factory successfully
May 25 12:12:14.508391 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.508261    2563 factory.go:103] Registering Raw factory
May 25 12:12:14.508391 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.508280    2563 manager.go:1196] Started watching for new ooms in manager
May 25 12:12:14.508750 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.508733    2563 manager.go:319] Starting recovery of all containers
May 25 12:12:14.509946 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:14.509924    2563 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 25 12:12:14.510537 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.510495    2563 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 12:12:14.513692 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:14.513667    2563 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-183.ec2.internal\" not found" node="ip-10-0-134-183.ec2.internal"
May 25 12:12:14.517370 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.517236    2563 manager.go:324] Recovery completed
May 25 12:12:14.521751 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.521736    2563 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 12:12:14.524753 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.524733    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-183.ec2.internal" event="NodeHasSufficientMemory"
May 25 12:12:14.524843 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.524780    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-183.ec2.internal" event="NodeHasNoDiskPressure"
May 25 12:12:14.524843 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.524796    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-183.ec2.internal" event="NodeHasSufficientPID"
May 25 12:12:14.525497 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.525481    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 25 12:12:14.525497 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.525496    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 25 12:12:14.525646 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.525531    2563 state_mem.go:36] "Initialized new in-memory state store"
May 25 12:12:14.528044 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.528031    2563 policy_none.go:49] "None policy: Start"
May 25 12:12:14.528085 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.528048    2563 memory_manager.go:186] "Starting memorymanager" policy="None"
May 25 12:12:14.528085 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.528062    2563 state_mem.go:35] "Initializing new in-memory state store"
May 25 12:12:14.567116 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.567101    2563 manager.go:341] "Starting Device Plugin manager"
May 25 12:12:14.575021 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:14.567132    2563 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 25 12:12:14.575021 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.567146    2563 server.go:85] "Starting device plugin registration server"
May 25 12:12:14.575021 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.567383    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
May 25 12:12:14.575021 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.567395    2563 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 25 12:12:14.575021 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.567489    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 25 12:12:14.575021 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.567592    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 25 12:12:14.575021 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.567601    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 25 12:12:14.575021 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:14.568014    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 25 12:12:14.575021 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:14.568063    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-183.ec2.internal\" not found"
May 25 12:12:14.631186 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.631157    2563 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 25 12:12:14.632425 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.632409    2563 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 25 12:12:14.632472 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.632437    2563 status_manager.go:230] "Starting to sync pod status with apiserver"
May 25 12:12:14.632472 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.632460    2563 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 25 12:12:14.632472 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.632470    2563 kubelet.go:2451] "Starting kubelet main sync loop"
May 25 12:12:14.632594 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:14.632516    2563 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 25 12:12:14.635889 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.635868    2563 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 12:12:14.667880 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.667826    2563 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 12:12:14.668760 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.668744    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-183.ec2.internal" event="NodeHasSufficientMemory"
May 25 12:12:14.668831 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.668776    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-183.ec2.internal" event="NodeHasNoDiskPressure"
May 25 12:12:14.668831 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.668789    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-183.ec2.internal" event="NodeHasSufficientPID"
May 25 12:12:14.668831 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.668811    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-183.ec2.internal"
May 25 12:12:14.677806 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.677788    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-183.ec2.internal"
May 25 12:12:14.677862 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:14.677809    2563 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-183.ec2.internal\": node \"ip-10-0-134-183.ec2.internal\" not found"
May 25 12:12:14.696737 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:14.696717    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-183.ec2.internal\" not found"
May 25 12:12:14.733855 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.732724    2563 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-183.ec2.internal"]
May 25 12:12:14.733932 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.733897    2563 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 12:12:14.734741 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.734723    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-183.ec2.internal" event="NodeHasSufficientMemory"
May 25 12:12:14.734828 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.734755    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-183.ec2.internal" event="NodeHasNoDiskPressure"
May 25 12:12:14.734828 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.734765    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-183.ec2.internal" event="NodeHasSufficientPID"
May 25 12:12:14.736202 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.736191    2563 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 12:12:14.736362 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.736347    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.ec2.internal"
May 25 12:12:14.736414 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.736379    2563 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 12:12:14.736839 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.736818    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-183.ec2.internal" event="NodeHasSufficientMemory"
May 25 12:12:14.736938 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.736843    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-183.ec2.internal" event="NodeHasNoDiskPressure"
May 25 12:12:14.736938 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.736849    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-183.ec2.internal" event="NodeHasSufficientMemory"
May 25 12:12:14.736938 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.736857    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-183.ec2.internal" event="NodeHasSufficientPID"
May 25 12:12:14.736938 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.736869    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-183.ec2.internal" event="NodeHasNoDiskPressure"
May 25 12:12:14.736938 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.736879    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-183.ec2.internal" event="NodeHasSufficientPID"
May 25 12:12:14.737962 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.737946    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-183.ec2.internal"
May 25 12:12:14.738034 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.737977    2563 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 12:12:14.738611 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.738596    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-183.ec2.internal" event="NodeHasSufficientMemory"
May 25 12:12:14.738675 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.738628    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-183.ec2.internal" event="NodeHasNoDiskPressure"
May 25 12:12:14.738675 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.738644    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-183.ec2.internal" event="NodeHasSufficientPID"
May 25 12:12:14.762128 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:14.762109    2563 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-183.ec2.internal\" not found" node="ip-10-0-134-183.ec2.internal"
May 25 12:12:14.766467 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:14.766453    2563 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-183.ec2.internal\" not found" node="ip-10-0-134-183.ec2.internal"
May 25 12:12:14.797289 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:14.797270    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-183.ec2.internal\" not found"
May 25 12:12:14.898254 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:14.898225    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-183.ec2.internal\" not found"
May 25 12:12:14.908608 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.908586    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fbb7f0c11b7bf6cb015cb8917e9cb796-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-183.ec2.internal\" (UID: \"fbb7f0c11b7bf6cb015cb8917e9cb796\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.ec2.internal"
May 25 12:12:14.908660 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.908614    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbb7f0c11b7bf6cb015cb8917e9cb796-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-183.ec2.internal\" (UID: \"fbb7f0c11b7bf6cb015cb8917e9cb796\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.ec2.internal"
May 25 12:12:14.908660 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:14.908634    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/44c84599553bc2d6ec69b56732f924ff-config\") pod \"kube-apiserver-proxy-ip-10-0-134-183.ec2.internal\" (UID: \"44c84599553bc2d6ec69b56732f924ff\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-183.ec2.internal"
May 25 12:12:14.999307 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:14.999239    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-183.ec2.internal\" not found"
May 25 12:12:15.009631 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.009603    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fbb7f0c11b7bf6cb015cb8917e9cb796-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-183.ec2.internal\" (UID: \"fbb7f0c11b7bf6cb015cb8917e9cb796\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.ec2.internal"
May 25 12:12:15.009680 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.009635    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbb7f0c11b7bf6cb015cb8917e9cb796-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-183.ec2.internal\" (UID: \"fbb7f0c11b7bf6cb015cb8917e9cb796\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.ec2.internal"
May 25 12:12:15.009680 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.009653    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/44c84599553bc2d6ec69b56732f924ff-config\") pod \"kube-apiserver-proxy-ip-10-0-134-183.ec2.internal\" (UID: \"44c84599553bc2d6ec69b56732f924ff\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-183.ec2.internal"
May 25 12:12:15.009744 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.009697    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fbb7f0c11b7bf6cb015cb8917e9cb796-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-183.ec2.internal\" (UID: \"fbb7f0c11b7bf6cb015cb8917e9cb796\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.ec2.internal"
May 25 12:12:15.009744 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.009696    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbb7f0c11b7bf6cb015cb8917e9cb796-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-183.ec2.internal\" (UID: \"fbb7f0c11b7bf6cb015cb8917e9cb796\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.ec2.internal"
May 25 12:12:15.009803 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.009769    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/44c84599553bc2d6ec69b56732f924ff-config\") pod \"kube-apiserver-proxy-ip-10-0-134-183.ec2.internal\" (UID: \"44c84599553bc2d6ec69b56732f924ff\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-183.ec2.internal"
May 25 12:12:15.065063 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.065020    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.ec2.internal"
May 25 12:12:15.068787 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.068766    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-183.ec2.internal"
May 25 12:12:15.099487 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:15.099463    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-183.ec2.internal\" not found"
May 25 12:12:15.200294 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:15.200265    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-183.ec2.internal\" not found"
May 25 12:12:15.300886 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:15.300810    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-183.ec2.internal\" not found"
May 25 12:12:15.401406 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:15.401369    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-183.ec2.internal\" not found"
May 25 12:12:15.408593 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.408575    2563 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 12:12:15.415814 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.415795    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 25 12:12:15.415936 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.415917    2563 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 25 12:12:15.415976 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.415955    2563 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 25 12:12:15.416018 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.415957    2563 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 25 12:12:15.486107 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.486084    2563 apiserver.go:52] "Watching apiserver"
May 25 12:12:15.502352 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.502328    2563 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 25 12:12:15.503763 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.503312    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-6jsxz","kube-system/konnectivity-agent-vq5g9","openshift-cluster-node-tuning-operator/tuned-5zd7t","openshift-dns/node-resolver-5mdjk","openshift-multus/multus-additional-cni-plugins-96cz9","openshift-network-diagnostics/network-check-target-slzvp","openshift-network-operator/iptables-alerter-k65b6","openshift-ovn-kubernetes/ovnkube-node-cw9bb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-579t6","openshift-image-registry/node-ca-sbtvl","openshift-multus/multus-qnq6q"]
May 25 12:12:15.504656 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.504175    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-24 12:07:14 +0000 UTC" deadline="2027-12-23 13:10:42.30224472 +0000 UTC"
May 25 12:12:15.504656 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.504205    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13848h58m26.798042391s"
May 25 12:12:15.506430 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.506407    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jsxz"
May 25 12:12:15.506546 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:15.506509    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jsxz" podUID="80098ceb-2e1b-4378-9371-6c6150ed88b6"
May 25 12:12:15.506702 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.506689    2563 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.ec2.internal"
May 25 12:12:15.506746 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.506709    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 25 12:12:15.508989 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.508847    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.510237 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.510214    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vq5g9"
May 25 12:12:15.510352 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.510316    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5mdjk"
May 25 12:12:15.510684 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.510663    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 25 12:12:15.510804 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.510769    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 25 12:12:15.510804 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.510798    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wrknx\""
May 25 12:12:15.511638 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.511619    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-96cz9"
May 25 12:12:15.511756 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.511727    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzsvq\" (UniqueName: \"kubernetes.io/projected/80098ceb-2e1b-4378-9371-6c6150ed88b6-kube-api-access-xzsvq\") pod \"network-metrics-daemon-6jsxz\" (UID: \"80098ceb-2e1b-4378-9371-6c6150ed88b6\") " pod="openshift-multus/network-metrics-daemon-6jsxz"
May 25 12:12:15.511829 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.511774    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/edeeb54c-d929-487b-9fd3-d09c06073cdd-etc-sysctl-conf\") pod \"tuned-5zd7t\" (UID: \"edeeb54c-d929-487b-9fd3-d09c06073cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.511829 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.511802    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/edeeb54c-d929-487b-9fd3-d09c06073cdd-etc-systemd\") pod \"tuned-5zd7t\" (UID: \"edeeb54c-d929-487b-9fd3-d09c06073cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.511829 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.511825    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/edeeb54c-d929-487b-9fd3-d09c06073cdd-sys\") pod \"tuned-5zd7t\" (UID: \"edeeb54c-d929-487b-9fd3-d09c06073cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.512011 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.511853    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/45b823f7-17fe-46bc-9c01-c126b9d66b9e-agent-certs\") pod \"konnectivity-agent-vq5g9\" (UID: \"45b823f7-17fe-46bc-9c01-c126b9d66b9e\") " pod="kube-system/konnectivity-agent-vq5g9"
May 25 12:12:15.512011 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.511875    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/45b823f7-17fe-46bc-9c01-c126b9d66b9e-konnectivity-ca\") pod \"konnectivity-agent-vq5g9\" (UID: \"45b823f7-17fe-46bc-9c01-c126b9d66b9e\") " pod="kube-system/konnectivity-agent-vq5g9"
May 25 12:12:15.512011 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.511887    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4xn6j\""
May 25 12:12:15.512011 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.511898    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/edeeb54c-d929-487b-9fd3-d09c06073cdd-etc-kubernetes\") pod \"tuned-5zd7t\" (UID: \"edeeb54c-d929-487b-9fd3-d09c06073cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.512011 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.511921    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/edeeb54c-d929-487b-9fd3-d09c06073cdd-etc-tuned\") pod \"tuned-5zd7t\" (UID: \"edeeb54c-d929-487b-9fd3-d09c06073cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.512011 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.511944    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/edeeb54c-d929-487b-9fd3-d09c06073cdd-tmp\") pod \"tuned-5zd7t\" (UID: \"edeeb54c-d929-487b-9fd3-d09c06073cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.512011 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.511965    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80098ceb-2e1b-4378-9371-6c6150ed88b6-metrics-certs\") pod \"network-metrics-daemon-6jsxz\" (UID: \"80098ceb-2e1b-4378-9371-6c6150ed88b6\") " pod="openshift-multus/network-metrics-daemon-6jsxz"
May 25 12:12:15.512011 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.511987    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/edeeb54c-d929-487b-9fd3-d09c06073cdd-etc-sysconfig\") pod \"tuned-5zd7t\" (UID: \"edeeb54c-d929-487b-9fd3-d09c06073cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.512011 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.512008    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/edeeb54c-d929-487b-9fd3-d09c06073cdd-etc-sysctl-d\") pod \"tuned-5zd7t\" (UID: \"edeeb54c-d929-487b-9fd3-d09c06073cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.512473 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.512031    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/edeeb54c-d929-487b-9fd3-d09c06073cdd-run\") pod \"tuned-5zd7t\" (UID: \"edeeb54c-d929-487b-9fd3-d09c06073cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.512473 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.512054    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/edeeb54c-d929-487b-9fd3-d09c06073cdd-var-lib-kubelet\") pod \"tuned-5zd7t\" (UID: \"edeeb54c-d929-487b-9fd3-d09c06073cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.512473 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.512076    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvdk9\" (UniqueName: \"kubernetes.io/projected/edeeb54c-d929-487b-9fd3-d09c06073cdd-kube-api-access-tvdk9\") pod \"tuned-5zd7t\" (UID: \"edeeb54c-d929-487b-9fd3-d09c06073cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.512473 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.512103    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/edeeb54c-d929-487b-9fd3-d09c06073cdd-etc-modprobe-d\") pod \"tuned-5zd7t\" (UID: \"edeeb54c-d929-487b-9fd3-d09c06073cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.512473 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.512149    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/edeeb54c-d929-487b-9fd3-d09c06073cdd-lib-modules\") pod \"tuned-5zd7t\" (UID: \"edeeb54c-d929-487b-9fd3-d09c06073cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.512473 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.512227    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/edeeb54c-d929-487b-9fd3-d09c06073cdd-host\") pod \"tuned-5zd7t\" (UID: \"edeeb54c-d929-487b-9fd3-d09c06073cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.512473 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.512286    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-z5jf5\""
May 25 12:12:15.512473 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.512366    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 25 12:12:15.512473 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.512460    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 25 12:12:15.512878 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.512603    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 25 12:12:15.512878 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.512618    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 25 12:12:15.512971 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.512946    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slzvp"
May 25 12:12:15.513034 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:15.513008    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slzvp" podUID="e54faaa8-8c11-4a93-b337-c914cc73a410"
May 25 12:12:15.513307 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.513290    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 25 12:12:15.513429 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.513414    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-n4j85\""
May 25 12:12:15.513674 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.513660    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 25 12:12:15.513879 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.513866    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 25 12:12:15.514051 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.514033    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 25 12:12:15.514132 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.514088    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 25 12:12:15.514132 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.514042    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k65b6"
May 25 12:12:15.515354 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.515339    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.515803 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.515781    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 25 12:12:15.515971 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.515952    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 25 12:12:15.516192 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.516162    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rgz9x\""
May 25 12:12:15.516345 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.516329    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 25 12:12:15.516668 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.516653    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-579t6"
May 25 12:12:15.517720 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.517706    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 25 12:12:15.517800 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.517728    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 25 12:12:15.517867 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.517851    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sbtvl"
May 25 12:12:15.518207 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.518192    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 25 12:12:15.518295 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.518268    2563 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-183.ec2.internal"
May 25 12:12:15.518601 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.518588    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 25 12:12:15.518692 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.518678    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 25 12:12:15.518751 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.518701    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-q2bfp\""
May 25 12:12:15.519197 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.519174    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.ec2.internal"]
May 25 12:12:15.519296 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.519279    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.519384 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.519285    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 25 12:12:15.521864 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.520090    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 25 12:12:15.522638 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.522203    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 25 12:12:15.522638 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.522257    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 25 12:12:15.522638 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.522314    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 25 12:12:15.522638 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.522262    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-sx822\""
May 25 12:12:15.522638 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.522477    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 25 12:12:15.522638 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.522546    2563 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 25 12:12:15.522638 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.522562    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 25 12:12:15.522638 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.522550    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 25 12:12:15.522638 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.522598    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-h59kd\""
May 25 12:12:15.523004 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.522972    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 25 12:12:15.523232 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.523216    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-n9rz2\""
May 25 12:12:15.525765 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.525740    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-183.ec2.internal"]
May 25 12:12:15.525765 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.525759    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 25 12:12:15.541118 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.541101    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-65s5v"
May 25 12:12:15.550179 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.550161    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-65s5v"
May 25 12:12:15.607831 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.607809    2563 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 25 12:12:15.611787 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:15.611745    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44c84599553bc2d6ec69b56732f924ff.slice/crio-b7423f3b298db95f410feac1b12414e7e8143e5be4477d6efcbec193d260b808 WatchSource:0}: Error finding container b7423f3b298db95f410feac1b12414e7e8143e5be4477d6efcbec193d260b808: Status 404 returned error can't find the container with id b7423f3b298db95f410feac1b12414e7e8143e5be4477d6efcbec193d260b808
May 25 12:12:15.612008 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:15.611984    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbb7f0c11b7bf6cb015cb8917e9cb796.slice/crio-5ec97f3056daed4d463c467eb304a968b98a31b2ba918754cc84dc4c53dac275 WatchSource:0}: Error finding container 5ec97f3056daed4d463c467eb304a968b98a31b2ba918754cc84dc4c53dac275: Status 404 returned error can't find the container with id 5ec97f3056daed4d463c467eb304a968b98a31b2ba918754cc84dc4c53dac275
May 25 12:12:15.613240 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.613132    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/edeeb54c-d929-487b-9fd3-d09c06073cdd-run\") pod \"tuned-5zd7t\" (UID: \"edeeb54c-d929-487b-9fd3-d09c06073cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.613240 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.613158    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/edeeb54c-d929-487b-9fd3-d09c06073cdd-var-lib-kubelet\") pod \"tuned-5zd7t\" (UID: \"edeeb54c-d929-487b-9fd3-d09c06073cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.613240 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.613186    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9f26ab5b-c341-495e-bac3-01d4475a795a-cnibin\") pod \"multus-additional-cni-plugins-96cz9\" (UID: \"9f26ab5b-c341-495e-bac3-01d4475a795a\") " pod="openshift-multus/multus-additional-cni-plugins-96cz9"
May 25 12:12:15.613366 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.613239    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/edeeb54c-d929-487b-9fd3-d09c06073cdd-run\") pod \"tuned-5zd7t\" (UID: \"edeeb54c-d929-487b-9fd3-d09c06073cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.613366 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.613248    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/edeeb54c-d929-487b-9fd3-d09c06073cdd-var-lib-kubelet\") pod \"tuned-5zd7t\" (UID: \"edeeb54c-d929-487b-9fd3-d09c06073cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.613366 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.613274    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-run-systemd\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.613366 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.613300    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/012c465e-2683-488a-abc1-18ad86529eb3-ovnkube-config\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.613366 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.613323    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4d8f666d-fd9f-4ff8-90c8-b6bfeb3be623-serviceca\") pod \"node-ca-sbtvl\" (UID: \"4d8f666d-fd9f-4ff8-90c8-b6bfeb3be623\") " pod="openshift-image-registry/node-ca-sbtvl"
May 25 12:12:15.613366 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.613351    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/edeeb54c-d929-487b-9fd3-d09c06073cdd-etc-modprobe-d\") pod \"tuned-5zd7t\" (UID: \"edeeb54c-d929-487b-9fd3-d09c06073cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.613664 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.613402    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/edeeb54c-d929-487b-9fd3-d09c06073cdd-lib-modules\") pod \"tuned-5zd7t\" (UID: \"edeeb54c-d929-487b-9fd3-d09c06073cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.613664 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.613437    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/edeeb54c-d929-487b-9fd3-d09c06073cdd-host\") pod \"tuned-5zd7t\" (UID: \"edeeb54c-d929-487b-9fd3-d09c06073cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.613664 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.613445    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/edeeb54c-d929-487b-9fd3-d09c06073cdd-etc-modprobe-d\") pod \"tuned-5zd7t\" (UID: \"edeeb54c-d929-487b-9fd3-d09c06073cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.613664 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.613483    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7cd43613-c252-4e65-9c14-f9bfa930be35-cni-binary-copy\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.613664 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.613509    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7cd43613-c252-4e65-9c14-f9bfa930be35-hostroot\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.613664 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.613553    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/edeeb54c-d929-487b-9fd3-d09c06073cdd-lib-modules\") pod \"tuned-5zd7t\" (UID: \"edeeb54c-d929-487b-9fd3-d09c06073cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.613664 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.613551    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9f26ab5b-c341-495e-bac3-01d4475a795a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-96cz9\" (UID: \"9f26ab5b-c341-495e-bac3-01d4475a795a\") " pod="openshift-multus/multus-additional-cni-plugins-96cz9"
May 25 12:12:15.613664 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.613606    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/edeeb54c-d929-487b-9fd3-d09c06073cdd-host\") pod \"tuned-5zd7t\" (UID: \"edeeb54c-d929-487b-9fd3-d09c06073cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.613664 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.613643    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-host-run-netns\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.614045 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.613673    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xzsvq\" (UniqueName: \"kubernetes.io/projected/80098ceb-2e1b-4378-9371-6c6150ed88b6-kube-api-access-xzsvq\") pod \"network-metrics-daemon-6jsxz\" (UID: \"80098ceb-2e1b-4378-9371-6c6150ed88b6\") " pod="openshift-multus/network-metrics-daemon-6jsxz"
May 25 12:12:15.614045 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.613700    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7cd43613-c252-4e65-9c14-f9bfa930be35-host-var-lib-cni-bin\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.614045 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.613744    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9f26ab5b-c341-495e-bac3-01d4475a795a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-96cz9\" (UID: \"9f26ab5b-c341-495e-bac3-01d4475a795a\") " pod="openshift-multus/multus-additional-cni-plugins-96cz9"
May 25 12:12:15.614045 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.613772    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/873ca31f-e11b-4104-a3ae-243fdc6b54c0-iptables-alerter-script\") pod \"iptables-alerter-k65b6\" (UID: \"873ca31f-e11b-4104-a3ae-243fdc6b54c0\") " pod="openshift-network-operator/iptables-alerter-k65b6"
May 25 12:12:15.614045 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.613811    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-systemd-units\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.614045 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.613850    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-run-openvswitch\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.614045 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.613907    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/45b823f7-17fe-46bc-9c01-c126b9d66b9e-agent-certs\") pod \"konnectivity-agent-vq5g9\" (UID: \"45b823f7-17fe-46bc-9c01-c126b9d66b9e\") " pod="kube-system/konnectivity-agent-vq5g9"
May 25 12:12:15.614045 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.613973    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/edeeb54c-d929-487b-9fd3-d09c06073cdd-etc-kubernetes\") pod \"tuned-5zd7t\" (UID: \"edeeb54c-d929-487b-9fd3-d09c06073cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.614460 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.614261    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 25 12:12:15.614460 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.614320    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7cd43613-c252-4e65-9c14-f9bfa930be35-host-var-lib-cni-multus\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.614460 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.614354    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7cd43613-c252-4e65-9c14-f9bfa930be35-multus-daemon-config\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.614460 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.614388    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7cd43613-c252-4e65-9c14-f9bfa930be35-host-run-multus-certs\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.614460 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.614409    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-host-slash\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.614460 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.614431    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/012c465e-2683-488a-abc1-18ad86529eb3-env-overrides\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.614460 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.614456    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/edeeb54c-d929-487b-9fd3-d09c06073cdd-etc-sysctl-d\") pod \"tuned-5zd7t\" (UID: \"edeeb54c-d929-487b-9fd3-d09c06073cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.614800 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.614557    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/edeeb54c-d929-487b-9fd3-d09c06073cdd-etc-kubernetes\") pod \"tuned-5zd7t\" (UID: \"edeeb54c-d929-487b-9fd3-d09c06073cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.614800 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.614564    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tvdk9\" (UniqueName: \"kubernetes.io/projected/edeeb54c-d929-487b-9fd3-d09c06073cdd-kube-api-access-tvdk9\") pod \"tuned-5zd7t\" (UID: \"edeeb54c-d929-487b-9fd3-d09c06073cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.614800 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.614641    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sskn\" (UniqueName: \"kubernetes.io/projected/9f26ab5b-c341-495e-bac3-01d4475a795a-kube-api-access-8sskn\") pod \"multus-additional-cni-plugins-96cz9\" (UID: \"9f26ab5b-c341-495e-bac3-01d4475a795a\") " pod="openshift-multus/multus-additional-cni-plugins-96cz9"
May 25 12:12:15.614800 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.614649    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/edeeb54c-d929-487b-9fd3-d09c06073cdd-etc-sysctl-d\") pod \"tuned-5zd7t\" (UID: \"edeeb54c-d929-487b-9fd3-d09c06073cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.614800 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.614671    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/873ca31f-e11b-4104-a3ae-243fdc6b54c0-host-slash\") pod \"iptables-alerter-k65b6\" (UID: \"873ca31f-e11b-4104-a3ae-243fdc6b54c0\") " pod="openshift-network-operator/iptables-alerter-k65b6"
May 25 12:12:15.614800 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.614716    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-log-socket\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.614800 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.614794    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-host-cni-netd\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.615131 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.614822    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7cd43613-c252-4e65-9c14-f9bfa930be35-multus-conf-dir\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.615131 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.614863    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqvxf\" (UniqueName: \"kubernetes.io/projected/27b08c8c-8256-4b27-86d6-6eb05ad8bfff-kube-api-access-tqvxf\") pod \"node-resolver-5mdjk\" (UID: \"27b08c8c-8256-4b27-86d6-6eb05ad8bfff\") " pod="openshift-dns/node-resolver-5mdjk"
May 25 12:12:15.615131 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.614889    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-node-log\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.615131 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.614925    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-host-run-ovn-kubernetes\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.615131 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.614958    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/012c465e-2683-488a-abc1-18ad86529eb3-ovn-node-metrics-cert\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.615131 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.614989    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8eff746e-a59a-4d74-8822-66d3ee1bb7df-sys-fs\") pod \"aws-ebs-csi-driver-node-579t6\" (UID: \"8eff746e-a59a-4d74-8822-66d3ee1bb7df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-579t6"
May 25 12:12:15.615131 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.615031    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7cd43613-c252-4e65-9c14-f9bfa930be35-system-cni-dir\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.615131 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.615055    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7cd43613-c252-4e65-9c14-f9bfa930be35-multus-cni-dir\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.615131 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.615093    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7cd43613-c252-4e65-9c14-f9bfa930be35-cnibin\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.615131 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.615118    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7cd43613-c252-4e65-9c14-f9bfa930be35-multus-socket-dir-parent\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.615657 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.615141    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f26ab5b-c341-495e-bac3-01d4475a795a-system-cni-dir\") pod \"multus-additional-cni-plugins-96cz9\" (UID: \"9f26ab5b-c341-495e-bac3-01d4475a795a\") " pod="openshift-multus/multus-additional-cni-plugins-96cz9"
May 25 12:12:15.615657 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.615181    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-var-lib-openvswitch\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.615657 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.615205    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-etc-openvswitch\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.615657 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.615250    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/edeeb54c-d929-487b-9fd3-d09c06073cdd-etc-tuned\") pod \"tuned-5zd7t\" (UID: \"edeeb54c-d929-487b-9fd3-d09c06073cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.615657 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.615277    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8eff746e-a59a-4d74-8822-66d3ee1bb7df-etc-selinux\") pod \"aws-ebs-csi-driver-node-579t6\" (UID: \"8eff746e-a59a-4d74-8822-66d3ee1bb7df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-579t6"
May 25 12:12:15.615657 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.615303    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7cd43613-c252-4e65-9c14-f9bfa930be35-host-var-lib-kubelet\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.615657 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.615343    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-host-kubelet\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.615657 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.615374    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gksk\" (UniqueName: \"kubernetes.io/projected/012c465e-2683-488a-abc1-18ad86529eb3-kube-api-access-7gksk\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.615657 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.615436    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4d8f666d-fd9f-4ff8-90c8-b6bfeb3be623-host\") pod \"node-ca-sbtvl\" (UID: \"4d8f666d-fd9f-4ff8-90c8-b6bfeb3be623\") " pod="openshift-image-registry/node-ca-sbtvl"
May 25 12:12:15.615657 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.615461    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-run-ovn\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.615657 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.615537    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zplrw\" (UniqueName: \"kubernetes.io/projected/4d8f666d-fd9f-4ff8-90c8-b6bfeb3be623-kube-api-access-zplrw\") pod \"node-ca-sbtvl\" (UID: \"4d8f666d-fd9f-4ff8-90c8-b6bfeb3be623\") " pod="openshift-image-registry/node-ca-sbtvl"
May 25 12:12:15.615657 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.615573    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8eff746e-a59a-4d74-8822-66d3ee1bb7df-socket-dir\") pod \"aws-ebs-csi-driver-node-579t6\" (UID: \"8eff746e-a59a-4d74-8822-66d3ee1bb7df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-579t6"
May 25 12:12:15.615657 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.615612    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8eff746e-a59a-4d74-8822-66d3ee1bb7df-device-dir\") pod \"aws-ebs-csi-driver-node-579t6\" (UID: \"8eff746e-a59a-4d74-8822-66d3ee1bb7df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-579t6"
May 25 12:12:15.616194 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.615656    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wphbc\" (UniqueName: \"kubernetes.io/projected/8eff746e-a59a-4d74-8822-66d3ee1bb7df-kube-api-access-wphbc\") pod \"aws-ebs-csi-driver-node-579t6\" (UID: \"8eff746e-a59a-4d74-8822-66d3ee1bb7df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-579t6"
May 25 12:12:15.616194 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.615702    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/edeeb54c-d929-487b-9fd3-d09c06073cdd-sys\") pod \"tuned-5zd7t\" (UID: \"edeeb54c-d929-487b-9fd3-d09c06073cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.616194 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.615743    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/edeeb54c-d929-487b-9fd3-d09c06073cdd-sys\") pod \"tuned-5zd7t\" (UID: \"edeeb54c-d929-487b-9fd3-d09c06073cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.616194 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.615779    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/45b823f7-17fe-46bc-9c01-c126b9d66b9e-konnectivity-ca\") pod \"konnectivity-agent-vq5g9\" (UID: \"45b823f7-17fe-46bc-9c01-c126b9d66b9e\") " pod="kube-system/konnectivity-agent-vq5g9"
May 25 12:12:15.616194 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.615802    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/edeeb54c-d929-487b-9fd3-d09c06073cdd-tmp\") pod \"tuned-5zd7t\" (UID: \"edeeb54c-d929-487b-9fd3-d09c06073cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.616194 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.615826    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8eff746e-a59a-4d74-8822-66d3ee1bb7df-registration-dir\") pod \"aws-ebs-csi-driver-node-579t6\" (UID: \"8eff746e-a59a-4d74-8822-66d3ee1bb7df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-579t6"
May 25 12:12:15.616194 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.615866    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7cd43613-c252-4e65-9c14-f9bfa930be35-etc-kubernetes\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.616194 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.615890    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrn2t\" (UniqueName: \"kubernetes.io/projected/e54faaa8-8c11-4a93-b337-c914cc73a410-kube-api-access-hrn2t\") pod \"network-check-target-slzvp\" (UID: \"e54faaa8-8c11-4a93-b337-c914cc73a410\") " pod="openshift-network-diagnostics/network-check-target-slzvp"
May 25 12:12:15.616194 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.615914    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80098ceb-2e1b-4378-9371-6c6150ed88b6-metrics-certs\") pod \"network-metrics-daemon-6jsxz\" (UID: \"80098ceb-2e1b-4378-9371-6c6150ed88b6\") " pod="openshift-multus/network-metrics-daemon-6jsxz"
May 25 12:12:15.616194 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.615939    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8eff746e-a59a-4d74-8822-66d3ee1bb7df-kubelet-dir\") pod \"aws-ebs-csi-driver-node-579t6\" (UID: \"8eff746e-a59a-4d74-8822-66d3ee1bb7df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-579t6"
May 25 12:12:15.616194 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.615964    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7cd43613-c252-4e65-9c14-f9bfa930be35-host-run-k8s-cni-cncf-io\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.616194 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.615986    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br8w7\" (UniqueName: \"kubernetes.io/projected/7cd43613-c252-4e65-9c14-f9bfa930be35-kube-api-access-br8w7\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.616194 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.616019    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9f26ab5b-c341-495e-bac3-01d4475a795a-os-release\") pod \"multus-additional-cni-plugins-96cz9\" (UID: \"9f26ab5b-c341-495e-bac3-01d4475a795a\") " pod="openshift-multus/multus-additional-cni-plugins-96cz9"
May 25 12:12:15.616829 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.616231    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/012c465e-2683-488a-abc1-18ad86529eb3-ovnkube-script-lib\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.616829 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.616266    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/27b08c8c-8256-4b27-86d6-6eb05ad8bfff-hosts-file\") pod \"node-resolver-5mdjk\" (UID: \"27b08c8c-8256-4b27-86d6-6eb05ad8bfff\") " pod="openshift-dns/node-resolver-5mdjk"
May 25 12:12:15.616829 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.616293    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9f26ab5b-c341-495e-bac3-01d4475a795a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-96cz9\" (UID: \"9f26ab5b-c341-495e-bac3-01d4475a795a\") " pod="openshift-multus/multus-additional-cni-plugins-96cz9"
May 25 12:12:15.616829 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.616322    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/edeeb54c-d929-487b-9fd3-d09c06073cdd-etc-sysctl-conf\") pod \"tuned-5zd7t\" (UID: \"edeeb54c-d929-487b-9fd3-d09c06073cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.616829 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.616348    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/edeeb54c-d929-487b-9fd3-d09c06073cdd-etc-systemd\") pod \"tuned-5zd7t\" (UID: \"edeeb54c-d929-487b-9fd3-d09c06073cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.616829 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.616374    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7cd43613-c252-4e65-9c14-f9bfa930be35-os-release\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.616829 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.616397    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7cd43613-c252-4e65-9c14-f9bfa930be35-host-run-netns\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.616829 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.616477    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27b08c8c-8256-4b27-86d6-6eb05ad8bfff-tmp-dir\") pod \"node-resolver-5mdjk\" (UID: \"27b08c8c-8256-4b27-86d6-6eb05ad8bfff\") " pod="openshift-dns/node-resolver-5mdjk"
May 25 12:12:15.617160 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:15.617023    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:12:15.617160 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.617105    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/45b823f7-17fe-46bc-9c01-c126b9d66b9e-konnectivity-ca\") pod \"konnectivity-agent-vq5g9\" (UID: \"45b823f7-17fe-46bc-9c01-c126b9d66b9e\") " pod="kube-system/konnectivity-agent-vq5g9"
May 25 12:12:15.617618 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:15.617329    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80098ceb-2e1b-4378-9371-6c6150ed88b6-metrics-certs podName:80098ceb-2e1b-4378-9371-6c6150ed88b6 nodeName:}" failed. No retries permitted until 2026-05-25 12:12:16.11710555 +0000 UTC m=+2.084497181 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80098ceb-2e1b-4378-9371-6c6150ed88b6-metrics-certs") pod "network-metrics-daemon-6jsxz" (UID: "80098ceb-2e1b-4378-9371-6c6150ed88b6") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:12:15.617618 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.616581    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/edeeb54c-d929-487b-9fd3-d09c06073cdd-etc-sysctl-conf\") pod \"tuned-5zd7t\" (UID: \"edeeb54c-d929-487b-9fd3-d09c06073cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.621580 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.616530    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9f26ab5b-c341-495e-bac3-01d4475a795a-cni-binary-copy\") pod \"multus-additional-cni-plugins-96cz9\" (UID: \"9f26ab5b-c341-495e-bac3-01d4475a795a\") " pod="openshift-multus/multus-additional-cni-plugins-96cz9"
May 25 12:12:15.621580 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.618301    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-host-cni-bin\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.621580 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.618339    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs5hn\" (UniqueName: \"kubernetes.io/projected/873ca31f-e11b-4104-a3ae-243fdc6b54c0-kube-api-access-hs5hn\") pod \"iptables-alerter-k65b6\" (UID: \"873ca31f-e11b-4104-a3ae-243fdc6b54c0\") " pod="openshift-network-operator/iptables-alerter-k65b6"
May 25 12:12:15.621580 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.618374    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.621580 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.618416    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/edeeb54c-d929-487b-9fd3-d09c06073cdd-etc-sysconfig\") pod \"tuned-5zd7t\" (UID: \"edeeb54c-d929-487b-9fd3-d09c06073cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.621580 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.618671    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/edeeb54c-d929-487b-9fd3-d09c06073cdd-etc-sysconfig\") pod \"tuned-5zd7t\" (UID: \"edeeb54c-d929-487b-9fd3-d09c06073cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.621580 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.618700    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/edeeb54c-d929-487b-9fd3-d09c06073cdd-etc-systemd\") pod \"tuned-5zd7t\" (UID: \"edeeb54c-d929-487b-9fd3-d09c06073cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.621580 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.618959    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/edeeb54c-d929-487b-9fd3-d09c06073cdd-etc-tuned\") pod \"tuned-5zd7t\" (UID: \"edeeb54c-d929-487b-9fd3-d09c06073cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.621580 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.619158    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/edeeb54c-d929-487b-9fd3-d09c06073cdd-tmp\") pod \"tuned-5zd7t\" (UID: \"edeeb54c-d929-487b-9fd3-d09c06073cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.621580 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.619255    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/45b823f7-17fe-46bc-9c01-c126b9d66b9e-agent-certs\") pod \"konnectivity-agent-vq5g9\" (UID: \"45b823f7-17fe-46bc-9c01-c126b9d66b9e\") " pod="kube-system/konnectivity-agent-vq5g9"
May 25 12:12:15.621580 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.620685    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 12:12:15.622866 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.622842    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvdk9\" (UniqueName: \"kubernetes.io/projected/edeeb54c-d929-487b-9fd3-d09c06073cdd-kube-api-access-tvdk9\") pod \"tuned-5zd7t\" (UID: \"edeeb54c-d929-487b-9fd3-d09c06073cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.623090 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.623023    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzsvq\" (UniqueName: \"kubernetes.io/projected/80098ceb-2e1b-4378-9371-6c6150ed88b6-kube-api-access-xzsvq\") pod \"network-metrics-daemon-6jsxz\" (UID: \"80098ceb-2e1b-4378-9371-6c6150ed88b6\") " pod="openshift-multus/network-metrics-daemon-6jsxz"
May 25 12:12:15.635209 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.635166    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-183.ec2.internal" event={"ID":"44c84599553bc2d6ec69b56732f924ff","Type":"ContainerStarted","Data":"b7423f3b298db95f410feac1b12414e7e8143e5be4477d6efcbec193d260b808"}
May 25 12:12:15.636069 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.636051    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.ec2.internal" event={"ID":"fbb7f0c11b7bf6cb015cb8917e9cb796","Type":"ContainerStarted","Data":"5ec97f3056daed4d463c467eb304a968b98a31b2ba918754cc84dc4c53dac275"}
May 25 12:12:15.718739 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.718710    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7cd43613-c252-4e65-9c14-f9bfa930be35-os-release\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.718894 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.718751    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7cd43613-c252-4e65-9c14-f9bfa930be35-host-run-netns\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.718894 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.718753    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7cd43613-c252-4e65-9c14-f9bfa930be35-os-release\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.718894 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.718771    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27b08c8c-8256-4b27-86d6-6eb05ad8bfff-tmp-dir\") pod \"node-resolver-5mdjk\" (UID: \"27b08c8c-8256-4b27-86d6-6eb05ad8bfff\") " pod="openshift-dns/node-resolver-5mdjk"
May 25 12:12:15.718894 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.718786    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9f26ab5b-c341-495e-bac3-01d4475a795a-cni-binary-copy\") pod \"multus-additional-cni-plugins-96cz9\" (UID: \"9f26ab5b-c341-495e-bac3-01d4475a795a\") " pod="openshift-multus/multus-additional-cni-plugins-96cz9"
May 25 12:12:15.718894 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.718791    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7cd43613-c252-4e65-9c14-f9bfa930be35-host-run-netns\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.718894 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.718801    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-host-cni-bin\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.718894 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.718824    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hs5hn\" (UniqueName: \"kubernetes.io/projected/873ca31f-e11b-4104-a3ae-243fdc6b54c0-kube-api-access-hs5hn\") pod \"iptables-alerter-k65b6\" (UID: \"873ca31f-e11b-4104-a3ae-243fdc6b54c0\") " pod="openshift-network-operator/iptables-alerter-k65b6"
May 25 12:12:15.718894 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.718868    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-host-cni-bin\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.719193 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.718905    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.719193 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.718935    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9f26ab5b-c341-495e-bac3-01d4475a795a-cnibin\") pod \"multus-additional-cni-plugins-96cz9\" (UID: \"9f26ab5b-c341-495e-bac3-01d4475a795a\") " pod="openshift-multus/multus-additional-cni-plugins-96cz9"
May 25 12:12:15.719193 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.718958    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-run-systemd\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.719193 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.718982    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/012c465e-2683-488a-abc1-18ad86529eb3-ovnkube-config\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.719193 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719022    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9f26ab5b-c341-495e-bac3-01d4475a795a-cnibin\") pod \"multus-additional-cni-plugins-96cz9\" (UID: \"9f26ab5b-c341-495e-bac3-01d4475a795a\") " pod="openshift-multus/multus-additional-cni-plugins-96cz9"
May 25 12:12:15.719193 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719043    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.719193 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719078    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27b08c8c-8256-4b27-86d6-6eb05ad8bfff-tmp-dir\") pod \"node-resolver-5mdjk\" (UID: \"27b08c8c-8256-4b27-86d6-6eb05ad8bfff\") " pod="openshift-dns/node-resolver-5mdjk"
May 25 12:12:15.719193 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719085    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-run-systemd\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.719193 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719155    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4d8f666d-fd9f-4ff8-90c8-b6bfeb3be623-serviceca\") pod \"node-ca-sbtvl\" (UID: \"4d8f666d-fd9f-4ff8-90c8-b6bfeb3be623\") " pod="openshift-image-registry/node-ca-sbtvl"
May 25 12:12:15.719193 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719187    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7cd43613-c252-4e65-9c14-f9bfa930be35-cni-binary-copy\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.719669 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719284    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9f26ab5b-c341-495e-bac3-01d4475a795a-cni-binary-copy\") pod \"multus-additional-cni-plugins-96cz9\" (UID: \"9f26ab5b-c341-495e-bac3-01d4475a795a\") " pod="openshift-multus/multus-additional-cni-plugins-96cz9"
May 25 12:12:15.719669 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719285    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7cd43613-c252-4e65-9c14-f9bfa930be35-hostroot\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.719669 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719313    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7cd43613-c252-4e65-9c14-f9bfa930be35-hostroot\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.719669 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719326    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9f26ab5b-c341-495e-bac3-01d4475a795a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-96cz9\" (UID: \"9f26ab5b-c341-495e-bac3-01d4475a795a\") " pod="openshift-multus/multus-additional-cni-plugins-96cz9"
May 25 12:12:15.719669 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719350    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-host-run-netns\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.719669 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719370    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7cd43613-c252-4e65-9c14-f9bfa930be35-host-var-lib-cni-bin\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.719669 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719386    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9f26ab5b-c341-495e-bac3-01d4475a795a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-96cz9\" (UID: \"9f26ab5b-c341-495e-bac3-01d4475a795a\") " pod="openshift-multus/multus-additional-cni-plugins-96cz9"
May 25 12:12:15.719669 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719403    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/873ca31f-e11b-4104-a3ae-243fdc6b54c0-iptables-alerter-script\") pod \"iptables-alerter-k65b6\" (UID: \"873ca31f-e11b-4104-a3ae-243fdc6b54c0\") " pod="openshift-network-operator/iptables-alerter-k65b6"
May 25 12:12:15.719669 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719415    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-host-run-netns\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.719669 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719429    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7cd43613-c252-4e65-9c14-f9bfa930be35-host-var-lib-cni-bin\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.719669 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719536    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-systemd-units\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.719669 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719569    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-run-openvswitch\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.719669 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719596    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/012c465e-2683-488a-abc1-18ad86529eb3-ovnkube-config\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.719669 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719606    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-systemd-units\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.719669 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719597    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7cd43613-c252-4e65-9c14-f9bfa930be35-host-var-lib-cni-multus\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.719669 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719627    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7cd43613-c252-4e65-9c14-f9bfa930be35-host-var-lib-cni-multus\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.719669 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719639    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4d8f666d-fd9f-4ff8-90c8-b6bfeb3be623-serviceca\") pod \"node-ca-sbtvl\" (UID: \"4d8f666d-fd9f-4ff8-90c8-b6bfeb3be623\") " pod="openshift-image-registry/node-ca-sbtvl"
May 25 12:12:15.720377 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719653    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7cd43613-c252-4e65-9c14-f9bfa930be35-multus-daemon-config\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.720377 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719657    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-run-openvswitch\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.720377 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719695    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7cd43613-c252-4e65-9c14-f9bfa930be35-host-run-multus-certs\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.720377 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719730    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-host-slash\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.720377 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719739    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7cd43613-c252-4e65-9c14-f9bfa930be35-host-run-multus-certs\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.720377 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719742    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7cd43613-c252-4e65-9c14-f9bfa930be35-cni-binary-copy\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.720377 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719756    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/012c465e-2683-488a-abc1-18ad86529eb3-env-overrides\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.720377 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719777    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-host-slash\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.720377 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719792    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8sskn\" (UniqueName: \"kubernetes.io/projected/9f26ab5b-c341-495e-bac3-01d4475a795a-kube-api-access-8sskn\") pod \"multus-additional-cni-plugins-96cz9\" (UID: \"9f26ab5b-c341-495e-bac3-01d4475a795a\") " pod="openshift-multus/multus-additional-cni-plugins-96cz9"
May 25 12:12:15.720377 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719820    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/873ca31f-e11b-4104-a3ae-243fdc6b54c0-host-slash\") pod \"iptables-alerter-k65b6\" (UID: \"873ca31f-e11b-4104-a3ae-243fdc6b54c0\") " pod="openshift-network-operator/iptables-alerter-k65b6"
May 25 12:12:15.720377 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719836    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9f26ab5b-c341-495e-bac3-01d4475a795a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-96cz9\" (UID: \"9f26ab5b-c341-495e-bac3-01d4475a795a\") " pod="openshift-multus/multus-additional-cni-plugins-96cz9"
May 25 12:12:15.720377 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719844    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-log-socket\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.720377 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719868    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-host-cni-netd\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.720377 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719887    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/873ca31f-e11b-4104-a3ae-243fdc6b54c0-host-slash\") pod \"iptables-alerter-k65b6\" (UID: \"873ca31f-e11b-4104-a3ae-243fdc6b54c0\") " pod="openshift-network-operator/iptables-alerter-k65b6"
May 25 12:12:15.720377 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719891    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7cd43613-c252-4e65-9c14-f9bfa930be35-multus-conf-dir\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.720377 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719925    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tqvxf\" (UniqueName: \"kubernetes.io/projected/27b08c8c-8256-4b27-86d6-6eb05ad8bfff-kube-api-access-tqvxf\") pod \"node-resolver-5mdjk\" (UID: \"27b08c8c-8256-4b27-86d6-6eb05ad8bfff\") " pod="openshift-dns/node-resolver-5mdjk"
May 25 12:12:15.720377 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719943    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9f26ab5b-c341-495e-bac3-01d4475a795a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-96cz9\" (UID: \"9f26ab5b-c341-495e-bac3-01d4475a795a\") " pod="openshift-multus/multus-additional-cni-plugins-96cz9"
May 25 12:12:15.721037 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719958    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-log-socket\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.721037 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719951    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-node-log\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.721037 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719992    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-host-cni-netd\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.721037 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719978    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-node-log\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.721037 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.719928    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7cd43613-c252-4e65-9c14-f9bfa930be35-multus-conf-dir\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.721037 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720011    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-host-run-ovn-kubernetes\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.721037 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720027    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/873ca31f-e11b-4104-a3ae-243fdc6b54c0-iptables-alerter-script\") pod \"iptables-alerter-k65b6\" (UID: \"873ca31f-e11b-4104-a3ae-243fdc6b54c0\") " pod="openshift-network-operator/iptables-alerter-k65b6"
May 25 12:12:15.721037 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720040    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/012c465e-2683-488a-abc1-18ad86529eb3-ovn-node-metrics-cert\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.721037 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720065    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8eff746e-a59a-4d74-8822-66d3ee1bb7df-sys-fs\") pod \"aws-ebs-csi-driver-node-579t6\" (UID: \"8eff746e-a59a-4d74-8822-66d3ee1bb7df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-579t6"
May 25 12:12:15.721037 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720077    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-host-run-ovn-kubernetes\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.721037 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720126    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8eff746e-a59a-4d74-8822-66d3ee1bb7df-sys-fs\") pod \"aws-ebs-csi-driver-node-579t6\" (UID: \"8eff746e-a59a-4d74-8822-66d3ee1bb7df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-579t6"
May 25 12:12:15.721037 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720129    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/012c465e-2683-488a-abc1-18ad86529eb3-env-overrides\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.721037 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720168    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7cd43613-c252-4e65-9c14-f9bfa930be35-system-cni-dir\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.721037 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720195    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7cd43613-c252-4e65-9c14-f9bfa930be35-multus-cni-dir\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.721037 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720200    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7cd43613-c252-4e65-9c14-f9bfa930be35-multus-daemon-config\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.721037 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720218    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7cd43613-c252-4e65-9c14-f9bfa930be35-cnibin\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.721037 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720232    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7cd43613-c252-4e65-9c14-f9bfa930be35-system-cni-dir\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.721037 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720243    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7cd43613-c252-4e65-9c14-f9bfa930be35-multus-socket-dir-parent\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.721568 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720259    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7cd43613-c252-4e65-9c14-f9bfa930be35-multus-cni-dir\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.721568 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720267    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7cd43613-c252-4e65-9c14-f9bfa930be35-cnibin\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.721568 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720291    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f26ab5b-c341-495e-bac3-01d4475a795a-system-cni-dir\") pod \"multus-additional-cni-plugins-96cz9\" (UID: \"9f26ab5b-c341-495e-bac3-01d4475a795a\") " pod="openshift-multus/multus-additional-cni-plugins-96cz9"
May 25 12:12:15.721568 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720320    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-var-lib-openvswitch\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.721568 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720353    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f26ab5b-c341-495e-bac3-01d4475a795a-system-cni-dir\") pod \"multus-additional-cni-plugins-96cz9\" (UID: \"9f26ab5b-c341-495e-bac3-01d4475a795a\") " pod="openshift-multus/multus-additional-cni-plugins-96cz9"
May 25 12:12:15.721568 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720371    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-etc-openvswitch\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.721568 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720382    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7cd43613-c252-4e65-9c14-f9bfa930be35-multus-socket-dir-parent\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.721568 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720391    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-var-lib-openvswitch\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.721568 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720398    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8eff746e-a59a-4d74-8822-66d3ee1bb7df-etc-selinux\") pod \"aws-ebs-csi-driver-node-579t6\" (UID: \"8eff746e-a59a-4d74-8822-66d3ee1bb7df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-579t6"
May 25 12:12:15.721568 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720424    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7cd43613-c252-4e65-9c14-f9bfa930be35-host-var-lib-kubelet\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.721568 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720449    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-host-kubelet\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.721568 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720429    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-etc-openvswitch\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.721568 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720476    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7gksk\" (UniqueName: \"kubernetes.io/projected/012c465e-2683-488a-abc1-18ad86529eb3-kube-api-access-7gksk\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.721568 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720495    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-host-kubelet\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.721568 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720499    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4d8f666d-fd9f-4ff8-90c8-b6bfeb3be623-host\") pod \"node-ca-sbtvl\" (UID: \"4d8f666d-fd9f-4ff8-90c8-b6bfeb3be623\") " pod="openshift-image-registry/node-ca-sbtvl"
May 25 12:12:15.721568 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720479    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7cd43613-c252-4e65-9c14-f9bfa930be35-host-var-lib-kubelet\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.721568 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720543    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-run-ovn\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.721568 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720571    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4d8f666d-fd9f-4ff8-90c8-b6bfeb3be623-host\") pod \"node-ca-sbtvl\" (UID: \"4d8f666d-fd9f-4ff8-90c8-b6bfeb3be623\") " pod="openshift-image-registry/node-ca-sbtvl"
May 25 12:12:15.722077 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720579    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8eff746e-a59a-4d74-8822-66d3ee1bb7df-etc-selinux\") pod \"aws-ebs-csi-driver-node-579t6\" (UID: \"8eff746e-a59a-4d74-8822-66d3ee1bb7df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-579t6"
May 25 12:12:15.722077 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720573    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zplrw\" (UniqueName: \"kubernetes.io/projected/4d8f666d-fd9f-4ff8-90c8-b6bfeb3be623-kube-api-access-zplrw\") pod \"node-ca-sbtvl\" (UID: \"4d8f666d-fd9f-4ff8-90c8-b6bfeb3be623\") " pod="openshift-image-registry/node-ca-sbtvl"
May 25 12:12:15.722077 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720613    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/012c465e-2683-488a-abc1-18ad86529eb3-run-ovn\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.722077 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720640    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8eff746e-a59a-4d74-8822-66d3ee1bb7df-socket-dir\") pod \"aws-ebs-csi-driver-node-579t6\" (UID: \"8eff746e-a59a-4d74-8822-66d3ee1bb7df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-579t6"
May 25 12:12:15.722077 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720667    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8eff746e-a59a-4d74-8822-66d3ee1bb7df-device-dir\") pod \"aws-ebs-csi-driver-node-579t6\" (UID: \"8eff746e-a59a-4d74-8822-66d3ee1bb7df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-579t6"
May 25 12:12:15.722077 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720691    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wphbc\" (UniqueName: \"kubernetes.io/projected/8eff746e-a59a-4d74-8822-66d3ee1bb7df-kube-api-access-wphbc\") pod \"aws-ebs-csi-driver-node-579t6\" (UID: \"8eff746e-a59a-4d74-8822-66d3ee1bb7df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-579t6"
May 25 12:12:15.722077 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720718    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8eff746e-a59a-4d74-8822-66d3ee1bb7df-registration-dir\") pod \"aws-ebs-csi-driver-node-579t6\" (UID: \"8eff746e-a59a-4d74-8822-66d3ee1bb7df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-579t6"
May 25 12:12:15.722077 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720743    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7cd43613-c252-4e65-9c14-f9bfa930be35-etc-kubernetes\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.722077 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720727    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8eff746e-a59a-4d74-8822-66d3ee1bb7df-device-dir\") pod \"aws-ebs-csi-driver-node-579t6\" (UID: \"8eff746e-a59a-4d74-8822-66d3ee1bb7df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-579t6"
May 25 12:12:15.722077 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720760    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8eff746e-a59a-4d74-8822-66d3ee1bb7df-socket-dir\") pod \"aws-ebs-csi-driver-node-579t6\" (UID: \"8eff746e-a59a-4d74-8822-66d3ee1bb7df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-579t6"
May 25 12:12:15.722077 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720782    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hrn2t\" (UniqueName: \"kubernetes.io/projected/e54faaa8-8c11-4a93-b337-c914cc73a410-kube-api-access-hrn2t\") pod \"network-check-target-slzvp\" (UID: \"e54faaa8-8c11-4a93-b337-c914cc73a410\") " pod="openshift-network-diagnostics/network-check-target-slzvp"
May 25 12:12:15.722077 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720789    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7cd43613-c252-4e65-9c14-f9bfa930be35-etc-kubernetes\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.722077 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720805    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8eff746e-a59a-4d74-8822-66d3ee1bb7df-registration-dir\") pod \"aws-ebs-csi-driver-node-579t6\" (UID: \"8eff746e-a59a-4d74-8822-66d3ee1bb7df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-579t6"
May 25 12:12:15.722077 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720820    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8eff746e-a59a-4d74-8822-66d3ee1bb7df-kubelet-dir\") pod \"aws-ebs-csi-driver-node-579t6\" (UID: \"8eff746e-a59a-4d74-8822-66d3ee1bb7df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-579t6"
May 25 12:12:15.722077 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720837    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7cd43613-c252-4e65-9c14-f9bfa930be35-host-run-k8s-cni-cncf-io\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.722077 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720859    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-br8w7\" (UniqueName: \"kubernetes.io/projected/7cd43613-c252-4e65-9c14-f9bfa930be35-kube-api-access-br8w7\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.722077 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720880    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8eff746e-a59a-4d74-8822-66d3ee1bb7df-kubelet-dir\") pod \"aws-ebs-csi-driver-node-579t6\" (UID: \"8eff746e-a59a-4d74-8822-66d3ee1bb7df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-579t6"
May 25 12:12:15.722556 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720887    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9f26ab5b-c341-495e-bac3-01d4475a795a-os-release\") pod \"multus-additional-cni-plugins-96cz9\" (UID: \"9f26ab5b-c341-495e-bac3-01d4475a795a\") " pod="openshift-multus/multus-additional-cni-plugins-96cz9"
May 25 12:12:15.722556 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720891    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7cd43613-c252-4e65-9c14-f9bfa930be35-host-run-k8s-cni-cncf-io\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.722556 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720912    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/012c465e-2683-488a-abc1-18ad86529eb3-ovnkube-script-lib\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.722556 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720941    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/27b08c8c-8256-4b27-86d6-6eb05ad8bfff-hosts-file\") pod \"node-resolver-5mdjk\" (UID: \"27b08c8c-8256-4b27-86d6-6eb05ad8bfff\") " pod="openshift-dns/node-resolver-5mdjk"
May 25 12:12:15.722556 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.720978    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/27b08c8c-8256-4b27-86d6-6eb05ad8bfff-hosts-file\") pod \"node-resolver-5mdjk\" (UID: \"27b08c8c-8256-4b27-86d6-6eb05ad8bfff\") " pod="openshift-dns/node-resolver-5mdjk"
May 25 12:12:15.722556 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.721011    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9f26ab5b-c341-495e-bac3-01d4475a795a-os-release\") pod \"multus-additional-cni-plugins-96cz9\" (UID: \"9f26ab5b-c341-495e-bac3-01d4475a795a\") " pod="openshift-multus/multus-additional-cni-plugins-96cz9"
May 25 12:12:15.722556 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.721029    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9f26ab5b-c341-495e-bac3-01d4475a795a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-96cz9\" (UID: \"9f26ab5b-c341-495e-bac3-01d4475a795a\") " pod="openshift-multus/multus-additional-cni-plugins-96cz9"
May 25 12:12:15.722556 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.721318    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/012c465e-2683-488a-abc1-18ad86529eb3-ovnkube-script-lib\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.722556 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.721358    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9f26ab5b-c341-495e-bac3-01d4475a795a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-96cz9\" (UID: \"9f26ab5b-c341-495e-bac3-01d4475a795a\") " pod="openshift-multus/multus-additional-cni-plugins-96cz9"
May 25 12:12:15.722556 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.722343    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/012c465e-2683-488a-abc1-18ad86529eb3-ovn-node-metrics-cert\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.726824 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:15.726799    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 12:12:15.726824 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:15.726824    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 12:12:15.727020 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:15.726837    2563 projected.go:194] Error preparing data for projected volume kube-api-access-hrn2t for pod openshift-network-diagnostics/network-check-target-slzvp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:12:15.727020 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:15.726915    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e54faaa8-8c11-4a93-b337-c914cc73a410-kube-api-access-hrn2t podName:e54faaa8-8c11-4a93-b337-c914cc73a410 nodeName:}" failed. No retries permitted until 2026-05-25 12:12:16.226896651 +0000 UTC m=+2.194288294 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hrn2t" (UniqueName: "kubernetes.io/projected/e54faaa8-8c11-4a93-b337-c914cc73a410-kube-api-access-hrn2t") pod "network-check-target-slzvp" (UID: "e54faaa8-8c11-4a93-b337-c914cc73a410") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:12:15.727129 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.727098    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs5hn\" (UniqueName: \"kubernetes.io/projected/873ca31f-e11b-4104-a3ae-243fdc6b54c0-kube-api-access-hs5hn\") pod \"iptables-alerter-k65b6\" (UID: \"873ca31f-e11b-4104-a3ae-243fdc6b54c0\") " pod="openshift-network-operator/iptables-alerter-k65b6"
May 25 12:12:15.728392 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.728365    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqvxf\" (UniqueName: \"kubernetes.io/projected/27b08c8c-8256-4b27-86d6-6eb05ad8bfff-kube-api-access-tqvxf\") pod \"node-resolver-5mdjk\" (UID: \"27b08c8c-8256-4b27-86d6-6eb05ad8bfff\") " pod="openshift-dns/node-resolver-5mdjk"
May 25 12:12:15.728472 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.728370    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sskn\" (UniqueName: \"kubernetes.io/projected/9f26ab5b-c341-495e-bac3-01d4475a795a-kube-api-access-8sskn\") pod \"multus-additional-cni-plugins-96cz9\" (UID: \"9f26ab5b-c341-495e-bac3-01d4475a795a\") " pod="openshift-multus/multus-additional-cni-plugins-96cz9"
May 25 12:12:15.728472 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.728449    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zplrw\" (UniqueName: \"kubernetes.io/projected/4d8f666d-fd9f-4ff8-90c8-b6bfeb3be623-kube-api-access-zplrw\") pod \"node-ca-sbtvl\" (UID: \"4d8f666d-fd9f-4ff8-90c8-b6bfeb3be623\") " pod="openshift-image-registry/node-ca-sbtvl"
May 25 12:12:15.728867 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.728848    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wphbc\" (UniqueName: \"kubernetes.io/projected/8eff746e-a59a-4d74-8822-66d3ee1bb7df-kube-api-access-wphbc\") pod \"aws-ebs-csi-driver-node-579t6\" (UID: \"8eff746e-a59a-4d74-8822-66d3ee1bb7df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-579t6"
May 25 12:12:15.728990 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.728973    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br8w7\" (UniqueName: \"kubernetes.io/projected/7cd43613-c252-4e65-9c14-f9bfa930be35-kube-api-access-br8w7\") pod \"multus-qnq6q\" (UID: \"7cd43613-c252-4e65-9c14-f9bfa930be35\") " pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.729208 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.729192    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gksk\" (UniqueName: \"kubernetes.io/projected/012c465e-2683-488a-abc1-18ad86529eb3-kube-api-access-7gksk\") pod \"ovnkube-node-cw9bb\" (UID: \"012c465e-2683-488a-abc1-18ad86529eb3\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.836471 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.836406    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5zd7t"
May 25 12:12:15.842904 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:15.842880    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedeeb54c_d929_487b_9fd3_d09c06073cdd.slice/crio-d32c71eaf7f23cf2791fc2a017faa2b67f86cca0c96adb9ba3cc5ede042acac6 WatchSource:0}: Error finding container d32c71eaf7f23cf2791fc2a017faa2b67f86cca0c96adb9ba3cc5ede042acac6: Status 404 returned error can't find the container with id d32c71eaf7f23cf2791fc2a017faa2b67f86cca0c96adb9ba3cc5ede042acac6
May 25 12:12:15.860855 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.860833    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vq5g9"
May 25 12:12:15.866291 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:15.866264    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b823f7_17fe_46bc_9c01_c126b9d66b9e.slice/crio-34944eaf6f3d63799129478b975e74899e5a7d1da93d0734dee9d855194e7a14 WatchSource:0}: Error finding container 34944eaf6f3d63799129478b975e74899e5a7d1da93d0734dee9d855194e7a14: Status 404 returned error can't find the container with id 34944eaf6f3d63799129478b975e74899e5a7d1da93d0734dee9d855194e7a14
May 25 12:12:15.873396 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.873382    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5mdjk"
May 25 12:12:15.878545 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:15.878507    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27b08c8c_8256_4b27_86d6_6eb05ad8bfff.slice/crio-449dff2aac90c28eeecf70463cd0cf0200d630b8ef4e14f3f089c6d30e4b0610 WatchSource:0}: Error finding container 449dff2aac90c28eeecf70463cd0cf0200d630b8ef4e14f3f089c6d30e4b0610: Status 404 returned error can't find the container with id 449dff2aac90c28eeecf70463cd0cf0200d630b8ef4e14f3f089c6d30e4b0610
May 25 12:12:15.890250 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.890219    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-96cz9"
May 25 12:12:15.896696 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:15.896669    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f26ab5b_c341_495e_bac3_01d4475a795a.slice/crio-f3503aad2c95cb5e00eecb0092ac0c060dc8737e3a5b862e0ea27775dd4f24f2 WatchSource:0}: Error finding container f3503aad2c95cb5e00eecb0092ac0c060dc8737e3a5b862e0ea27775dd4f24f2: Status 404 returned error can't find the container with id f3503aad2c95cb5e00eecb0092ac0c060dc8737e3a5b862e0ea27775dd4f24f2
May 25 12:12:15.905848 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.905827    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k65b6"
May 25 12:12:15.913835 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:15.913806    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod873ca31f_e11b_4104_a3ae_243fdc6b54c0.slice/crio-0b481e5c84d46ce9303f5afd7fe263675f43d3aebd57230201beb69dc2e82158 WatchSource:0}: Error finding container 0b481e5c84d46ce9303f5afd7fe263675f43d3aebd57230201beb69dc2e82158: Status 404 returned error can't find the container with id 0b481e5c84d46ce9303f5afd7fe263675f43d3aebd57230201beb69dc2e82158
May 25 12:12:15.922484 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.922467    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:15.929156 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.929127    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-579t6"
May 25 12:12:15.929828 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:15.929804    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod012c465e_2683_488a_abc1_18ad86529eb3.slice/crio-9a89cc9a4f3511d3af2971a3a5839402516e066f333e53e109a5b4835283b461 WatchSource:0}: Error finding container 9a89cc9a4f3511d3af2971a3a5839402516e066f333e53e109a5b4835283b461: Status 404 returned error can't find the container with id 9a89cc9a4f3511d3af2971a3a5839402516e066f333e53e109a5b4835283b461
May 25 12:12:15.935008 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.934988    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sbtvl"
May 25 12:12:15.935313 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:15.935290    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eff746e_a59a_4d74_8822_66d3ee1bb7df.slice/crio-54aec1fd2c0c1f1ed4ade783792b44558e1249e88adcca49a1d52b17b5ba3ec2 WatchSource:0}: Error finding container 54aec1fd2c0c1f1ed4ade783792b44558e1249e88adcca49a1d52b17b5ba3ec2: Status 404 returned error can't find the container with id 54aec1fd2c0c1f1ed4ade783792b44558e1249e88adcca49a1d52b17b5ba3ec2
May 25 12:12:15.939490 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:15.939469    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qnq6q"
May 25 12:12:15.941896 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:15.941869    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d8f666d_fd9f_4ff8_90c8_b6bfeb3be623.slice/crio-6f4af01828b2ffccd4188bc05f4b8f5efff509e4a67207959b583147783fea12 WatchSource:0}: Error finding container 6f4af01828b2ffccd4188bc05f4b8f5efff509e4a67207959b583147783fea12: Status 404 returned error can't find the container with id 6f4af01828b2ffccd4188bc05f4b8f5efff509e4a67207959b583147783fea12
May 25 12:12:15.947950 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:12:15.947928    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cd43613_c252_4e65_9c14_f9bfa930be35.slice/crio-01ab9e7594b496dd3a5cf9663d639acf05ae2c902f9ae5e19534088bd624fe76 WatchSource:0}: Error finding container 01ab9e7594b496dd3a5cf9663d639acf05ae2c902f9ae5e19534088bd624fe76: Status 404 returned error can't find the container with id 01ab9e7594b496dd3a5cf9663d639acf05ae2c902f9ae5e19534088bd624fe76
May 25 12:12:16.104687 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:16.104255    2563 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 12:12:16.124884 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:16.124469    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80098ceb-2e1b-4378-9371-6c6150ed88b6-metrics-certs\") pod \"network-metrics-daemon-6jsxz\" (UID: \"80098ceb-2e1b-4378-9371-6c6150ed88b6\") " pod="openshift-multus/network-metrics-daemon-6jsxz"
May 25 12:12:16.124884 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:16.124659    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:12:16.124884 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:16.124725    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80098ceb-2e1b-4378-9371-6c6150ed88b6-metrics-certs podName:80098ceb-2e1b-4378-9371-6c6150ed88b6 nodeName:}" failed. No retries permitted until 2026-05-25 12:12:17.124701778 +0000 UTC m=+3.092093395 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80098ceb-2e1b-4378-9371-6c6150ed88b6-metrics-certs") pod "network-metrics-daemon-6jsxz" (UID: "80098ceb-2e1b-4378-9371-6c6150ed88b6") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:12:16.264268 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:16.264234    2563 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 12:12:16.326960 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:16.326910    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hrn2t\" (UniqueName: \"kubernetes.io/projected/e54faaa8-8c11-4a93-b337-c914cc73a410-kube-api-access-hrn2t\") pod \"network-check-target-slzvp\" (UID: \"e54faaa8-8c11-4a93-b337-c914cc73a410\") " pod="openshift-network-diagnostics/network-check-target-slzvp"
May 25 12:12:16.327128 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:16.327112    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 12:12:16.327213 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:16.327134    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 12:12:16.327213 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:16.327147    2563 projected.go:194] Error preparing data for projected volume kube-api-access-hrn2t for pod openshift-network-diagnostics/network-check-target-slzvp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:12:16.327213 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:16.327208    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e54faaa8-8c11-4a93-b337-c914cc73a410-kube-api-access-hrn2t podName:e54faaa8-8c11-4a93-b337-c914cc73a410 nodeName:}" failed. No retries permitted until 2026-05-25 12:12:17.327188427 +0000 UTC m=+3.294580048 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hrn2t" (UniqueName: "kubernetes.io/projected/e54faaa8-8c11-4a93-b337-c914cc73a410-kube-api-access-hrn2t") pod "network-check-target-slzvp" (UID: "e54faaa8-8c11-4a93-b337-c914cc73a410") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:12:16.551094 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:16.550986    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-24 12:07:15 +0000 UTC" deadline="2027-11-23 17:45:18.315258101 +0000 UTC"
May 25 12:12:16.551094 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:16.551038    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13133h33m1.764223131s"
May 25 12:12:16.635155 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:16.635126    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slzvp"
May 25 12:12:16.635326 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:16.635243    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slzvp" podUID="e54faaa8-8c11-4a93-b337-c914cc73a410"
May 25 12:12:16.640735 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:16.640704    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qnq6q" event={"ID":"7cd43613-c252-4e65-9c14-f9bfa930be35","Type":"ContainerStarted","Data":"01ab9e7594b496dd3a5cf9663d639acf05ae2c902f9ae5e19534088bd624fe76"}
May 25 12:12:16.643125 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:16.643092    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-579t6" event={"ID":"8eff746e-a59a-4d74-8822-66d3ee1bb7df","Type":"ContainerStarted","Data":"54aec1fd2c0c1f1ed4ade783792b44558e1249e88adcca49a1d52b17b5ba3ec2"}
May 25 12:12:16.644722 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:16.644672    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5mdjk" event={"ID":"27b08c8c-8256-4b27-86d6-6eb05ad8bfff","Type":"ContainerStarted","Data":"449dff2aac90c28eeecf70463cd0cf0200d630b8ef4e14f3f089c6d30e4b0610"}
May 25 12:12:16.663584 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:16.659782    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5zd7t" event={"ID":"edeeb54c-d929-487b-9fd3-d09c06073cdd","Type":"ContainerStarted","Data":"d32c71eaf7f23cf2791fc2a017faa2b67f86cca0c96adb9ba3cc5ede042acac6"}
May 25 12:12:16.670741 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:16.670714    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sbtvl" event={"ID":"4d8f666d-fd9f-4ff8-90c8-b6bfeb3be623","Type":"ContainerStarted","Data":"6f4af01828b2ffccd4188bc05f4b8f5efff509e4a67207959b583147783fea12"}
May 25 12:12:16.683759 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:16.683700    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb" event={"ID":"012c465e-2683-488a-abc1-18ad86529eb3","Type":"ContainerStarted","Data":"9a89cc9a4f3511d3af2971a3a5839402516e066f333e53e109a5b4835283b461"}
May 25 12:12:16.701118 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:16.701087    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k65b6" event={"ID":"873ca31f-e11b-4104-a3ae-243fdc6b54c0","Type":"ContainerStarted","Data":"0b481e5c84d46ce9303f5afd7fe263675f43d3aebd57230201beb69dc2e82158"}
May 25 12:12:16.713278 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:16.713248    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-96cz9" event={"ID":"9f26ab5b-c341-495e-bac3-01d4475a795a","Type":"ContainerStarted","Data":"f3503aad2c95cb5e00eecb0092ac0c060dc8737e3a5b862e0ea27775dd4f24f2"}
May 25 12:12:16.721410 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:16.721366    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vq5g9" event={"ID":"45b823f7-17fe-46bc-9c01-c126b9d66b9e","Type":"ContainerStarted","Data":"34944eaf6f3d63799129478b975e74899e5a7d1da93d0734dee9d855194e7a14"}
May 25 12:12:16.894322 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:16.894288    2563 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 12:12:17.134015 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:17.133414    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80098ceb-2e1b-4378-9371-6c6150ed88b6-metrics-certs\") pod \"network-metrics-daemon-6jsxz\" (UID: \"80098ceb-2e1b-4378-9371-6c6150ed88b6\") " pod="openshift-multus/network-metrics-daemon-6jsxz"
May 25 12:12:17.134015 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:17.133579    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:12:17.134015 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:17.133642    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80098ceb-2e1b-4378-9371-6c6150ed88b6-metrics-certs podName:80098ceb-2e1b-4378-9371-6c6150ed88b6 nodeName:}" failed. No retries permitted until 2026-05-25 12:12:19.133622798 +0000 UTC m=+5.101014418 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80098ceb-2e1b-4378-9371-6c6150ed88b6-metrics-certs") pod "network-metrics-daemon-6jsxz" (UID: "80098ceb-2e1b-4378-9371-6c6150ed88b6") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:12:17.335837 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:17.335173    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hrn2t\" (UniqueName: \"kubernetes.io/projected/e54faaa8-8c11-4a93-b337-c914cc73a410-kube-api-access-hrn2t\") pod \"network-check-target-slzvp\" (UID: \"e54faaa8-8c11-4a93-b337-c914cc73a410\") " pod="openshift-network-diagnostics/network-check-target-slzvp"
May 25 12:12:17.335837 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:17.335380    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 12:12:17.335837 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:17.335399    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 12:12:17.335837 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:17.335412    2563 projected.go:194] Error preparing data for projected volume kube-api-access-hrn2t for pod openshift-network-diagnostics/network-check-target-slzvp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:12:17.335837 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:17.335470    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e54faaa8-8c11-4a93-b337-c914cc73a410-kube-api-access-hrn2t podName:e54faaa8-8c11-4a93-b337-c914cc73a410 nodeName:}" failed. No retries permitted until 2026-05-25 12:12:19.335451713 +0000 UTC m=+5.302843333 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hrn2t" (UniqueName: "kubernetes.io/projected/e54faaa8-8c11-4a93-b337-c914cc73a410-kube-api-access-hrn2t") pod "network-check-target-slzvp" (UID: "e54faaa8-8c11-4a93-b337-c914cc73a410") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:12:17.551566 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:17.551451    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-24 12:07:15 +0000 UTC" deadline="2028-01-20 07:45:50.082827748 +0000 UTC"
May 25 12:12:17.551566 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:17.551494    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14515h33m32.531336946s"
May 25 12:12:17.633718 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:17.633206    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jsxz"
May 25 12:12:17.633718 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:17.633342    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jsxz" podUID="80098ceb-2e1b-4378-9371-6c6150ed88b6"
May 25 12:12:17.932207 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:17.931888    2563 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 12:12:18.634187 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:18.633698    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slzvp"
May 25 12:12:18.634187 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:18.633831    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slzvp" podUID="e54faaa8-8c11-4a93-b337-c914cc73a410"
May 25 12:12:19.152590 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:19.152557    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80098ceb-2e1b-4378-9371-6c6150ed88b6-metrics-certs\") pod \"network-metrics-daemon-6jsxz\" (UID: \"80098ceb-2e1b-4378-9371-6c6150ed88b6\") " pod="openshift-multus/network-metrics-daemon-6jsxz"
May 25 12:12:19.152773 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:19.152726    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:12:19.152879 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:19.152812    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80098ceb-2e1b-4378-9371-6c6150ed88b6-metrics-certs podName:80098ceb-2e1b-4378-9371-6c6150ed88b6 nodeName:}" failed. No retries permitted until 2026-05-25 12:12:23.152791237 +0000 UTC m=+9.120182869 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80098ceb-2e1b-4378-9371-6c6150ed88b6-metrics-certs") pod "network-metrics-daemon-6jsxz" (UID: "80098ceb-2e1b-4378-9371-6c6150ed88b6") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:12:19.354611 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:19.354513    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hrn2t\" (UniqueName: \"kubernetes.io/projected/e54faaa8-8c11-4a93-b337-c914cc73a410-kube-api-access-hrn2t\") pod \"network-check-target-slzvp\" (UID: \"e54faaa8-8c11-4a93-b337-c914cc73a410\") " pod="openshift-network-diagnostics/network-check-target-slzvp"
May 25 12:12:19.354787 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:19.354667    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 12:12:19.354787 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:19.354695    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 12:12:19.354787 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:19.354708    2563 projected.go:194] Error preparing data for projected volume kube-api-access-hrn2t for pod openshift-network-diagnostics/network-check-target-slzvp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:12:19.354787 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:19.354774    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e54faaa8-8c11-4a93-b337-c914cc73a410-kube-api-access-hrn2t podName:e54faaa8-8c11-4a93-b337-c914cc73a410 nodeName:}" failed. No retries permitted until 2026-05-25 12:12:23.354754708 +0000 UTC m=+9.322146330 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hrn2t" (UniqueName: "kubernetes.io/projected/e54faaa8-8c11-4a93-b337-c914cc73a410-kube-api-access-hrn2t") pod "network-check-target-slzvp" (UID: "e54faaa8-8c11-4a93-b337-c914cc73a410") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:12:19.633036 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:19.632945    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jsxz"
May 25 12:12:19.633196 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:19.633084    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jsxz" podUID="80098ceb-2e1b-4378-9371-6c6150ed88b6"
May 25 12:12:20.633156 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:20.633125    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slzvp"
May 25 12:12:20.633592 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:20.633254    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slzvp" podUID="e54faaa8-8c11-4a93-b337-c914cc73a410"
May 25 12:12:21.633791 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:21.633753    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jsxz"
May 25 12:12:21.634242 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:21.633929    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jsxz" podUID="80098ceb-2e1b-4378-9371-6c6150ed88b6"
May 25 12:12:22.633692 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:22.633623    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slzvp"
May 25 12:12:22.633849 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:22.633744    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slzvp" podUID="e54faaa8-8c11-4a93-b337-c914cc73a410"
May 25 12:12:23.186496 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:23.186445    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80098ceb-2e1b-4378-9371-6c6150ed88b6-metrics-certs\") pod \"network-metrics-daemon-6jsxz\" (UID: \"80098ceb-2e1b-4378-9371-6c6150ed88b6\") " pod="openshift-multus/network-metrics-daemon-6jsxz"
May 25 12:12:23.186680 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:23.186667    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:12:23.186742 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:23.186732    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80098ceb-2e1b-4378-9371-6c6150ed88b6-metrics-certs podName:80098ceb-2e1b-4378-9371-6c6150ed88b6 nodeName:}" failed. No retries permitted until 2026-05-25 12:12:31.186712567 +0000 UTC m=+17.154104198 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80098ceb-2e1b-4378-9371-6c6150ed88b6-metrics-certs") pod "network-metrics-daemon-6jsxz" (UID: "80098ceb-2e1b-4378-9371-6c6150ed88b6") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:12:23.387770 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:23.387732    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hrn2t\" (UniqueName: \"kubernetes.io/projected/e54faaa8-8c11-4a93-b337-c914cc73a410-kube-api-access-hrn2t\") pod \"network-check-target-slzvp\" (UID: \"e54faaa8-8c11-4a93-b337-c914cc73a410\") " pod="openshift-network-diagnostics/network-check-target-slzvp"
May 25 12:12:23.387956 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:23.387903    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 12:12:23.387956 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:23.387925    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 12:12:23.387956 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:23.387938    2563 projected.go:194] Error preparing data for projected volume kube-api-access-hrn2t for pod openshift-network-diagnostics/network-check-target-slzvp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:12:23.388105 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:23.387999    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e54faaa8-8c11-4a93-b337-c914cc73a410-kube-api-access-hrn2t podName:e54faaa8-8c11-4a93-b337-c914cc73a410 nodeName:}" failed. No retries permitted until 2026-05-25 12:12:31.387979778 +0000 UTC m=+17.355371401 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hrn2t" (UniqueName: "kubernetes.io/projected/e54faaa8-8c11-4a93-b337-c914cc73a410-kube-api-access-hrn2t") pod "network-check-target-slzvp" (UID: "e54faaa8-8c11-4a93-b337-c914cc73a410") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:12:23.632782 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:23.632696    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jsxz"
May 25 12:12:23.632972 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:23.632857    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jsxz" podUID="80098ceb-2e1b-4378-9371-6c6150ed88b6"
May 25 12:12:24.634404 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:24.634229    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slzvp"
May 25 12:12:24.634854 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:24.634415    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slzvp" podUID="e54faaa8-8c11-4a93-b337-c914cc73a410"
May 25 12:12:25.632918 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:25.632886    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jsxz"
May 25 12:12:25.633100 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:25.633003    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jsxz" podUID="80098ceb-2e1b-4378-9371-6c6150ed88b6"
May 25 12:12:26.633310 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:26.633277    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slzvp"
May 25 12:12:26.633672 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:26.633369    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slzvp" podUID="e54faaa8-8c11-4a93-b337-c914cc73a410"
May 25 12:12:27.632882 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:27.632844    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jsxz"
May 25 12:12:27.633042 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:27.632981    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jsxz" podUID="80098ceb-2e1b-4378-9371-6c6150ed88b6"
May 25 12:12:28.633700 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:28.633661    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slzvp"
May 25 12:12:28.634175 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:28.633781    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slzvp" podUID="e54faaa8-8c11-4a93-b337-c914cc73a410"
May 25 12:12:29.633259 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:29.633217    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jsxz"
May 25 12:12:29.633435 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:29.633340    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jsxz" podUID="80098ceb-2e1b-4378-9371-6c6150ed88b6"
May 25 12:12:30.635185 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:30.634996    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slzvp"
May 25 12:12:30.635569 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:30.635279    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slzvp" podUID="e54faaa8-8c11-4a93-b337-c914cc73a410"
May 25 12:12:31.242814 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:31.242770    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80098ceb-2e1b-4378-9371-6c6150ed88b6-metrics-certs\") pod \"network-metrics-daemon-6jsxz\" (UID: \"80098ceb-2e1b-4378-9371-6c6150ed88b6\") " pod="openshift-multus/network-metrics-daemon-6jsxz"
May 25 12:12:31.243045 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:31.242971    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:12:31.243112 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:31.243054    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80098ceb-2e1b-4378-9371-6c6150ed88b6-metrics-certs podName:80098ceb-2e1b-4378-9371-6c6150ed88b6 nodeName:}" failed. No retries permitted until 2026-05-25 12:12:47.243032939 +0000 UTC m=+33.210424570 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80098ceb-2e1b-4378-9371-6c6150ed88b6-metrics-certs") pod "network-metrics-daemon-6jsxz" (UID: "80098ceb-2e1b-4378-9371-6c6150ed88b6") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:12:31.444809 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:31.444756    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hrn2t\" (UniqueName: \"kubernetes.io/projected/e54faaa8-8c11-4a93-b337-c914cc73a410-kube-api-access-hrn2t\") pod \"network-check-target-slzvp\" (UID: \"e54faaa8-8c11-4a93-b337-c914cc73a410\") " pod="openshift-network-diagnostics/network-check-target-slzvp"
May 25 12:12:31.444967 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:31.444938    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 12:12:31.444967 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:31.444965    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 12:12:31.445068 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:31.444974    2563 projected.go:194] Error preparing data for projected volume kube-api-access-hrn2t for pod openshift-network-diagnostics/network-check-target-slzvp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:12:31.445068 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:31.445040    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e54faaa8-8c11-4a93-b337-c914cc73a410-kube-api-access-hrn2t podName:e54faaa8-8c11-4a93-b337-c914cc73a410 nodeName:}" failed. No retries permitted until 2026-05-25 12:12:47.445021735 +0000 UTC m=+33.412413365 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hrn2t" (UniqueName: "kubernetes.io/projected/e54faaa8-8c11-4a93-b337-c914cc73a410-kube-api-access-hrn2t") pod "network-check-target-slzvp" (UID: "e54faaa8-8c11-4a93-b337-c914cc73a410") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:12:31.633428 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:31.633346    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jsxz"
May 25 12:12:31.633618 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:31.633465    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jsxz" podUID="80098ceb-2e1b-4378-9371-6c6150ed88b6"
May 25 12:12:32.633018 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:32.632977    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slzvp"
May 25 12:12:32.633466 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:32.633124    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slzvp" podUID="e54faaa8-8c11-4a93-b337-c914cc73a410"
May 25 12:12:33.632963 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:33.632930    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jsxz"
May 25 12:12:33.633121 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:33.633058    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jsxz" podUID="80098ceb-2e1b-4378-9371-6c6150ed88b6"
May 25 12:12:34.633320 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:34.633289    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slzvp"
May 25 12:12:34.633672 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:34.633369    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slzvp" podUID="e54faaa8-8c11-4a93-b337-c914cc73a410"
May 25 12:12:35.633379 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:35.633218    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jsxz"
May 25 12:12:35.633847 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:35.633459    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jsxz" podUID="80098ceb-2e1b-4378-9371-6c6150ed88b6"
May 25 12:12:35.754752 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:35.754722    2563 generic.go:358] "Generic (PLEG): container finished" podID="fbb7f0c11b7bf6cb015cb8917e9cb796" containerID="5b3b3e6c9eae86b9cca9ee40ec7491e32c4df9d46d619baa4b755f0620b887b0" exitCode=0
May 25 12:12:35.754875 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:35.754789    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.ec2.internal" event={"ID":"fbb7f0c11b7bf6cb015cb8917e9cb796","Type":"ContainerDied","Data":"5b3b3e6c9eae86b9cca9ee40ec7491e32c4df9d46d619baa4b755f0620b887b0"}
May 25 12:12:35.756458 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:35.756422    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qnq6q" event={"ID":"7cd43613-c252-4e65-9c14-f9bfa930be35","Type":"ContainerStarted","Data":"24c8901b14140326dd7872af1fef890c353fdc4dc326529eb50f0d46ead44ab1"}
May 25 12:12:35.757898 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:35.757849    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-579t6" event={"ID":"8eff746e-a59a-4d74-8822-66d3ee1bb7df","Type":"ContainerStarted","Data":"2fdf1ca0df8fce393f5059a490a3861ec7f798f8da68c7685732b96b227ab650"}
May 25 12:12:35.759286 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:35.759264    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5mdjk" event={"ID":"27b08c8c-8256-4b27-86d6-6eb05ad8bfff","Type":"ContainerStarted","Data":"b626af538f945f1c7c46a8fb48f481201b177be57c3dccd90e1a51f53aa2ba13"}
May 25 12:12:35.760568 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:35.760542    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5zd7t" event={"ID":"edeeb54c-d929-487b-9fd3-d09c06073cdd","Type":"ContainerStarted","Data":"5b4d60860f6ad7e493de1ee3321adb9ce8fe3e755256db7aed1c70dde878d4b1"}
May 25 12:12:35.761870 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:35.761845    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-183.ec2.internal" event={"ID":"44c84599553bc2d6ec69b56732f924ff","Type":"ContainerStarted","Data":"20b21881e2cb34722188fc2ebb1a0f51659917d31b83a942002d59d39b22905e"}
May 25 12:12:35.763156 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:35.763125    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sbtvl" event={"ID":"4d8f666d-fd9f-4ff8-90c8-b6bfeb3be623","Type":"ContainerStarted","Data":"247b9b0c8e6335ac6443daa9dcec9eae1643bdc057f2cd44b69b59af02730e7b"}
May 25 12:12:35.765909 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:35.765890    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw9bb_012c465e-2683-488a-abc1-18ad86529eb3/ovn-acl-logging/0.log"
May 25 12:12:35.766255 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:35.766228    2563 generic.go:358] "Generic (PLEG): container finished" podID="012c465e-2683-488a-abc1-18ad86529eb3" containerID="ee22c9c84eac133ab074393bdbdc0db7b15762286634164dcb698e5a9829f2f0" exitCode=1
May 25 12:12:35.766361 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:35.766302    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb" event={"ID":"012c465e-2683-488a-abc1-18ad86529eb3","Type":"ContainerStarted","Data":"1d2d743c40440e3d49cbd185239b712932a271a2cba1b6c3436b009ebff97c57"}
May 25 12:12:35.766361 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:35.766329    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb" event={"ID":"012c465e-2683-488a-abc1-18ad86529eb3","Type":"ContainerStarted","Data":"95db092b1ca17f69a94f665a5821b1c3ab06924417b2c698c93f849e951c2d14"}
May 25 12:12:35.766361 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:35.766342    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb" event={"ID":"012c465e-2683-488a-abc1-18ad86529eb3","Type":"ContainerStarted","Data":"58b650aa83b3d1a59f4b91cc7c0c4639750d2f11b4e648d2ad5d418aeff390ba"}
May 25 12:12:35.766361 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:35.766354    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb" event={"ID":"012c465e-2683-488a-abc1-18ad86529eb3","Type":"ContainerStarted","Data":"f626e26ea483ca64b2b3dfe8113d8a129869d0c57726bd7f99865f5b0db38f4f"}
May 25 12:12:35.766598 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:35.766365    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb" event={"ID":"012c465e-2683-488a-abc1-18ad86529eb3","Type":"ContainerDied","Data":"ee22c9c84eac133ab074393bdbdc0db7b15762286634164dcb698e5a9829f2f0"}
May 25 12:12:35.766598 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:35.766379    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb" event={"ID":"012c465e-2683-488a-abc1-18ad86529eb3","Type":"ContainerStarted","Data":"13f70e1aed3de7229157593b7d12123f8f07b244e29fc9fc3331f900ef6c1b29"}
May 25 12:12:35.767653 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:35.767632    2563 generic.go:358] "Generic (PLEG): container finished" podID="9f26ab5b-c341-495e-bac3-01d4475a795a" containerID="aff2d6a0dbf1ebd693691c117959f9a77449db44c4c1a7fe96e8de69efddd24f" exitCode=0
May 25 12:12:35.767737 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:35.767703    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-96cz9" event={"ID":"9f26ab5b-c341-495e-bac3-01d4475a795a","Type":"ContainerDied","Data":"aff2d6a0dbf1ebd693691c117959f9a77449db44c4c1a7fe96e8de69efddd24f"}
May 25 12:12:35.769121 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:35.769086    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vq5g9" event={"ID":"45b823f7-17fe-46bc-9c01-c126b9d66b9e","Type":"ContainerStarted","Data":"198bc3d7f46f9e44dabc614da45b77156ff00aa56d2b8165b86f28c42e3fd062"}
May 25 12:12:35.787900 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:35.787659    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-sbtvl" podStartSLOduration=2.721280712 podStartE2EDuration="21.787641155s" podCreationTimestamp="2026-05-25 12:12:14 +0000 UTC" firstStartedPulling="2026-05-25 12:12:15.945449209 +0000 UTC m=+1.912840828" lastFinishedPulling="2026-05-25 12:12:35.01180964 +0000 UTC m=+20.979201271" observedRunningTime="2026-05-25 12:12:35.78726783 +0000 UTC m=+21.754659469" watchObservedRunningTime="2026-05-25 12:12:35.787641155 +0000 UTC m=+21.755032795"
May 25 12:12:35.803734 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:35.803644    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5zd7t" podStartSLOduration=2.613244661 podStartE2EDuration="21.803628162s" podCreationTimestamp="2026-05-25 12:12:14 +0000 UTC" firstStartedPulling="2026-05-25 12:12:15.844164216 +0000 UTC m=+1.811555836" lastFinishedPulling="2026-05-25 12:12:35.034547716 +0000 UTC m=+21.001939337" observedRunningTime="2026-05-25 12:12:35.803394419 +0000 UTC m=+21.770786060" watchObservedRunningTime="2026-05-25 12:12:35.803628162 +0000 UTC m=+21.771019802"
May 25 12:12:35.819332 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:35.819295    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-5mdjk" podStartSLOduration=2.660455674 podStartE2EDuration="21.81928044s" podCreationTimestamp="2026-05-25 12:12:14 +0000 UTC" firstStartedPulling="2026-05-25 12:12:15.879956636 +0000 UTC m=+1.847348256" lastFinishedPulling="2026-05-25 12:12:35.038781404 +0000 UTC m=+21.006173022" observedRunningTime="2026-05-25 12:12:35.818781424 +0000 UTC m=+21.786173065" watchObservedRunningTime="2026-05-25 12:12:35.81928044 +0000 UTC m=+21.786672080"
May 25 12:12:35.836456 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:35.836397    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-qnq6q" podStartSLOduration=2.519015241 podStartE2EDuration="21.836377445s" podCreationTimestamp="2026-05-25 12:12:14 +0000 UTC" firstStartedPulling="2026-05-25 12:12:15.949866751 +0000 UTC m=+1.917258374" lastFinishedPulling="2026-05-25 12:12:35.267228947 +0000 UTC m=+21.234620578" observedRunningTime="2026-05-25 12:12:35.835537797 +0000 UTC m=+21.802929430" watchObservedRunningTime="2026-05-25 12:12:35.836377445 +0000 UTC m=+21.803769085"
May 25 12:12:35.847625 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:35.847573    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-183.ec2.internal" podStartSLOduration=20.847545217 podStartE2EDuration="20.847545217s" podCreationTimestamp="2026-05-25 12:12:15 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:12:35.847088254 +0000 UTC m=+21.814479894" watchObservedRunningTime="2026-05-25 12:12:35.847545217 +0000 UTC m=+21.814936858"
May 25 12:12:35.861334 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:35.861283    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vq5g9" podStartSLOduration=3.144476957 podStartE2EDuration="21.861267903s" podCreationTimestamp="2026-05-25 12:12:14 +0000 UTC" firstStartedPulling="2026-05-25 12:12:15.867865411 +0000 UTC m=+1.835257028" lastFinishedPulling="2026-05-25 12:12:34.584656343 +0000 UTC m=+20.552047974" observedRunningTime="2026-05-25 12:12:35.860782428 +0000 UTC m=+21.828174068" watchObservedRunningTime="2026-05-25 12:12:35.861267903 +0000 UTC m=+21.828659544"
May 25 12:12:36.632921 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:36.632890    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slzvp"
May 25 12:12:36.633077 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:36.633019    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slzvp" podUID="e54faaa8-8c11-4a93-b337-c914cc73a410"
May 25 12:12:36.641161 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:36.641125    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 25 12:12:36.773184 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:36.773097    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.ec2.internal" event={"ID":"fbb7f0c11b7bf6cb015cb8917e9cb796","Type":"ContainerStarted","Data":"ab808cd0886ef37a7c5e4daf1dbd2bb23b087d938736b2b33499563ae3fad8c5"}
May 25 12:12:36.774824 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:36.774777    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-579t6" event={"ID":"8eff746e-a59a-4d74-8822-66d3ee1bb7df","Type":"ContainerStarted","Data":"e70e4a7e689ae0638b3d6e875e4c7dd150f380dff36e55cb4dad7b7fab2e7794"}
May 25 12:12:36.776286 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:36.776190    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k65b6" event={"ID":"873ca31f-e11b-4104-a3ae-243fdc6b54c0","Type":"ContainerStarted","Data":"fbb41cace132d1740cd1d19fb50714a99e76f431cfd51e581545aa8c0cefc66f"}
May 25 12:12:36.787245 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:36.787201    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.ec2.internal" podStartSLOduration=21.787186029 podStartE2EDuration="21.787186029s" podCreationTimestamp="2026-05-25 12:12:15 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:12:36.786711845 +0000 UTC m=+22.754103483" watchObservedRunningTime="2026-05-25 12:12:36.787186029 +0000 UTC m=+22.754577668"
May 25 12:12:36.799678 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:36.799633    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-k65b6" podStartSLOduration=3.679505797 podStartE2EDuration="22.799613618s" podCreationTimestamp="2026-05-25 12:12:14 +0000 UTC" firstStartedPulling="2026-05-25 12:12:15.915790615 +0000 UTC m=+1.883182233" lastFinishedPulling="2026-05-25 12:12:35.035898422 +0000 UTC m=+21.003290054" observedRunningTime="2026-05-25 12:12:36.799112505 +0000 UTC m=+22.766504144" watchObservedRunningTime="2026-05-25 12:12:36.799613618 +0000 UTC m=+22.767005256"
May 25 12:12:37.126367 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:37.126249    2563 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vq5g9"
May 25 12:12:37.127342 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:37.127304    2563 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vq5g9"
May 25 12:12:37.584242 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:37.584130    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-25T12:12:36.641142915Z","UUID":"845e6358-3d7d-45a9-849b-69096ee35b83","Handler":null,"Name":"","Endpoint":""}
May 25 12:12:37.587217 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:37.587194    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 25 12:12:37.587344 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:37.587226    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 25 12:12:37.633040 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:37.633010    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jsxz"
May 25 12:12:37.633180 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:37.633132    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jsxz" podUID="80098ceb-2e1b-4378-9371-6c6150ed88b6"
May 25 12:12:37.780679 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:37.780580    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-579t6" event={"ID":"8eff746e-a59a-4d74-8822-66d3ee1bb7df","Type":"ContainerStarted","Data":"093121fe1e5151263543226181d418f9b3a8e8838a65971e330bb6ca5655da6f"}
May 25 12:12:37.781119 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:37.781098    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vq5g9"
May 25 12:12:37.781402 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:37.781380    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vq5g9"
May 25 12:12:37.799488 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:37.798502    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-579t6" podStartSLOduration=2.223347898 podStartE2EDuration="23.798486045s" podCreationTimestamp="2026-05-25 12:12:14 +0000 UTC" firstStartedPulling="2026-05-25 12:12:15.93802245 +0000 UTC m=+1.905414071" lastFinishedPulling="2026-05-25 12:12:37.513160601 +0000 UTC m=+23.480552218" observedRunningTime="2026-05-25 12:12:37.797685466 +0000 UTC m=+23.765077107" watchObservedRunningTime="2026-05-25 12:12:37.798486045 +0000 UTC m=+23.765877693"
May 25 12:12:38.633461 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:38.633432    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slzvp"
May 25 12:12:38.633667 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:38.633569    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slzvp" podUID="e54faaa8-8c11-4a93-b337-c914cc73a410"
May 25 12:12:38.784734 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:38.784704    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw9bb_012c465e-2683-488a-abc1-18ad86529eb3/ovn-acl-logging/0.log"
May 25 12:12:38.785180 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:38.785088    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb" event={"ID":"012c465e-2683-488a-abc1-18ad86529eb3","Type":"ContainerStarted","Data":"d4102a9ea9a9c7524f1f8f1e584c16b3e1e25a5e86d94a21ea0469f84b336128"}
May 25 12:12:39.632887 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:39.632858    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jsxz"
May 25 12:12:39.633050 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:39.632957    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jsxz" podUID="80098ceb-2e1b-4378-9371-6c6150ed88b6"
May 25 12:12:40.633784 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:40.633565    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slzvp"
May 25 12:12:40.634288 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:40.633800    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slzvp" podUID="e54faaa8-8c11-4a93-b337-c914cc73a410"
May 25 12:12:40.790815 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:40.790783    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw9bb_012c465e-2683-488a-abc1-18ad86529eb3/ovn-acl-logging/0.log"
May 25 12:12:40.791104 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:40.791082    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb" event={"ID":"012c465e-2683-488a-abc1-18ad86529eb3","Type":"ContainerStarted","Data":"dda57181a332c8bf7e8b4e57d3468bfcfc647d64bd071c50218c100277007395"}
May 25 12:12:40.791440 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:40.791418    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:40.791608 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:40.791593    2563 scope.go:117] "RemoveContainer" containerID="ee22c9c84eac133ab074393bdbdc0db7b15762286634164dcb698e5a9829f2f0"
May 25 12:12:40.792940 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:40.792910    2563 generic.go:358] "Generic (PLEG): container finished" podID="9f26ab5b-c341-495e-bac3-01d4475a795a" containerID="8571bcf3f5489a72b90bb6c7ecc02c70141025ef31c2550060fad075ae55ea68" exitCode=0
May 25 12:12:40.793032 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:40.792960    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-96cz9" event={"ID":"9f26ab5b-c341-495e-bac3-01d4475a795a","Type":"ContainerDied","Data":"8571bcf3f5489a72b90bb6c7ecc02c70141025ef31c2550060fad075ae55ea68"}
May 25 12:12:40.808351 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:40.808333    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:41.633171 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:41.633139    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jsxz"
May 25 12:12:41.633308 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:41.633247    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jsxz" podUID="80098ceb-2e1b-4378-9371-6c6150ed88b6"
May 25 12:12:41.798273 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:41.798127    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw9bb_012c465e-2683-488a-abc1-18ad86529eb3/ovn-acl-logging/0.log"
May 25 12:12:41.798752 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:41.798566    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb" event={"ID":"012c465e-2683-488a-abc1-18ad86529eb3","Type":"ContainerStarted","Data":"6ee2098639557ec2c55152e90813cfda0667d007ceddbf6f69015ed7353104cb"}
May 25 12:12:41.798917 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:41.798881    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:41.798917 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:41.798916    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:41.800499 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:41.800478    2563 generic.go:358] "Generic (PLEG): container finished" podID="9f26ab5b-c341-495e-bac3-01d4475a795a" containerID="695cedf4483b3c9a18a82dffefdcd7b20e9d22e0c910082a91b1b2322ca99e3f" exitCode=0
May 25 12:12:41.800595 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:41.800511    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-96cz9" event={"ID":"9f26ab5b-c341-495e-bac3-01d4475a795a","Type":"ContainerDied","Data":"695cedf4483b3c9a18a82dffefdcd7b20e9d22e0c910082a91b1b2322ca99e3f"}
May 25 12:12:41.813977 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:41.813958    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:12:41.827248 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:41.827211    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb" podStartSLOduration=8.708226087 podStartE2EDuration="27.827198517s" podCreationTimestamp="2026-05-25 12:12:14 +0000 UTC" firstStartedPulling="2026-05-25 12:12:15.932237644 +0000 UTC m=+1.899629266" lastFinishedPulling="2026-05-25 12:12:35.051210076 +0000 UTC m=+21.018601696" observedRunningTime="2026-05-25 12:12:41.826170787 +0000 UTC m=+27.793562425" watchObservedRunningTime="2026-05-25 12:12:41.827198517 +0000 UTC m=+27.794590179"
May 25 12:12:41.962086 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:41.962040    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-slzvp"]
May 25 12:12:41.962329 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:41.962195    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slzvp"
May 25 12:12:41.962329 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:41.962292    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slzvp" podUID="e54faaa8-8c11-4a93-b337-c914cc73a410"
May 25 12:12:41.962653 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:41.962626    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6jsxz"]
May 25 12:12:41.962758 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:41.962721    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jsxz"
May 25 12:12:41.962860 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:41.962834    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jsxz" podUID="80098ceb-2e1b-4378-9371-6c6150ed88b6"
May 25 12:12:42.803779 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:42.803688    2563 generic.go:358] "Generic (PLEG): container finished" podID="9f26ab5b-c341-495e-bac3-01d4475a795a" containerID="0b9c7003ff53811dc197b03ba4f797e3693deec905467c049189a823fd4ae755" exitCode=0
May 25 12:12:42.803779 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:42.803733    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-96cz9" event={"ID":"9f26ab5b-c341-495e-bac3-01d4475a795a","Type":"ContainerDied","Data":"0b9c7003ff53811dc197b03ba4f797e3693deec905467c049189a823fd4ae755"}
May 25 12:12:43.632967 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:43.632934    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jsxz"
May 25 12:12:43.633167 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:43.632934    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slzvp"
May 25 12:12:43.633167 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:43.633068    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jsxz" podUID="80098ceb-2e1b-4378-9371-6c6150ed88b6"
May 25 12:12:43.633167 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:43.633154    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slzvp" podUID="e54faaa8-8c11-4a93-b337-c914cc73a410"
May 25 12:12:45.633079 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:45.633046    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slzvp"
May 25 12:12:45.633607 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:45.633123    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jsxz"
May 25 12:12:45.633607 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:45.633248    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slzvp" podUID="e54faaa8-8c11-4a93-b337-c914cc73a410"
May 25 12:12:45.633607 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:45.633371    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jsxz" podUID="80098ceb-2e1b-4378-9371-6c6150ed88b6"
May 25 12:12:47.273421 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:47.273376    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80098ceb-2e1b-4378-9371-6c6150ed88b6-metrics-certs\") pod \"network-metrics-daemon-6jsxz\" (UID: \"80098ceb-2e1b-4378-9371-6c6150ed88b6\") " pod="openshift-multus/network-metrics-daemon-6jsxz"
May 25 12:12:47.273895 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:47.273552    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:12:47.273895 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:47.273632    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80098ceb-2e1b-4378-9371-6c6150ed88b6-metrics-certs podName:80098ceb-2e1b-4378-9371-6c6150ed88b6 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:19.273610882 +0000 UTC m=+65.241002516 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80098ceb-2e1b-4378-9371-6c6150ed88b6-metrics-certs") pod "network-metrics-daemon-6jsxz" (UID: "80098ceb-2e1b-4378-9371-6c6150ed88b6") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:12:47.474206 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:47.474168    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hrn2t\" (UniqueName: \"kubernetes.io/projected/e54faaa8-8c11-4a93-b337-c914cc73a410-kube-api-access-hrn2t\") pod \"network-check-target-slzvp\" (UID: \"e54faaa8-8c11-4a93-b337-c914cc73a410\") " pod="openshift-network-diagnostics/network-check-target-slzvp"
May 25 12:12:47.474415 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:47.474352    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 12:12:47.474415 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:47.474376    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 12:12:47.474415 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:47.474392    2563 projected.go:194] Error preparing data for projected volume kube-api-access-hrn2t for pod openshift-network-diagnostics/network-check-target-slzvp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:12:47.474552 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:47.474454    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e54faaa8-8c11-4a93-b337-c914cc73a410-kube-api-access-hrn2t podName:e54faaa8-8c11-4a93-b337-c914cc73a410 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:19.474436029 +0000 UTC m=+65.441827651 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hrn2t" (UniqueName: "kubernetes.io/projected/e54faaa8-8c11-4a93-b337-c914cc73a410-kube-api-access-hrn2t") pod "network-check-target-slzvp" (UID: "e54faaa8-8c11-4a93-b337-c914cc73a410") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:12:47.633261 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:47.633186    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slzvp"
May 25 12:12:47.633407 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:47.633186    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jsxz"
May 25 12:12:47.633407 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:47.633288    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slzvp" podUID="e54faaa8-8c11-4a93-b337-c914cc73a410"
May 25 12:12:47.633407 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:47.633370    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jsxz" podUID="80098ceb-2e1b-4378-9371-6c6150ed88b6"
May 25 12:12:47.844171 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:47.844143    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-183.ec2.internal" event="NodeReady"
May 25 12:12:47.844327 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:47.844250    2563 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 25 12:12:47.884862 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:47.884797    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mwj87"]
May 25 12:12:47.898310 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:47.898288    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jv4s8"]
May 25 12:12:47.898447 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:47.898429    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mwj87"
May 25 12:12:47.900620 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:47.900572    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 25 12:12:47.900620 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:47.900591    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 25 12:12:47.901115 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:47.901098    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mckb4\""
May 25 12:12:47.919067 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:47.919041    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mwj87"]
May 25 12:12:47.919180 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:47.919077    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jv4s8"]
May 25 12:12:47.919180 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:47.919168    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jv4s8"
May 25 12:12:47.922590 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:47.922571    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wgqbc\""
May 25 12:12:47.922710 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:47.922574    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 25 12:12:47.922710 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:47.922615    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 25 12:12:47.922710 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:47.922636    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 25 12:12:48.077239 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:48.077207    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/59e9e457-5adf-41c0-ab76-f1a65d2fd966-config-volume\") pod \"dns-default-mwj87\" (UID: \"59e9e457-5adf-41c0-ab76-f1a65d2fd966\") " pod="openshift-dns/dns-default-mwj87"
May 25 12:12:48.077239 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:48.077246    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79e13d71-6227-450f-b6fb-2f361de9fc36-cert\") pod \"ingress-canary-jv4s8\" (UID: \"79e13d71-6227-450f-b6fb-2f361de9fc36\") " pod="openshift-ingress-canary/ingress-canary-jv4s8"
May 25 12:12:48.077469 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:48.077291    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59e9e457-5adf-41c0-ab76-f1a65d2fd966-metrics-tls\") pod \"dns-default-mwj87\" (UID: \"59e9e457-5adf-41c0-ab76-f1a65d2fd966\") " pod="openshift-dns/dns-default-mwj87"
May 25 12:12:48.077469 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:48.077362    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67gbn\" (UniqueName: \"kubernetes.io/projected/59e9e457-5adf-41c0-ab76-f1a65d2fd966-kube-api-access-67gbn\") pod \"dns-default-mwj87\" (UID: \"59e9e457-5adf-41c0-ab76-f1a65d2fd966\") " pod="openshift-dns/dns-default-mwj87"
May 25 12:12:48.077469 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:48.077417    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfqmw\" (UniqueName: \"kubernetes.io/projected/79e13d71-6227-450f-b6fb-2f361de9fc36-kube-api-access-rfqmw\") pod \"ingress-canary-jv4s8\" (UID: \"79e13d71-6227-450f-b6fb-2f361de9fc36\") " pod="openshift-ingress-canary/ingress-canary-jv4s8"
May 25 12:12:48.077469 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:48.077454    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59e9e457-5adf-41c0-ab76-f1a65d2fd966-tmp-dir\") pod \"dns-default-mwj87\" (UID: \"59e9e457-5adf-41c0-ab76-f1a65d2fd966\") " pod="openshift-dns/dns-default-mwj87"
May 25 12:12:48.178664 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:48.178583    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/59e9e457-5adf-41c0-ab76-f1a65d2fd966-config-volume\") pod \"dns-default-mwj87\" (UID: \"59e9e457-5adf-41c0-ab76-f1a65d2fd966\") " pod="openshift-dns/dns-default-mwj87"
May 25 12:12:48.178664 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:48.178639    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79e13d71-6227-450f-b6fb-2f361de9fc36-cert\") pod \"ingress-canary-jv4s8\" (UID: \"79e13d71-6227-450f-b6fb-2f361de9fc36\") " pod="openshift-ingress-canary/ingress-canary-jv4s8"
May 25 12:12:48.178879 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:48.178679    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59e9e457-5adf-41c0-ab76-f1a65d2fd966-metrics-tls\") pod \"dns-default-mwj87\" (UID: \"59e9e457-5adf-41c0-ab76-f1a65d2fd966\") " pod="openshift-dns/dns-default-mwj87"
May 25 12:12:48.178879 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:48.178698    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-67gbn\" (UniqueName: \"kubernetes.io/projected/59e9e457-5adf-41c0-ab76-f1a65d2fd966-kube-api-access-67gbn\") pod \"dns-default-mwj87\" (UID: \"59e9e457-5adf-41c0-ab76-f1a65d2fd966\") " pod="openshift-dns/dns-default-mwj87"
May 25 12:12:48.178879 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:48.178723    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rfqmw\" (UniqueName: \"kubernetes.io/projected/79e13d71-6227-450f-b6fb-2f361de9fc36-kube-api-access-rfqmw\") pod \"ingress-canary-jv4s8\" (UID: \"79e13d71-6227-450f-b6fb-2f361de9fc36\") " pod="openshift-ingress-canary/ingress-canary-jv4s8"
May 25 12:12:48.178879 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:48.178752    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59e9e457-5adf-41c0-ab76-f1a65d2fd966-tmp-dir\") pod \"dns-default-mwj87\" (UID: \"59e9e457-5adf-41c0-ab76-f1a65d2fd966\") " pod="openshift-dns/dns-default-mwj87"
May 25 12:12:48.178879 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:48.178806    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 12:12:48.178879 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:48.178823    2563 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 12:12:48.178879 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:48.178882    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79e13d71-6227-450f-b6fb-2f361de9fc36-cert podName:79e13d71-6227-450f-b6fb-2f361de9fc36 nodeName:}" failed. No retries permitted until 2026-05-25 12:12:48.678864296 +0000 UTC m=+34.646255946 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/79e13d71-6227-450f-b6fb-2f361de9fc36-cert") pod "ingress-canary-jv4s8" (UID: "79e13d71-6227-450f-b6fb-2f361de9fc36") : secret "canary-serving-cert" not found
May 25 12:12:48.179154 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:48.178901    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59e9e457-5adf-41c0-ab76-f1a65d2fd966-metrics-tls podName:59e9e457-5adf-41c0-ab76-f1a65d2fd966 nodeName:}" failed. No retries permitted until 2026-05-25 12:12:48.678892638 +0000 UTC m=+34.646284260 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/59e9e457-5adf-41c0-ab76-f1a65d2fd966-metrics-tls") pod "dns-default-mwj87" (UID: "59e9e457-5adf-41c0-ab76-f1a65d2fd966") : secret "dns-default-metrics-tls" not found
May 25 12:12:48.179154 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:48.179048    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59e9e457-5adf-41c0-ab76-f1a65d2fd966-tmp-dir\") pod \"dns-default-mwj87\" (UID: \"59e9e457-5adf-41c0-ab76-f1a65d2fd966\") " pod="openshift-dns/dns-default-mwj87"
May 25 12:12:48.179256 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:48.179239    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/59e9e457-5adf-41c0-ab76-f1a65d2fd966-config-volume\") pod \"dns-default-mwj87\" (UID: \"59e9e457-5adf-41c0-ab76-f1a65d2fd966\") " pod="openshift-dns/dns-default-mwj87"
May 25 12:12:48.192657 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:48.192534    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67gbn\" (UniqueName: \"kubernetes.io/projected/59e9e457-5adf-41c0-ab76-f1a65d2fd966-kube-api-access-67gbn\") pod \"dns-default-mwj87\" (UID: \"59e9e457-5adf-41c0-ab76-f1a65d2fd966\") " pod="openshift-dns/dns-default-mwj87"
May 25 12:12:48.193377 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:48.193357    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfqmw\" (UniqueName: \"kubernetes.io/projected/79e13d71-6227-450f-b6fb-2f361de9fc36-kube-api-access-rfqmw\") pod \"ingress-canary-jv4s8\" (UID: \"79e13d71-6227-450f-b6fb-2f361de9fc36\") " pod="openshift-ingress-canary/ingress-canary-jv4s8"
May 25 12:12:48.682107 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:48.682074    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59e9e457-5adf-41c0-ab76-f1a65d2fd966-metrics-tls\") pod \"dns-default-mwj87\" (UID: \"59e9e457-5adf-41c0-ab76-f1a65d2fd966\") " pod="openshift-dns/dns-default-mwj87"
May 25 12:12:48.682645 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:48.682166    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79e13d71-6227-450f-b6fb-2f361de9fc36-cert\") pod \"ingress-canary-jv4s8\" (UID: \"79e13d71-6227-450f-b6fb-2f361de9fc36\") " pod="openshift-ingress-canary/ingress-canary-jv4s8"
May 25 12:12:48.682645 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:48.682237    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 12:12:48.682645 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:48.682284    2563 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 12:12:48.682645 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:48.682315    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59e9e457-5adf-41c0-ab76-f1a65d2fd966-metrics-tls podName:59e9e457-5adf-41c0-ab76-f1a65d2fd966 nodeName:}" failed. No retries permitted until 2026-05-25 12:12:49.682292308 +0000 UTC m=+35.649683928 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/59e9e457-5adf-41c0-ab76-f1a65d2fd966-metrics-tls") pod "dns-default-mwj87" (UID: "59e9e457-5adf-41c0-ab76-f1a65d2fd966") : secret "dns-default-metrics-tls" not found
May 25 12:12:48.682645 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:48.682347    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79e13d71-6227-450f-b6fb-2f361de9fc36-cert podName:79e13d71-6227-450f-b6fb-2f361de9fc36 nodeName:}" failed. No retries permitted until 2026-05-25 12:12:49.682329387 +0000 UTC m=+35.649721007 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/79e13d71-6227-450f-b6fb-2f361de9fc36-cert") pod "ingress-canary-jv4s8" (UID: "79e13d71-6227-450f-b6fb-2f361de9fc36") : secret "canary-serving-cert" not found
May 25 12:12:49.633380 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:49.633341    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slzvp"
May 25 12:12:49.633588 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:49.633385    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jsxz"
May 25 12:12:49.636503 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:49.636478    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v7xdz\""
May 25 12:12:49.636503 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:49.636488    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 25 12:12:49.636503 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:49.636476    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 25 12:12:49.636773 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:49.636482    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 25 12:12:49.636773 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:49.636478    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kvtsk\""
May 25 12:12:49.690248 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:49.690210    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79e13d71-6227-450f-b6fb-2f361de9fc36-cert\") pod \"ingress-canary-jv4s8\" (UID: \"79e13d71-6227-450f-b6fb-2f361de9fc36\") " pod="openshift-ingress-canary/ingress-canary-jv4s8"
May 25 12:12:49.690720 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:49.690266    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59e9e457-5adf-41c0-ab76-f1a65d2fd966-metrics-tls\") pod \"dns-default-mwj87\" (UID: \"59e9e457-5adf-41c0-ab76-f1a65d2fd966\") " pod="openshift-dns/dns-default-mwj87"
May 25 12:12:49.690720 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:49.690371    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 12:12:49.690720 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:49.690443    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59e9e457-5adf-41c0-ab76-f1a65d2fd966-metrics-tls podName:59e9e457-5adf-41c0-ab76-f1a65d2fd966 nodeName:}" failed. No retries permitted until 2026-05-25 12:12:51.690423244 +0000 UTC m=+37.657814882 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/59e9e457-5adf-41c0-ab76-f1a65d2fd966-metrics-tls") pod "dns-default-mwj87" (UID: "59e9e457-5adf-41c0-ab76-f1a65d2fd966") : secret "dns-default-metrics-tls" not found
May 25 12:12:49.690720 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:49.690371    2563 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 12:12:49.690720 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:49.690547    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79e13d71-6227-450f-b6fb-2f361de9fc36-cert podName:79e13d71-6227-450f-b6fb-2f361de9fc36 nodeName:}" failed. No retries permitted until 2026-05-25 12:12:51.690514155 +0000 UTC m=+37.657905787 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/79e13d71-6227-450f-b6fb-2f361de9fc36-cert") pod "ingress-canary-jv4s8" (UID: "79e13d71-6227-450f-b6fb-2f361de9fc36") : secret "canary-serving-cert" not found
May 25 12:12:50.822350 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:50.822318    2563 generic.go:358] "Generic (PLEG): container finished" podID="9f26ab5b-c341-495e-bac3-01d4475a795a" containerID="6105ab314f57d7003ad4ddf6c8e030b2982df5b22ee2c5038039d0cb8178f26a" exitCode=0
May 25 12:12:50.822839 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:50.822365    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-96cz9" event={"ID":"9f26ab5b-c341-495e-bac3-01d4475a795a","Type":"ContainerDied","Data":"6105ab314f57d7003ad4ddf6c8e030b2982df5b22ee2c5038039d0cb8178f26a"}
May 25 12:12:51.702804 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:51.702768    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79e13d71-6227-450f-b6fb-2f361de9fc36-cert\") pod \"ingress-canary-jv4s8\" (UID: \"79e13d71-6227-450f-b6fb-2f361de9fc36\") " pod="openshift-ingress-canary/ingress-canary-jv4s8"
May 25 12:12:51.702966 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:51.702821    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59e9e457-5adf-41c0-ab76-f1a65d2fd966-metrics-tls\") pod \"dns-default-mwj87\" (UID: \"59e9e457-5adf-41c0-ab76-f1a65d2fd966\") " pod="openshift-dns/dns-default-mwj87"
May 25 12:12:51.702966 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:51.702909    2563 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 12:12:51.702966 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:51.702958    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 12:12:51.703060 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:51.702972    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79e13d71-6227-450f-b6fb-2f361de9fc36-cert podName:79e13d71-6227-450f-b6fb-2f361de9fc36 nodeName:}" failed. No retries permitted until 2026-05-25 12:12:55.702957462 +0000 UTC m=+41.670349083 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/79e13d71-6227-450f-b6fb-2f361de9fc36-cert") pod "ingress-canary-jv4s8" (UID: "79e13d71-6227-450f-b6fb-2f361de9fc36") : secret "canary-serving-cert" not found
May 25 12:12:51.703060 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:51.703000    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59e9e457-5adf-41c0-ab76-f1a65d2fd966-metrics-tls podName:59e9e457-5adf-41c0-ab76-f1a65d2fd966 nodeName:}" failed. No retries permitted until 2026-05-25 12:12:55.702988567 +0000 UTC m=+41.670380183 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/59e9e457-5adf-41c0-ab76-f1a65d2fd966-metrics-tls") pod "dns-default-mwj87" (UID: "59e9e457-5adf-41c0-ab76-f1a65d2fd966") : secret "dns-default-metrics-tls" not found
May 25 12:12:51.827051 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:51.826969    2563 generic.go:358] "Generic (PLEG): container finished" podID="9f26ab5b-c341-495e-bac3-01d4475a795a" containerID="87516e97dfe96972f725a58cf4571b902f6f64ef61950f28bfe15de79079b868" exitCode=0
May 25 12:12:51.827051 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:51.827019    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-96cz9" event={"ID":"9f26ab5b-c341-495e-bac3-01d4475a795a","Type":"ContainerDied","Data":"87516e97dfe96972f725a58cf4571b902f6f64ef61950f28bfe15de79079b868"}
May 25 12:12:52.831055 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:52.831012    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-96cz9" event={"ID":"9f26ab5b-c341-495e-bac3-01d4475a795a","Type":"ContainerStarted","Data":"2e9a32dcf590de53ad86d05d367193a2456a15b138b6150851b64bde03daa993"}
May 25 12:12:52.853921 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:52.853877    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-96cz9" podStartSLOduration=4.328908871 podStartE2EDuration="38.853864372s" podCreationTimestamp="2026-05-25 12:12:14 +0000 UTC" firstStartedPulling="2026-05-25 12:12:15.898579282 +0000 UTC m=+1.865970906" lastFinishedPulling="2026-05-25 12:12:50.423534777 +0000 UTC m=+36.390926407" observedRunningTime="2026-05-25 12:12:52.852207949 +0000 UTC m=+38.819599588" watchObservedRunningTime="2026-05-25 12:12:52.853864372 +0000 UTC m=+38.821256010"
May 25 12:12:55.733615 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:55.733569    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79e13d71-6227-450f-b6fb-2f361de9fc36-cert\") pod \"ingress-canary-jv4s8\" (UID: \"79e13d71-6227-450f-b6fb-2f361de9fc36\") " pod="openshift-ingress-canary/ingress-canary-jv4s8"
May 25 12:12:55.734002 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:12:55.733636    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59e9e457-5adf-41c0-ab76-f1a65d2fd966-metrics-tls\") pod \"dns-default-mwj87\" (UID: \"59e9e457-5adf-41c0-ab76-f1a65d2fd966\") " pod="openshift-dns/dns-default-mwj87"
May 25 12:12:55.734002 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:55.733752    2563 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 12:12:55.734002 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:55.733799    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 12:12:55.734002 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:55.733821    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79e13d71-6227-450f-b6fb-2f361de9fc36-cert podName:79e13d71-6227-450f-b6fb-2f361de9fc36 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:03.733806048 +0000 UTC m=+49.701197665 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/79e13d71-6227-450f-b6fb-2f361de9fc36-cert") pod "ingress-canary-jv4s8" (UID: "79e13d71-6227-450f-b6fb-2f361de9fc36") : secret "canary-serving-cert" not found
May 25 12:12:55.734002 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:12:55.733850    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59e9e457-5adf-41c0-ab76-f1a65d2fd966-metrics-tls podName:59e9e457-5adf-41c0-ab76-f1a65d2fd966 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:03.733836351 +0000 UTC m=+49.701227969 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/59e9e457-5adf-41c0-ab76-f1a65d2fd966-metrics-tls") pod "dns-default-mwj87" (UID: "59e9e457-5adf-41c0-ab76-f1a65d2fd966") : secret "dns-default-metrics-tls" not found
May 25 12:13:01.592217 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:01.592186    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-5sbp7"]
May 25 12:13:01.641735 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:01.641698    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-5sbp7"]
May 25 12:13:01.641892 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:01.641840    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-5sbp7"
May 25 12:13:01.645493 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:01.645468    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
May 25 12:13:01.645493 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:01.645480    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-pz7xm\""
May 25 12:13:01.645686 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:01.645471    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
May 25 12:13:01.669351 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:01.669317    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lzzd\" (UniqueName: \"kubernetes.io/projected/9251ed75-3aac-4f81-88ab-e0af8a0ee19a-kube-api-access-6lzzd\") pod \"migrator-b985f5d89-5sbp7\" (UID: \"9251ed75-3aac-4f81-88ab-e0af8a0ee19a\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-5sbp7"
May 25 12:13:01.770276 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:01.770242    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6lzzd\" (UniqueName: \"kubernetes.io/projected/9251ed75-3aac-4f81-88ab-e0af8a0ee19a-kube-api-access-6lzzd\") pod \"migrator-b985f5d89-5sbp7\" (UID: \"9251ed75-3aac-4f81-88ab-e0af8a0ee19a\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-5sbp7"
May 25 12:13:01.780000 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:01.779980    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lzzd\" (UniqueName: \"kubernetes.io/projected/9251ed75-3aac-4f81-88ab-e0af8a0ee19a-kube-api-access-6lzzd\") pod \"migrator-b985f5d89-5sbp7\" (UID: \"9251ed75-3aac-4f81-88ab-e0af8a0ee19a\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-5sbp7"
May 25 12:13:01.950292 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:01.950188    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-5sbp7"
May 25 12:13:02.125174 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:02.125139    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-5sbp7"]
May 25 12:13:02.128707 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:13:02.128675    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9251ed75_3aac_4f81_88ab_e0af8a0ee19a.slice/crio-599d49701f2f72ac69b085e508c27e8d5f838d1b238b302ff504b5d2f96615c1 WatchSource:0}: Error finding container 599d49701f2f72ac69b085e508c27e8d5f838d1b238b302ff504b5d2f96615c1: Status 404 returned error can't find the container with id 599d49701f2f72ac69b085e508c27e8d5f838d1b238b302ff504b5d2f96615c1
May 25 12:13:02.849587 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:02.849548    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-5sbp7" event={"ID":"9251ed75-3aac-4f81-88ab-e0af8a0ee19a","Type":"ContainerStarted","Data":"599d49701f2f72ac69b085e508c27e8d5f838d1b238b302ff504b5d2f96615c1"}
May 25 12:13:02.947209 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:02.947182    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5mdjk_27b08c8c-8256-4b27-86d6-6eb05ad8bfff/dns-node-resolver/0.log"
May 25 12:13:03.357371 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:03.357338    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-w5vmq"]
May 25 12:13:03.373204 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:03.373166    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-w5vmq"]
May 25 12:13:03.373360 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:03.373222    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-67d667c7d-w5vmq"
May 25 12:13:03.375304 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:03.375279    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
May 25 12:13:03.376103 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:03.376084    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
May 25 12:13:03.376261 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:03.376118    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
May 25 12:13:03.376261 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:03.376092    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-jd597\""
May 25 12:13:03.376261 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:03.376127    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
May 25 12:13:03.381689 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:03.381668    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/63dc3c46-a0e2-4fb3-9325-bad4e7bcf23c-signing-key\") pod \"service-ca-67d667c7d-w5vmq\" (UID: \"63dc3c46-a0e2-4fb3-9325-bad4e7bcf23c\") " pod="openshift-service-ca/service-ca-67d667c7d-w5vmq"
May 25 12:13:03.381785 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:03.381714    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/63dc3c46-a0e2-4fb3-9325-bad4e7bcf23c-signing-cabundle\") pod \"service-ca-67d667c7d-w5vmq\" (UID: \"63dc3c46-a0e2-4fb3-9325-bad4e7bcf23c\") " pod="openshift-service-ca/service-ca-67d667c7d-w5vmq"
May 25 12:13:03.381785 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:03.381733    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxlvk\" (UniqueName: \"kubernetes.io/projected/63dc3c46-a0e2-4fb3-9325-bad4e7bcf23c-kube-api-access-pxlvk\") pod \"service-ca-67d667c7d-w5vmq\" (UID: \"63dc3c46-a0e2-4fb3-9325-bad4e7bcf23c\") " pod="openshift-service-ca/service-ca-67d667c7d-w5vmq"
May 25 12:13:03.482288 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:03.482251    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/63dc3c46-a0e2-4fb3-9325-bad4e7bcf23c-signing-key\") pod \"service-ca-67d667c7d-w5vmq\" (UID: \"63dc3c46-a0e2-4fb3-9325-bad4e7bcf23c\") " pod="openshift-service-ca/service-ca-67d667c7d-w5vmq"
May 25 12:13:03.482472 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:03.482305    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/63dc3c46-a0e2-4fb3-9325-bad4e7bcf23c-signing-cabundle\") pod \"service-ca-67d667c7d-w5vmq\" (UID: \"63dc3c46-a0e2-4fb3-9325-bad4e7bcf23c\") " pod="openshift-service-ca/service-ca-67d667c7d-w5vmq"
May 25 12:13:03.482472 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:03.482327    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pxlvk\" (UniqueName: \"kubernetes.io/projected/63dc3c46-a0e2-4fb3-9325-bad4e7bcf23c-kube-api-access-pxlvk\") pod \"service-ca-67d667c7d-w5vmq\" (UID: \"63dc3c46-a0e2-4fb3-9325-bad4e7bcf23c\") " pod="openshift-service-ca/service-ca-67d667c7d-w5vmq"
May 25 12:13:03.482911 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:03.482891    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/63dc3c46-a0e2-4fb3-9325-bad4e7bcf23c-signing-cabundle\") pod \"service-ca-67d667c7d-w5vmq\" (UID: \"63dc3c46-a0e2-4fb3-9325-bad4e7bcf23c\") " pod="openshift-service-ca/service-ca-67d667c7d-w5vmq"
May 25 12:13:03.484760 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:03.484737    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/63dc3c46-a0e2-4fb3-9325-bad4e7bcf23c-signing-key\") pod \"service-ca-67d667c7d-w5vmq\" (UID: \"63dc3c46-a0e2-4fb3-9325-bad4e7bcf23c\") " pod="openshift-service-ca/service-ca-67d667c7d-w5vmq"
May 25 12:13:03.491662 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:03.491635    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxlvk\" (UniqueName: \"kubernetes.io/projected/63dc3c46-a0e2-4fb3-9325-bad4e7bcf23c-kube-api-access-pxlvk\") pod \"service-ca-67d667c7d-w5vmq\" (UID: \"63dc3c46-a0e2-4fb3-9325-bad4e7bcf23c\") " pod="openshift-service-ca/service-ca-67d667c7d-w5vmq"
May 25 12:13:03.681994 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:03.681966    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-67d667c7d-w5vmq"
May 25 12:13:03.784269 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:03.784236    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79e13d71-6227-450f-b6fb-2f361de9fc36-cert\") pod \"ingress-canary-jv4s8\" (UID: \"79e13d71-6227-450f-b6fb-2f361de9fc36\") " pod="openshift-ingress-canary/ingress-canary-jv4s8"
May 25 12:13:03.784422 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:03.784287    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59e9e457-5adf-41c0-ab76-f1a65d2fd966-metrics-tls\") pod \"dns-default-mwj87\" (UID: \"59e9e457-5adf-41c0-ab76-f1a65d2fd966\") " pod="openshift-dns/dns-default-mwj87"
May 25 12:13:03.784422 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:13:03.784386    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 12:13:03.784422 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:13:03.784388    2563 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 12:13:03.784602 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:13:03.784431    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59e9e457-5adf-41c0-ab76-f1a65d2fd966-metrics-tls podName:59e9e457-5adf-41c0-ab76-f1a65d2fd966 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:19.784418143 +0000 UTC m=+65.751809760 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/59e9e457-5adf-41c0-ab76-f1a65d2fd966-metrics-tls") pod "dns-default-mwj87" (UID: "59e9e457-5adf-41c0-ab76-f1a65d2fd966") : secret "dns-default-metrics-tls" not found
May 25 12:13:03.784602 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:13:03.784444    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79e13d71-6227-450f-b6fb-2f361de9fc36-cert podName:79e13d71-6227-450f-b6fb-2f361de9fc36 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:19.784438196 +0000 UTC m=+65.751829813 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/79e13d71-6227-450f-b6fb-2f361de9fc36-cert") pod "ingress-canary-jv4s8" (UID: "79e13d71-6227-450f-b6fb-2f361de9fc36") : secret "canary-serving-cert" not found
May 25 12:13:03.804150 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:03.803961    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-w5vmq"]
May 25 12:13:03.808556 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:13:03.808506    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63dc3c46_a0e2_4fb3_9325_bad4e7bcf23c.slice/crio-f58905b9ffb440fc065cdc4ea0ccf193ca6bedd9bd75fd287b3533c55165010a WatchSource:0}: Error finding container f58905b9ffb440fc065cdc4ea0ccf193ca6bedd9bd75fd287b3533c55165010a: Status 404 returned error can't find the container with id f58905b9ffb440fc065cdc4ea0ccf193ca6bedd9bd75fd287b3533c55165010a
May 25 12:13:03.853254 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:03.853218    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-5sbp7" event={"ID":"9251ed75-3aac-4f81-88ab-e0af8a0ee19a","Type":"ContainerStarted","Data":"01b8772180994d4f5bc16a1a9d562c1d386e42fcb4844522460135b9e3bb0dd5"}
May 25 12:13:03.853254 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:03.853259    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-5sbp7" event={"ID":"9251ed75-3aac-4f81-88ab-e0af8a0ee19a","Type":"ContainerStarted","Data":"bafa15d3f54fef266a2f72d8bc3857e0aa4102a4e4be02e3a92ae8ae738d1a50"}
May 25 12:13:03.854248 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:03.854230    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-67d667c7d-w5vmq" event={"ID":"63dc3c46-a0e2-4fb3-9325-bad4e7bcf23c","Type":"ContainerStarted","Data":"f58905b9ffb440fc065cdc4ea0ccf193ca6bedd9bd75fd287b3533c55165010a"}
May 25 12:13:03.868413 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:03.868364    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-5sbp7" podStartSLOduration=1.457908891 podStartE2EDuration="2.868352134s" podCreationTimestamp="2026-05-25 12:13:01 +0000 UTC" firstStartedPulling="2026-05-25 12:13:02.130660854 +0000 UTC m=+48.098052471" lastFinishedPulling="2026-05-25 12:13:03.541104098 +0000 UTC m=+49.508495714" observedRunningTime="2026-05-25 12:13:03.867852683 +0000 UTC m=+49.835244316" watchObservedRunningTime="2026-05-25 12:13:03.868352134 +0000 UTC m=+49.835743772"
May 25 12:13:04.348911 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:04.348886    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sbtvl_4d8f666d-fd9f-4ff8-90c8-b6bfeb3be623/node-ca/0.log"
May 25 12:13:05.859996 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:05.859956    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-67d667c7d-w5vmq" event={"ID":"63dc3c46-a0e2-4fb3-9325-bad4e7bcf23c","Type":"ContainerStarted","Data":"ba2a3b0987880c0d0a84231136dee07f0e18fd7e13e799e9f08d5cd966407a68"}
May 25 12:13:05.876330 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:05.876273    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca/service-ca-67d667c7d-w5vmq" podStartSLOduration=0.95673353 podStartE2EDuration="2.876256938s" podCreationTimestamp="2026-05-25 12:13:03 +0000 UTC" firstStartedPulling="2026-05-25 12:13:03.810343478 +0000 UTC m=+49.777735098" lastFinishedPulling="2026-05-25 12:13:05.729866889 +0000 UTC m=+51.697258506" observedRunningTime="2026-05-25 12:13:05.87497901 +0000 UTC m=+51.842370651" watchObservedRunningTime="2026-05-25 12:13:05.876256938 +0000 UTC m=+51.843648574"
May 25 12:13:13.817590 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:13.817560    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cw9bb"
May 25 12:13:19.304796 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:19.304757    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80098ceb-2e1b-4378-9371-6c6150ed88b6-metrics-certs\") pod \"network-metrics-daemon-6jsxz\" (UID: \"80098ceb-2e1b-4378-9371-6c6150ed88b6\") " pod="openshift-multus/network-metrics-daemon-6jsxz"
May 25 12:13:19.307762 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:19.307744    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 25 12:13:19.317442 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:19.317414    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80098ceb-2e1b-4378-9371-6c6150ed88b6-metrics-certs\") pod \"network-metrics-daemon-6jsxz\" (UID: \"80098ceb-2e1b-4378-9371-6c6150ed88b6\") " pod="openshift-multus/network-metrics-daemon-6jsxz"
May 25 12:13:19.352612 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:19.352581    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kvtsk\""
May 25 12:13:19.361139 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:19.361117    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jsxz"
May 25 12:13:19.474193 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:19.474106    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6jsxz"]
May 25 12:13:19.477482 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:13:19.477456    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80098ceb_2e1b_4378_9371_6c6150ed88b6.slice/crio-1e5e5a160bcc3e795ca413ed9f13bdf0f20d8a4697e15bf518ab6a177a962110 WatchSource:0}: Error finding container 1e5e5a160bcc3e795ca413ed9f13bdf0f20d8a4697e15bf518ab6a177a962110: Status 404 returned error can't find the container with id 1e5e5a160bcc3e795ca413ed9f13bdf0f20d8a4697e15bf518ab6a177a962110
May 25 12:13:19.506121 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:19.506093    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hrn2t\" (UniqueName: \"kubernetes.io/projected/e54faaa8-8c11-4a93-b337-c914cc73a410-kube-api-access-hrn2t\") pod \"network-check-target-slzvp\" (UID: \"e54faaa8-8c11-4a93-b337-c914cc73a410\") " pod="openshift-network-diagnostics/network-check-target-slzvp"
May 25 12:13:19.508196 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:19.508175    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 25 12:13:19.518923 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:19.518904    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 25 12:13:19.530088 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:19.530064    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrn2t\" (UniqueName: \"kubernetes.io/projected/e54faaa8-8c11-4a93-b337-c914cc73a410-kube-api-access-hrn2t\") pod \"network-check-target-slzvp\" (UID: \"e54faaa8-8c11-4a93-b337-c914cc73a410\") " pod="openshift-network-diagnostics/network-check-target-slzvp"
May 25 12:13:19.647045 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:19.646967    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v7xdz\""
May 25 12:13:19.655551 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:19.655533    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slzvp"
May 25 12:13:19.772440 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:19.772395    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-slzvp"]
May 25 12:13:19.775991 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:13:19.775962    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode54faaa8_8c11_4a93_b337_c914cc73a410.slice/crio-7f17fbe1bf6bd159c19ea790bf6cfb5f8a2f71c325e2880c78afb5bf9be824a8 WatchSource:0}: Error finding container 7f17fbe1bf6bd159c19ea790bf6cfb5f8a2f71c325e2880c78afb5bf9be824a8: Status 404 returned error can't find the container with id 7f17fbe1bf6bd159c19ea790bf6cfb5f8a2f71c325e2880c78afb5bf9be824a8
May 25 12:13:19.808179 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:19.808148    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79e13d71-6227-450f-b6fb-2f361de9fc36-cert\") pod \"ingress-canary-jv4s8\" (UID: \"79e13d71-6227-450f-b6fb-2f361de9fc36\") " pod="openshift-ingress-canary/ingress-canary-jv4s8"
May 25 12:13:19.808328 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:19.808211    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59e9e457-5adf-41c0-ab76-f1a65d2fd966-metrics-tls\") pod \"dns-default-mwj87\" (UID: \"59e9e457-5adf-41c0-ab76-f1a65d2fd966\") " pod="openshift-dns/dns-default-mwj87"
May 25 12:13:19.810863 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:19.810840    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79e13d71-6227-450f-b6fb-2f361de9fc36-cert\") pod \"ingress-canary-jv4s8\" (UID: \"79e13d71-6227-450f-b6fb-2f361de9fc36\") " pod="openshift-ingress-canary/ingress-canary-jv4s8"
May 25 12:13:19.810962 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:19.810898    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59e9e457-5adf-41c0-ab76-f1a65d2fd966-metrics-tls\") pod \"dns-default-mwj87\" (UID: \"59e9e457-5adf-41c0-ab76-f1a65d2fd966\") " pod="openshift-dns/dns-default-mwj87"
May 25 12:13:19.888827 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:19.888781    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-slzvp" event={"ID":"e54faaa8-8c11-4a93-b337-c914cc73a410","Type":"ContainerStarted","Data":"7f17fbe1bf6bd159c19ea790bf6cfb5f8a2f71c325e2880c78afb5bf9be824a8"}
May 25 12:13:19.889977 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:19.889935    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6jsxz" event={"ID":"80098ceb-2e1b-4378-9371-6c6150ed88b6","Type":"ContainerStarted","Data":"1e5e5a160bcc3e795ca413ed9f13bdf0f20d8a4697e15bf518ab6a177a962110"}
May 25 12:13:20.010575 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:20.010541    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mckb4\""
May 25 12:13:20.019171 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:20.019140    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mwj87"
May 25 12:13:20.030728 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:20.030703    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wgqbc\""
May 25 12:13:20.039272 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:20.039251    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jv4s8"
May 25 12:13:20.168509 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:20.168477    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mwj87"]
May 25 12:13:20.180793 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:20.180768    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jv4s8"]
May 25 12:13:20.418569 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:13:20.418482    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59e9e457_5adf_41c0_ab76_f1a65d2fd966.slice/crio-9a0b4423523f65fa3dbde3bcfeea0f3f8b179d9ff2939aea6ee91442e32f8ca2 WatchSource:0}: Error finding container 9a0b4423523f65fa3dbde3bcfeea0f3f8b179d9ff2939aea6ee91442e32f8ca2: Status 404 returned error can't find the container with id 9a0b4423523f65fa3dbde3bcfeea0f3f8b179d9ff2939aea6ee91442e32f8ca2
May 25 12:13:20.419566 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:13:20.419451    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e13d71_6227_450f_b6fb_2f361de9fc36.slice/crio-580b3b6239cdde312782e8701fcf4253c12332b1291b685b3ef2c2deebe0453d WatchSource:0}: Error finding container 580b3b6239cdde312782e8701fcf4253c12332b1291b685b3ef2c2deebe0453d: Status 404 returned error can't find the container with id 580b3b6239cdde312782e8701fcf4253c12332b1291b685b3ef2c2deebe0453d
May 25 12:13:20.894128 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:20.894064    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mwj87" event={"ID":"59e9e457-5adf-41c0-ab76-f1a65d2fd966","Type":"ContainerStarted","Data":"9a0b4423523f65fa3dbde3bcfeea0f3f8b179d9ff2939aea6ee91442e32f8ca2"}
May 25 12:13:20.895488 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:20.895458    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jv4s8" event={"ID":"79e13d71-6227-450f-b6fb-2f361de9fc36","Type":"ContainerStarted","Data":"580b3b6239cdde312782e8701fcf4253c12332b1291b685b3ef2c2deebe0453d"}
May 25 12:13:20.897240 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:20.897213    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6jsxz" event={"ID":"80098ceb-2e1b-4378-9371-6c6150ed88b6","Type":"ContainerStarted","Data":"4699a052ffc4d4dfc08cb5ca975e2202eb069148d8b9f89f7177b83917bbf286"}
May 25 12:13:20.897346 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:20.897245    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6jsxz" event={"ID":"80098ceb-2e1b-4378-9371-6c6150ed88b6","Type":"ContainerStarted","Data":"4f5fd93c011ca4dc4528b7c8bd0e6282e0d7eabafa42deb95823dcd641364f8b"}
May 25 12:13:20.914505 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:20.914272    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6jsxz" podStartSLOduration=65.948971904 podStartE2EDuration="1m6.914251891s" podCreationTimestamp="2026-05-25 12:12:14 +0000 UTC" firstStartedPulling="2026-05-25 12:13:19.47934315 +0000 UTC m=+65.446734768" lastFinishedPulling="2026-05-25 12:13:20.444623125 +0000 UTC m=+66.412014755" observedRunningTime="2026-05-25 12:13:20.913407862 +0000 UTC m=+66.880799506" watchObservedRunningTime="2026-05-25 12:13:20.914251891 +0000 UTC m=+66.881643531"
May 25 12:13:23.905057 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:23.904969    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jv4s8" event={"ID":"79e13d71-6227-450f-b6fb-2f361de9fc36","Type":"ContainerStarted","Data":"a7716942b3efa9a4ef77007ed1a097c80987059625dfdba03f8eea152a12723b"}
May 25 12:13:23.906317 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:23.906291    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-slzvp" event={"ID":"e54faaa8-8c11-4a93-b337-c914cc73a410","Type":"ContainerStarted","Data":"f9fae62fa2406f4ca1458e447da501d04f3b0dded206b33124019871502fc7cf"}
May 25 12:13:23.906431 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:23.906396    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-slzvp"
May 25 12:13:23.907887 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:23.907867    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mwj87" event={"ID":"59e9e457-5adf-41c0-ab76-f1a65d2fd966","Type":"ContainerStarted","Data":"cf8ca7fb69b16d9fed22f8160e1fbc8403644f42c9b7f041558847a395de9e65"}
May 25 12:13:23.907961 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:23.907893    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mwj87" event={"ID":"59e9e457-5adf-41c0-ab76-f1a65d2fd966","Type":"ContainerStarted","Data":"610f6f32c5b6c5052a412e0bedf7bfdcfff7ef58693d8b27957099a5d9028633"}
May 25 12:13:23.908042 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:23.908028    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mwj87"
May 25 12:13:23.920975 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:23.920930    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jv4s8" podStartSLOduration=33.961579643 podStartE2EDuration="36.920918751s" podCreationTimestamp="2026-05-25 12:12:47 +0000 UTC" firstStartedPulling="2026-05-25 12:13:20.42200083 +0000 UTC m=+66.389392448" lastFinishedPulling="2026-05-25 12:13:23.381339936 +0000 UTC m=+69.348731556" observedRunningTime="2026-05-25 12:13:23.919730383 +0000 UTC m=+69.887122025" watchObservedRunningTime="2026-05-25 12:13:23.920918751 +0000 UTC m=+69.888310422"
May 25 12:13:23.934881 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:23.934837    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-slzvp" podStartSLOduration=66.335285172 podStartE2EDuration="1m9.934824403s" podCreationTimestamp="2026-05-25 12:12:14 +0000 UTC" firstStartedPulling="2026-05-25 12:13:19.777933081 +0000 UTC m=+65.745324712" lastFinishedPulling="2026-05-25 12:13:23.377472309 +0000 UTC m=+69.344863943" observedRunningTime="2026-05-25 12:13:23.934688236 +0000 UTC m=+69.902079888" watchObservedRunningTime="2026-05-25 12:13:23.934824403 +0000 UTC m=+69.902216038"
May 25 12:13:25.742392 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:25.742336    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-mwj87" podStartSLOduration=35.785902488 podStartE2EDuration="38.742319337s" podCreationTimestamp="2026-05-25 12:12:47 +0000 UTC" firstStartedPulling="2026-05-25 12:13:20.421059095 +0000 UTC m=+66.388450717" lastFinishedPulling="2026-05-25 12:13:23.377475934 +0000 UTC m=+69.344867566" observedRunningTime="2026-05-25 12:13:23.951926628 +0000 UTC m=+69.919318267" watchObservedRunningTime="2026-05-25 12:13:25.742319337 +0000 UTC m=+71.709711025"
May 25 12:13:25.742943 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:25.742912    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nd42c"]
May 25 12:13:25.769037 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:25.769001    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nd42c"]
May 25 12:13:25.769188 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:25.769099    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nd42c"
May 25 12:13:25.771807 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:25.771787    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 25 12:13:25.771807 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:25.771803    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 25 12:13:25.772001 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:25.771787    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wk9rc\""
May 25 12:13:25.772001 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:25.771871    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 25 12:13:25.772001 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:25.771923    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 25 12:13:25.832534 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:25.832482    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d8f856f7c-fk5cm"]
May 25 12:13:25.850974 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:25.850949    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ca12cb94-76a6-451c-965d-f9ef23d61c90-crio-socket\") pod \"insights-runtime-extractor-nd42c\" (UID: \"ca12cb94-76a6-451c-965d-f9ef23d61c90\") " pod="openshift-insights/insights-runtime-extractor-nd42c"
May 25 12:13:25.851125 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:25.850981    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ca12cb94-76a6-451c-965d-f9ef23d61c90-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nd42c\" (UID: \"ca12cb94-76a6-451c-965d-f9ef23d61c90\") " pod="openshift-insights/insights-runtime-extractor-nd42c"
May 25 12:13:25.851125 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:25.851028    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ca12cb94-76a6-451c-965d-f9ef23d61c90-data-volume\") pod \"insights-runtime-extractor-nd42c\" (UID: \"ca12cb94-76a6-451c-965d-f9ef23d61c90\") " pod="openshift-insights/insights-runtime-extractor-nd42c"
May 25 12:13:25.851125 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:25.851101    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjvn8\" (UniqueName: \"kubernetes.io/projected/ca12cb94-76a6-451c-965d-f9ef23d61c90-kube-api-access-vjvn8\") pod \"insights-runtime-extractor-nd42c\" (UID: \"ca12cb94-76a6-451c-965d-f9ef23d61c90\") " pod="openshift-insights/insights-runtime-extractor-nd42c"
May 25 12:13:25.851237 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:25.851131    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ca12cb94-76a6-451c-965d-f9ef23d61c90-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nd42c\" (UID: \"ca12cb94-76a6-451c-965d-f9ef23d61c90\") " pod="openshift-insights/insights-runtime-extractor-nd42c"
May 25 12:13:25.854578 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:25.854552    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d8f856f7c-fk5cm"]
May 25 12:13:25.854696 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:25.854678    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d8f856f7c-fk5cm"
May 25 12:13:25.858508 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:25.858490    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vptdj\""
May 25 12:13:25.858644 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:25.858531    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
May 25 12:13:25.858788 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:25.858762    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
May 25 12:13:25.858877 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:25.858862    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
May 25 12:13:25.864904 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:25.864887    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
May 25 12:13:25.952327 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:25.952297    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e2f19b37-4109-4d51-b6a9-4463b728ad1d-registry-certificates\") pod \"image-registry-5d8f856f7c-fk5cm\" (UID: \"e2f19b37-4109-4d51-b6a9-4463b728ad1d\") " pod="openshift-image-registry/image-registry-5d8f856f7c-fk5cm"
May 25 12:13:25.952487 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:25.952350    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ca12cb94-76a6-451c-965d-f9ef23d61c90-data-volume\") pod \"insights-runtime-extractor-nd42c\" (UID: \"ca12cb94-76a6-451c-965d-f9ef23d61c90\") " pod="openshift-insights/insights-runtime-extractor-nd42c"
May 25 12:13:25.952487 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:25.952397    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e2f19b37-4109-4d51-b6a9-4463b728ad1d-image-registry-private-configuration\") pod \"image-registry-5d8f856f7c-fk5cm\" (UID: \"e2f19b37-4109-4d51-b6a9-4463b728ad1d\") " pod="openshift-image-registry/image-registry-5d8f856f7c-fk5cm"
May 25 12:13:25.952487 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:25.952444    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vjvn8\" (UniqueName: \"kubernetes.io/projected/ca12cb94-76a6-451c-965d-f9ef23d61c90-kube-api-access-vjvn8\") pod \"insights-runtime-extractor-nd42c\" (UID: \"ca12cb94-76a6-451c-965d-f9ef23d61c90\") " pod="openshift-insights/insights-runtime-extractor-nd42c"
May 25 12:13:25.952487 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:25.952463    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ca12cb94-76a6-451c-965d-f9ef23d61c90-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nd42c\" (UID: \"ca12cb94-76a6-451c-965d-f9ef23d61c90\") " pod="openshift-insights/insights-runtime-extractor-nd42c"
May 25 12:13:25.952487 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:25.952486    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e2f19b37-4109-4d51-b6a9-4463b728ad1d-bound-sa-token\") pod \"image-registry-5d8f856f7c-fk5cm\" (UID: \"e2f19b37-4109-4d51-b6a9-4463b728ad1d\") " pod="openshift-image-registry/image-registry-5d8f856f7c-fk5cm"
May 25 12:13:25.952702 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:25.952614    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e2f19b37-4109-4d51-b6a9-4463b728ad1d-installation-pull-secrets\") pod \"image-registry-5d8f856f7c-fk5cm\" (UID: \"e2f19b37-4109-4d51-b6a9-4463b728ad1d\") " pod="openshift-image-registry/image-registry-5d8f856f7c-fk5cm"
May 25 12:13:25.952702 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:25.952637    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ca12cb94-76a6-451c-965d-f9ef23d61c90-data-volume\") pod \"insights-runtime-extractor-nd42c\" (UID: \"ca12cb94-76a6-451c-965d-f9ef23d61c90\") " pod="openshift-insights/insights-runtime-extractor-nd42c"
May 25 12:13:25.952702 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:25.952670    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ca12cb94-76a6-451c-965d-f9ef23d61c90-crio-socket\") pod \"insights-runtime-extractor-nd42c\" (UID: \"ca12cb94-76a6-451c-965d-f9ef23d61c90\") " pod="openshift-insights/insights-runtime-extractor-nd42c"
May 25 12:13:25.952702 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:25.952692    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ca12cb94-76a6-451c-965d-f9ef23d61c90-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nd42c\" (UID: \"ca12cb94-76a6-451c-965d-f9ef23d61c90\") " pod="openshift-insights/insights-runtime-extractor-nd42c"
May 25 12:13:25.952863 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:25.952710    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e2f19b37-4109-4d51-b6a9-4463b728ad1d-trusted-ca\") pod \"image-registry-5d8f856f7c-fk5cm\" (UID: \"e2f19b37-4109-4d51-b6a9-4463b728ad1d\") " pod="openshift-image-registry/image-registry-5d8f856f7c-fk5cm"
May 25 12:13:25.952863 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:25.952726    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n787k\" (UniqueName: \"kubernetes.io/projected/e2f19b37-4109-4d51-b6a9-4463b728ad1d-kube-api-access-n787k\") pod \"image-registry-5d8f856f7c-fk5cm\" (UID: \"e2f19b37-4109-4d51-b6a9-4463b728ad1d\") " pod="openshift-image-registry/image-registry-5d8f856f7c-fk5cm"
May 25 12:13:25.952863 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:25.952748    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e2f19b37-4109-4d51-b6a9-4463b728ad1d-registry-tls\") pod \"image-registry-5d8f856f7c-fk5cm\" (UID: \"e2f19b37-4109-4d51-b6a9-4463b728ad1d\") " pod="openshift-image-registry/image-registry-5d8f856f7c-fk5cm"
May 25 12:13:25.952863 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:25.952773    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e2f19b37-4109-4d51-b6a9-4463b728ad1d-ca-trust-extracted\") pod \"image-registry-5d8f856f7c-fk5cm\" (UID: \"e2f19b37-4109-4d51-b6a9-4463b728ad1d\") " pod="openshift-image-registry/image-registry-5d8f856f7c-fk5cm"
May 25 12:13:25.953089 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:25.952951    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ca12cb94-76a6-451c-965d-f9ef23d61c90-crio-socket\") pod \"insights-runtime-extractor-nd42c\" (UID: \"ca12cb94-76a6-451c-965d-f9ef23d61c90\") " pod="openshift-insights/insights-runtime-extractor-nd42c"
May 25 12:13:25.953133 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:25.953121    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ca12cb94-76a6-451c-965d-f9ef23d61c90-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nd42c\" (UID: \"ca12cb94-76a6-451c-965d-f9ef23d61c90\") " pod="openshift-insights/insights-runtime-extractor-nd42c"
May 25 12:13:25.954757 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:25.954736    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ca12cb94-76a6-451c-965d-f9ef23d61c90-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nd42c\" (UID: \"ca12cb94-76a6-451c-965d-f9ef23d61c90\") " pod="openshift-insights/insights-runtime-extractor-nd42c"
May 25 12:13:25.961799 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:25.961775    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjvn8\" (UniqueName: \"kubernetes.io/projected/ca12cb94-76a6-451c-965d-f9ef23d61c90-kube-api-access-vjvn8\") pod \"insights-runtime-extractor-nd42c\" (UID: \"ca12cb94-76a6-451c-965d-f9ef23d61c90\") " pod="openshift-insights/insights-runtime-extractor-nd42c"
May 25 12:13:26.053423 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:26.053345    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e2f19b37-4109-4d51-b6a9-4463b728ad1d-trusted-ca\") pod \"image-registry-5d8f856f7c-fk5cm\" (UID: \"e2f19b37-4109-4d51-b6a9-4463b728ad1d\") " pod="openshift-image-registry/image-registry-5d8f856f7c-fk5cm"
May 25 12:13:26.053423 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:26.053384    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-n787k\" (UniqueName: \"kubernetes.io/projected/e2f19b37-4109-4d51-b6a9-4463b728ad1d-kube-api-access-n787k\") pod \"image-registry-5d8f856f7c-fk5cm\" (UID: \"e2f19b37-4109-4d51-b6a9-4463b728ad1d\") " pod="openshift-image-registry/image-registry-5d8f856f7c-fk5cm"
May 25 12:13:26.053423 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:26.053407    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e2f19b37-4109-4d51-b6a9-4463b728ad1d-registry-tls\") pod \"image-registry-5d8f856f7c-fk5cm\" (UID: \"e2f19b37-4109-4d51-b6a9-4463b728ad1d\") " pod="openshift-image-registry/image-registry-5d8f856f7c-fk5cm"
May 25 12:13:26.053690 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:26.053667    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e2f19b37-4109-4d51-b6a9-4463b728ad1d-ca-trust-extracted\") pod \"image-registry-5d8f856f7c-fk5cm\" (UID: \"e2f19b37-4109-4d51-b6a9-4463b728ad1d\") " pod="openshift-image-registry/image-registry-5d8f856f7c-fk5cm"
May 25 12:13:26.053742 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:26.053726    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e2f19b37-4109-4d51-b6a9-4463b728ad1d-registry-certificates\") pod \"image-registry-5d8f856f7c-fk5cm\" (UID: \"e2f19b37-4109-4d51-b6a9-4463b728ad1d\") " pod="openshift-image-registry/image-registry-5d8f856f7c-fk5cm"
May 25 12:13:26.053797 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:26.053772    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e2f19b37-4109-4d51-b6a9-4463b728ad1d-image-registry-private-configuration\") pod \"image-registry-5d8f856f7c-fk5cm\" (UID: \"e2f19b37-4109-4d51-b6a9-4463b728ad1d\") " pod="openshift-image-registry/image-registry-5d8f856f7c-fk5cm"
May 25 12:13:26.053851 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:26.053822    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e2f19b37-4109-4d51-b6a9-4463b728ad1d-bound-sa-token\") pod \"image-registry-5d8f856f7c-fk5cm\" (UID: \"e2f19b37-4109-4d51-b6a9-4463b728ad1d\") " pod="openshift-image-registry/image-registry-5d8f856f7c-fk5cm"
May 25 12:13:26.053901 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:26.053857    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e2f19b37-4109-4d51-b6a9-4463b728ad1d-installation-pull-secrets\") pod \"image-registry-5d8f856f7c-fk5cm\" (UID: \"e2f19b37-4109-4d51-b6a9-4463b728ad1d\") " pod="openshift-image-registry/image-registry-5d8f856f7c-fk5cm"
May 25 12:13:26.054086 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:26.054065    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e2f19b37-4109-4d51-b6a9-4463b728ad1d-ca-trust-extracted\") pod \"image-registry-5d8f856f7c-fk5cm\" (UID: \"e2f19b37-4109-4d51-b6a9-4463b728ad1d\") " pod="openshift-image-registry/image-registry-5d8f856f7c-fk5cm"
May 25 12:13:26.054389 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:26.054364    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e2f19b37-4109-4d51-b6a9-4463b728ad1d-trusted-ca\") pod \"image-registry-5d8f856f7c-fk5cm\" (UID: \"e2f19b37-4109-4d51-b6a9-4463b728ad1d\") " pod="openshift-image-registry/image-registry-5d8f856f7c-fk5cm"
May 25 12:13:26.054468 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:26.054439    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e2f19b37-4109-4d51-b6a9-4463b728ad1d-registry-certificates\") pod \"image-registry-5d8f856f7c-fk5cm\" (UID: \"e2f19b37-4109-4d51-b6a9-4463b728ad1d\") " pod="openshift-image-registry/image-registry-5d8f856f7c-fk5cm"
May 25 12:13:26.056044 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:26.056025    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e2f19b37-4109-4d51-b6a9-4463b728ad1d-registry-tls\") pod \"image-registry-5d8f856f7c-fk5cm\" (UID: \"e2f19b37-4109-4d51-b6a9-4463b728ad1d\") " pod="openshift-image-registry/image-registry-5d8f856f7c-fk5cm"
May 25 12:13:26.056223 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:26.056197    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e2f19b37-4109-4d51-b6a9-4463b728ad1d-image-registry-private-configuration\") pod \"image-registry-5d8f856f7c-fk5cm\" (UID: \"e2f19b37-4109-4d51-b6a9-4463b728ad1d\") " pod="openshift-image-registry/image-registry-5d8f856f7c-fk5cm"
May 25 12:13:26.056223 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:26.056214    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e2f19b37-4109-4d51-b6a9-4463b728ad1d-installation-pull-secrets\") pod \"image-registry-5d8f856f7c-fk5cm\" (UID: \"e2f19b37-4109-4d51-b6a9-4463b728ad1d\") " pod="openshift-image-registry/image-registry-5d8f856f7c-fk5cm"
May 25 12:13:26.060874 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:26.060850    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n787k\" (UniqueName: \"kubernetes.io/projected/e2f19b37-4109-4d51-b6a9-4463b728ad1d-kube-api-access-n787k\") pod \"image-registry-5d8f856f7c-fk5cm\" (UID: \"e2f19b37-4109-4d51-b6a9-4463b728ad1d\") " pod="openshift-image-registry/image-registry-5d8f856f7c-fk5cm"
May 25 12:13:26.061068 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:26.061045    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e2f19b37-4109-4d51-b6a9-4463b728ad1d-bound-sa-token\") pod \"image-registry-5d8f856f7c-fk5cm\" (UID: \"e2f19b37-4109-4d51-b6a9-4463b728ad1d\") " pod="openshift-image-registry/image-registry-5d8f856f7c-fk5cm"
May 25 12:13:26.077924 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:26.077901    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nd42c"
May 25 12:13:26.163989 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:26.163961    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d8f856f7c-fk5cm"
May 25 12:13:26.194088 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:26.194057    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nd42c"]
May 25 12:13:26.198297 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:13:26.198266    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca12cb94_76a6_451c_965d_f9ef23d61c90.slice/crio-927fca7c828747c3071cddc2a76a3bf1f7a762ec11b1d823767f30e59dd0ac1b WatchSource:0}: Error finding container 927fca7c828747c3071cddc2a76a3bf1f7a762ec11b1d823767f30e59dd0ac1b: Status 404 returned error can't find the container with id 927fca7c828747c3071cddc2a76a3bf1f7a762ec11b1d823767f30e59dd0ac1b
May 25 12:13:26.290753 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:26.290718    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d8f856f7c-fk5cm"]
May 25 12:13:26.293971 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:13:26.293949    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2f19b37_4109_4d51_b6a9_4463b728ad1d.slice/crio-6b7a9c3d4ca7e50ca83fef10dfc1aaf5ad258ae041265f98a3f1e6cb7080b101 WatchSource:0}: Error finding container 6b7a9c3d4ca7e50ca83fef10dfc1aaf5ad258ae041265f98a3f1e6cb7080b101: Status 404 returned error can't find the container with id 6b7a9c3d4ca7e50ca83fef10dfc1aaf5ad258ae041265f98a3f1e6cb7080b101
May 25 12:13:26.916323 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:26.916290    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d8f856f7c-fk5cm" event={"ID":"e2f19b37-4109-4d51-b6a9-4463b728ad1d","Type":"ContainerStarted","Data":"652e83c029cefaca15f43f17eab9762012c66ce4acd862931905d34546be4eec"}
May 25 12:13:26.916735 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:26.916328    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d8f856f7c-fk5cm" event={"ID":"e2f19b37-4109-4d51-b6a9-4463b728ad1d","Type":"ContainerStarted","Data":"6b7a9c3d4ca7e50ca83fef10dfc1aaf5ad258ae041265f98a3f1e6cb7080b101"}
May 25 12:13:26.916735 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:26.916411    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d8f856f7c-fk5cm"
May 25 12:13:26.917637 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:26.917617    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nd42c" event={"ID":"ca12cb94-76a6-451c-965d-f9ef23d61c90","Type":"ContainerStarted","Data":"6af96487f447d4d15b32ec4ec5f40b2c4125d0f710f4dcca9576ed7851d12da2"}
May 25 12:13:26.917722 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:26.917645    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nd42c" event={"ID":"ca12cb94-76a6-451c-965d-f9ef23d61c90","Type":"ContainerStarted","Data":"927fca7c828747c3071cddc2a76a3bf1f7a762ec11b1d823767f30e59dd0ac1b"}
May 25 12:13:26.935875 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:26.935839    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5d8f856f7c-fk5cm" podStartSLOduration=1.935826872 podStartE2EDuration="1.935826872s" podCreationTimestamp="2026-05-25 12:13:25 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:13:26.934043953 +0000 UTC m=+72.901435591" watchObservedRunningTime="2026-05-25 12:13:26.935826872 +0000 UTC m=+72.903218510"
May 25 12:13:27.925879 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:27.925839    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nd42c" event={"ID":"ca12cb94-76a6-451c-965d-f9ef23d61c90","Type":"ContainerStarted","Data":"1a8ff1f39f6bb67acff2523da6a46c0f4ace05bbff1213a7b8682901fefe60a6"}
May 25 12:13:28.930681 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:28.930644    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nd42c" event={"ID":"ca12cb94-76a6-451c-965d-f9ef23d61c90","Type":"ContainerStarted","Data":"3525245b61839baecf7b2a3e30f575b21498bfe9e77b522a98a4b1d537a6bb10"}
May 25 12:13:28.948658 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:28.948609    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-nd42c" podStartSLOduration=2.211591316 podStartE2EDuration="3.948593404s" podCreationTimestamp="2026-05-25 12:13:25 +0000 UTC" firstStartedPulling="2026-05-25 12:13:26.340545858 +0000 UTC m=+72.307937474" lastFinishedPulling="2026-05-25 12:13:28.077547938 +0000 UTC m=+74.044939562" observedRunningTime="2026-05-25 12:13:28.947020739 +0000 UTC m=+74.914412389" watchObservedRunningTime="2026-05-25 12:13:28.948593404 +0000 UTC m=+74.915985043"
May 25 12:13:32.547219 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:32.547178    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c7fbd8d55-9bjdw"]
May 25 12:13:32.550124 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:32.550101    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c7fbd8d55-9bjdw"
May 25 12:13:32.552062 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:32.552032    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
May 25 12:13:32.552162 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:32.552131    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
May 25 12:13:32.552835 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:32.552809    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
May 25 12:13:32.552941 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:32.552913    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
May 25 12:13:32.553009 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:32.552952    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
May 25 12:13:32.553009 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:32.552974    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-l8gmm\""
May 25 12:13:32.553218 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:32.553203    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
May 25 12:13:32.553307 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:32.553292    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
May 25 12:13:32.559956 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:32.559923    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
May 25 12:13:32.562571 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:32.562548    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c7fbd8d55-9bjdw"]
May 25 12:13:32.696267 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:32.696226    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dxvd\" (UniqueName: \"kubernetes.io/projected/bd87e21b-6a7a-43e7-991a-c43ec3a112ce-kube-api-access-7dxvd\") pod \"console-6c7fbd8d55-9bjdw\" (UID: \"bd87e21b-6a7a-43e7-991a-c43ec3a112ce\") " pod="openshift-console/console-6c7fbd8d55-9bjdw"
May 25 12:13:32.696267 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:32.696269    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bd87e21b-6a7a-43e7-991a-c43ec3a112ce-console-oauth-config\") pod \"console-6c7fbd8d55-9bjdw\" (UID: \"bd87e21b-6a7a-43e7-991a-c43ec3a112ce\") " pod="openshift-console/console-6c7fbd8d55-9bjdw"
May 25 12:13:32.696482 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:32.696290    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bd87e21b-6a7a-43e7-991a-c43ec3a112ce-console-serving-cert\") pod \"console-6c7fbd8d55-9bjdw\" (UID: \"bd87e21b-6a7a-43e7-991a-c43ec3a112ce\") " pod="openshift-console/console-6c7fbd8d55-9bjdw"
May 25 12:13:32.696482 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:32.696306    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd87e21b-6a7a-43e7-991a-c43ec3a112ce-service-ca\") pod \"console-6c7fbd8d55-9bjdw\" (UID: \"bd87e21b-6a7a-43e7-991a-c43ec3a112ce\") " pod="openshift-console/console-6c7fbd8d55-9bjdw"
May 25 12:13:32.696482 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:32.696326    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bd87e21b-6a7a-43e7-991a-c43ec3a112ce-console-config\") pod \"console-6c7fbd8d55-9bjdw\" (UID: \"bd87e21b-6a7a-43e7-991a-c43ec3a112ce\") " pod="openshift-console/console-6c7fbd8d55-9bjdw"
May 25 12:13:32.696482 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:32.696383    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd87e21b-6a7a-43e7-991a-c43ec3a112ce-trusted-ca-bundle\") pod \"console-6c7fbd8d55-9bjdw\" (UID: \"bd87e21b-6a7a-43e7-991a-c43ec3a112ce\") " pod="openshift-console/console-6c7fbd8d55-9bjdw"
May 25 12:13:32.696482 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:32.696437    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bd87e21b-6a7a-43e7-991a-c43ec3a112ce-oauth-serving-cert\") pod \"console-6c7fbd8d55-9bjdw\" (UID: \"bd87e21b-6a7a-43e7-991a-c43ec3a112ce\") " pod="openshift-console/console-6c7fbd8d55-9bjdw"
May 25 12:13:32.797575 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:32.797489    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7dxvd\" (UniqueName: \"kubernetes.io/projected/bd87e21b-6a7a-43e7-991a-c43ec3a112ce-kube-api-access-7dxvd\") pod \"console-6c7fbd8d55-9bjdw\" (UID: \"bd87e21b-6a7a-43e7-991a-c43ec3a112ce\") " pod="openshift-console/console-6c7fbd8d55-9bjdw"
May 25 12:13:32.797575 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:32.797547    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bd87e21b-6a7a-43e7-991a-c43ec3a112ce-console-oauth-config\") pod \"console-6c7fbd8d55-9bjdw\" (UID: \"bd87e21b-6a7a-43e7-991a-c43ec3a112ce\") " pod="openshift-console/console-6c7fbd8d55-9bjdw"
May 25 12:13:32.797729 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:32.797579    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bd87e21b-6a7a-43e7-991a-c43ec3a112ce-console-serving-cert\") pod \"console-6c7fbd8d55-9bjdw\" (UID: \"bd87e21b-6a7a-43e7-991a-c43ec3a112ce\") " pod="openshift-console/console-6c7fbd8d55-9bjdw"
May 25 12:13:32.797729 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:32.797599    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd87e21b-6a7a-43e7-991a-c43ec3a112ce-service-ca\") pod \"console-6c7fbd8d55-9bjdw\" (UID: \"bd87e21b-6a7a-43e7-991a-c43ec3a112ce\") " pod="openshift-console/console-6c7fbd8d55-9bjdw"
May 25 12:13:32.797729 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:32.797617    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bd87e21b-6a7a-43e7-991a-c43ec3a112ce-console-config\") pod \"console-6c7fbd8d55-9bjdw\" (UID: \"bd87e21b-6a7a-43e7-991a-c43ec3a112ce\") " pod="openshift-console/console-6c7fbd8d55-9bjdw"
May 25 12:13:32.797729 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:32.797651    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd87e21b-6a7a-43e7-991a-c43ec3a112ce-trusted-ca-bundle\") pod \"console-6c7fbd8d55-9bjdw\" (UID: \"bd87e21b-6a7a-43e7-991a-c43ec3a112ce\") " pod="openshift-console/console-6c7fbd8d55-9bjdw"
May 25 12:13:32.797729 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:32.797706    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bd87e21b-6a7a-43e7-991a-c43ec3a112ce-oauth-serving-cert\") pod \"console-6c7fbd8d55-9bjdw\" (UID: \"bd87e21b-6a7a-43e7-991a-c43ec3a112ce\") " pod="openshift-console/console-6c7fbd8d55-9bjdw"
May 25 12:13:32.798350 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:32.798327    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bd87e21b-6a7a-43e7-991a-c43ec3a112ce-oauth-serving-cert\") pod \"console-6c7fbd8d55-9bjdw\" (UID: \"bd87e21b-6a7a-43e7-991a-c43ec3a112ce\") " pod="openshift-console/console-6c7fbd8d55-9bjdw"
May 25 12:13:32.798467 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:32.798446    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bd87e21b-6a7a-43e7-991a-c43ec3a112ce-console-config\") pod \"console-6c7fbd8d55-9bjdw\" (UID: \"bd87e21b-6a7a-43e7-991a-c43ec3a112ce\") " pod="openshift-console/console-6c7fbd8d55-9bjdw"
May 25 12:13:32.798507 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:32.798463    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd87e21b-6a7a-43e7-991a-c43ec3a112ce-service-ca\") pod \"console-6c7fbd8d55-9bjdw\" (UID: \"bd87e21b-6a7a-43e7-991a-c43ec3a112ce\") " pod="openshift-console/console-6c7fbd8d55-9bjdw"
May 25 12:13:32.798974 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:32.798948    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd87e21b-6a7a-43e7-991a-c43ec3a112ce-trusted-ca-bundle\") pod \"console-6c7fbd8d55-9bjdw\" (UID: \"bd87e21b-6a7a-43e7-991a-c43ec3a112ce\") " pod="openshift-console/console-6c7fbd8d55-9bjdw"
May 25 12:13:32.799916 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:32.799888    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bd87e21b-6a7a-43e7-991a-c43ec3a112ce-console-oauth-config\") pod \"console-6c7fbd8d55-9bjdw\" (UID: \"bd87e21b-6a7a-43e7-991a-c43ec3a112ce\") " pod="openshift-console/console-6c7fbd8d55-9bjdw"
May 25 12:13:32.800155 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:32.800136    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bd87e21b-6a7a-43e7-991a-c43ec3a112ce-console-serving-cert\") pod \"console-6c7fbd8d55-9bjdw\" (UID: \"bd87e21b-6a7a-43e7-991a-c43ec3a112ce\") " pod="openshift-console/console-6c7fbd8d55-9bjdw"
May 25 12:13:32.805727 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:32.805709    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dxvd\" (UniqueName: \"kubernetes.io/projected/bd87e21b-6a7a-43e7-991a-c43ec3a112ce-kube-api-access-7dxvd\") pod \"console-6c7fbd8d55-9bjdw\" (UID: \"bd87e21b-6a7a-43e7-991a-c43ec3a112ce\") " pod="openshift-console/console-6c7fbd8d55-9bjdw"
May 25 12:13:32.863701 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:32.863672    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c7fbd8d55-9bjdw"
May 25 12:13:32.989064 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:32.989034    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c7fbd8d55-9bjdw"]
May 25 12:13:32.991927 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:13:32.991895    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd87e21b_6a7a_43e7_991a_c43ec3a112ce.slice/crio-260bdbe0176ccd03268f792e6620cd305bcef87d5700036a765dcadd53f76fa9 WatchSource:0}: Error finding container 260bdbe0176ccd03268f792e6620cd305bcef87d5700036a765dcadd53f76fa9: Status 404 returned error can't find the container with id 260bdbe0176ccd03268f792e6620cd305bcef87d5700036a765dcadd53f76fa9
May 25 12:13:33.912471 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:33.912340    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mwj87"
May 25 12:13:33.944274 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:33.944242    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c7fbd8d55-9bjdw" event={"ID":"bd87e21b-6a7a-43e7-991a-c43ec3a112ce","Type":"ContainerStarted","Data":"260bdbe0176ccd03268f792e6620cd305bcef87d5700036a765dcadd53f76fa9"}
May 25 12:13:34.474372 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.474324    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-w7xgj"]
May 25 12:13:34.477724 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.477684    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-n6wtl"]
May 25 12:13:34.477857 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.477838    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-w7xgj"
May 25 12:13:34.480983 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.480959    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
May 25 12:13:34.481111 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.480969    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-hvpnz\""
May 25 12:13:34.481111 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.481025    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 25 12:13:34.481455 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.481393    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
May 25 12:13:34.481754 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.481579    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
May 25 12:13:34.481754 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.481637    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 25 12:13:34.481754 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.481720    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 25 12:13:34.482280 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.482241    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n6wtl"
May 25 12:13:34.484346 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.484160    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 25 12:13:34.484346 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.484181    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 25 12:13:34.484568 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.484551    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-htmsf\""
May 25 12:13:34.484788 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.484774    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 25 12:13:34.490768 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.488586    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-w7xgj"]
May 25 12:13:34.613421 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.613282    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/92294667-9ef8-4c97-8bc2-4cb222142abe-root\") pod \"node-exporter-n6wtl\" (UID: \"92294667-9ef8-4c97-8bc2-4cb222142abe\") " pod="openshift-monitoring/node-exporter-n6wtl"
May 25 12:13:34.613421 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.613320    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92294667-9ef8-4c97-8bc2-4cb222142abe-metrics-client-ca\") pod \"node-exporter-n6wtl\" (UID: \"92294667-9ef8-4c97-8bc2-4cb222142abe\") " pod="openshift-monitoring/node-exporter-n6wtl"
May 25 12:13:34.613421 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.613349    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/87277b1c-0afb-4fb9-8304-d9cf97d95458-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-w7xgj\" (UID: \"87277b1c-0afb-4fb9-8304-d9cf97d95458\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-w7xgj"
May 25 12:13:34.613421 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.613377    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/87277b1c-0afb-4fb9-8304-d9cf97d95458-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-w7xgj\" (UID: \"87277b1c-0afb-4fb9-8304-d9cf97d95458\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-w7xgj"
May 25 12:13:34.613421 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.613407    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/92294667-9ef8-4c97-8bc2-4cb222142abe-node-exporter-accelerators-collector-config\") pod \"node-exporter-n6wtl\" (UID: \"92294667-9ef8-4c97-8bc2-4cb222142abe\") " pod="openshift-monitoring/node-exporter-n6wtl"
May 25 12:13:34.613421 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.613431    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87277b1c-0afb-4fb9-8304-d9cf97d95458-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-w7xgj\" (UID: \"87277b1c-0afb-4fb9-8304-d9cf97d95458\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-w7xgj"
May 25 12:13:34.613888 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.613464    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl6s5\" (UniqueName: \"kubernetes.io/projected/92294667-9ef8-4c97-8bc2-4cb222142abe-kube-api-access-dl6s5\") pod \"node-exporter-n6wtl\" (UID: \"92294667-9ef8-4c97-8bc2-4cb222142abe\") " pod="openshift-monitoring/node-exporter-n6wtl"
May 25 12:13:34.613888 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.613495    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/92294667-9ef8-4c97-8bc2-4cb222142abe-node-exporter-wtmp\") pod \"node-exporter-n6wtl\" (UID: \"92294667-9ef8-4c97-8bc2-4cb222142abe\") " pod="openshift-monitoring/node-exporter-n6wtl"
May 25 12:13:34.613888 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.613529    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/87277b1c-0afb-4fb9-8304-d9cf97d95458-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-w7xgj\" (UID: \"87277b1c-0afb-4fb9-8304-d9cf97d95458\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-w7xgj"
May 25 12:13:34.613888 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.613561    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/92294667-9ef8-4c97-8bc2-4cb222142abe-node-exporter-tls\") pod \"node-exporter-n6wtl\" (UID: \"92294667-9ef8-4c97-8bc2-4cb222142abe\") " pod="openshift-monitoring/node-exporter-n6wtl"
May 25 12:13:34.613888 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.613648    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt5jw\" (UniqueName: \"kubernetes.io/projected/87277b1c-0afb-4fb9-8304-d9cf97d95458-kube-api-access-vt5jw\") pod \"kube-state-metrics-54fd6bc5c4-w7xgj\" (UID: \"87277b1c-0afb-4fb9-8304-d9cf97d95458\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-w7xgj"
May 25 12:13:34.613888 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.613728    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92294667-9ef8-4c97-8bc2-4cb222142abe-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n6wtl\" (UID: \"92294667-9ef8-4c97-8bc2-4cb222142abe\") " pod="openshift-monitoring/node-exporter-n6wtl"
May 25 12:13:34.613888 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.613763    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92294667-9ef8-4c97-8bc2-4cb222142abe-sys\") pod \"node-exporter-n6wtl\" (UID: \"92294667-9ef8-4c97-8bc2-4cb222142abe\") " pod="openshift-monitoring/node-exporter-n6wtl"
May 25 12:13:34.613888 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.613792    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/92294667-9ef8-4c97-8bc2-4cb222142abe-node-exporter-textfile\") pod \"node-exporter-n6wtl\" (UID: \"92294667-9ef8-4c97-8bc2-4cb222142abe\") " pod="openshift-monitoring/node-exporter-n6wtl"
May 25 12:13:34.613888 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.613841    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/87277b1c-0afb-4fb9-8304-d9cf97d95458-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-w7xgj\" (UID: \"87277b1c-0afb-4fb9-8304-d9cf97d95458\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-w7xgj"
May 25 12:13:34.716210 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:13:34.715928    2563 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
May 25 12:13:34.716210 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:13:34.715997    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92294667-9ef8-4c97-8bc2-4cb222142abe-node-exporter-tls podName:92294667-9ef8-4c97-8bc2-4cb222142abe nodeName:}" failed. No retries permitted until 2026-05-25 12:13:35.215975932 +0000 UTC m=+81.183367553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/92294667-9ef8-4c97-8bc2-4cb222142abe-node-exporter-tls") pod "node-exporter-n6wtl" (UID: "92294667-9ef8-4c97-8bc2-4cb222142abe") : secret "node-exporter-tls" not found
May 25 12:13:34.717344 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.717294    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/92294667-9ef8-4c97-8bc2-4cb222142abe-node-exporter-tls\") pod \"node-exporter-n6wtl\" (UID: \"92294667-9ef8-4c97-8bc2-4cb222142abe\") " pod="openshift-monitoring/node-exporter-n6wtl"
May 25 12:13:34.717750 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.717722    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vt5jw\" (UniqueName: \"kubernetes.io/projected/87277b1c-0afb-4fb9-8304-d9cf97d95458-kube-api-access-vt5jw\") pod \"kube-state-metrics-54fd6bc5c4-w7xgj\" (UID: \"87277b1c-0afb-4fb9-8304-d9cf97d95458\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-w7xgj"
May 25 12:13:34.720014 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.719965    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92294667-9ef8-4c97-8bc2-4cb222142abe-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n6wtl\" (UID: \"92294667-9ef8-4c97-8bc2-4cb222142abe\") " pod="openshift-monitoring/node-exporter-n6wtl"
May 25 12:13:34.721117 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.721078    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92294667-9ef8-4c97-8bc2-4cb222142abe-sys\") pod \"node-exporter-n6wtl\" (UID: \"92294667-9ef8-4c97-8bc2-4cb222142abe\") " pod="openshift-monitoring/node-exporter-n6wtl"
May 25 12:13:34.721353 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.721339    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92294667-9ef8-4c97-8bc2-4cb222142abe-sys\") pod \"node-exporter-n6wtl\" (UID: \"92294667-9ef8-4c97-8bc2-4cb222142abe\") " pod="openshift-monitoring/node-exporter-n6wtl"
May 25 12:13:34.722392 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.722351    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/92294667-9ef8-4c97-8bc2-4cb222142abe-node-exporter-textfile\") pod \"node-exporter-n6wtl\" (UID: \"92294667-9ef8-4c97-8bc2-4cb222142abe\") " pod="openshift-monitoring/node-exporter-n6wtl"
May 25 12:13:34.722882 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.722791    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/92294667-9ef8-4c97-8bc2-4cb222142abe-node-exporter-textfile\") pod \"node-exporter-n6wtl\" (UID: \"92294667-9ef8-4c97-8bc2-4cb222142abe\") " pod="openshift-monitoring/node-exporter-n6wtl"
May 25 12:13:34.724890 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.724433    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/87277b1c-0afb-4fb9-8304-d9cf97d95458-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-w7xgj\" (UID: \"87277b1c-0afb-4fb9-8304-d9cf97d95458\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-w7xgj"
May 25 12:13:34.739554 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.738955    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/92294667-9ef8-4c97-8bc2-4cb222142abe-root\") pod \"node-exporter-n6wtl\" (UID: \"92294667-9ef8-4c97-8bc2-4cb222142abe\") " pod="openshift-monitoring/node-exporter-n6wtl"
May 25 12:13:34.739554 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.738997    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92294667-9ef8-4c97-8bc2-4cb222142abe-metrics-client-ca\") pod \"node-exporter-n6wtl\" (UID: \"92294667-9ef8-4c97-8bc2-4cb222142abe\") " pod="openshift-monitoring/node-exporter-n6wtl"
May 25 12:13:34.739554 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.739033    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/87277b1c-0afb-4fb9-8304-d9cf97d95458-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-w7xgj\" (UID: \"87277b1c-0afb-4fb9-8304-d9cf97d95458\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-w7xgj"
May 25 12:13:34.739554 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.739063    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/87277b1c-0afb-4fb9-8304-d9cf97d95458-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-w7xgj\" (UID: \"87277b1c-0afb-4fb9-8304-d9cf97d95458\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-w7xgj"
May 25 12:13:34.739554 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.739098    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/92294667-9ef8-4c97-8bc2-4cb222142abe-node-exporter-accelerators-collector-config\") pod \"node-exporter-n6wtl\" (UID: \"92294667-9ef8-4c97-8bc2-4cb222142abe\") " pod="openshift-monitoring/node-exporter-n6wtl"
May 25 12:13:34.739554 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.739138    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87277b1c-0afb-4fb9-8304-d9cf97d95458-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-w7xgj\" (UID: \"87277b1c-0afb-4fb9-8304-d9cf97d95458\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-w7xgj"
May 25 12:13:34.739554 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.739189    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dl6s5\" (UniqueName: \"kubernetes.io/projected/92294667-9ef8-4c97-8bc2-4cb222142abe-kube-api-access-dl6s5\") pod \"node-exporter-n6wtl\" (UID: \"92294667-9ef8-4c97-8bc2-4cb222142abe\") " pod="openshift-monitoring/node-exporter-n6wtl"
May 25 12:13:34.739554 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.739236    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/92294667-9ef8-4c97-8bc2-4cb222142abe-node-exporter-wtmp\") pod \"node-exporter-n6wtl\" (UID: \"92294667-9ef8-4c97-8bc2-4cb222142abe\") " pod="openshift-monitoring/node-exporter-n6wtl"
May 25 12:13:34.739554 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.739262    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/87277b1c-0afb-4fb9-8304-d9cf97d95458-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-w7xgj\" (UID: \"87277b1c-0afb-4fb9-8304-d9cf97d95458\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-w7xgj"
May 25 12:13:34.739554 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:13:34.739406    2563 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
May 25 12:13:34.739554 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:13:34.739467    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87277b1c-0afb-4fb9-8304-d9cf97d95458-kube-state-metrics-tls podName:87277b1c-0afb-4fb9-8304-d9cf97d95458 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:35.23944069 +0000 UTC m=+81.206832312 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/87277b1c-0afb-4fb9-8304-d9cf97d95458-kube-state-metrics-tls") pod "kube-state-metrics-54fd6bc5c4-w7xgj" (UID: "87277b1c-0afb-4fb9-8304-d9cf97d95458") : secret "kube-state-metrics-tls" not found
May 25 12:13:34.740151 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.738883    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/87277b1c-0afb-4fb9-8304-d9cf97d95458-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-w7xgj\" (UID: \"87277b1c-0afb-4fb9-8304-d9cf97d95458\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-w7xgj"
May 25 12:13:34.740151 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.739772    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/92294667-9ef8-4c97-8bc2-4cb222142abe-root\") pod \"node-exporter-n6wtl\" (UID: \"92294667-9ef8-4c97-8bc2-4cb222142abe\") " pod="openshift-monitoring/node-exporter-n6wtl"
May 25 12:13:34.740282 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.740260    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92294667-9ef8-4c97-8bc2-4cb222142abe-metrics-client-ca\") pod \"node-exporter-n6wtl\" (UID: \"92294667-9ef8-4c97-8bc2-4cb222142abe\") " pod="openshift-monitoring/node-exporter-n6wtl"
May 25 12:13:34.743250 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.741589    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87277b1c-0afb-4fb9-8304-d9cf97d95458-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-w7xgj\" (UID: \"87277b1c-0afb-4fb9-8304-d9cf97d95458\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-w7xgj"
May 25 12:13:34.743250 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.741874    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/87277b1c-0afb-4fb9-8304-d9cf97d95458-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-w7xgj\" (UID: \"87277b1c-0afb-4fb9-8304-d9cf97d95458\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-w7xgj"
May 25 12:13:34.743250 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.742294    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/92294667-9ef8-4c97-8bc2-4cb222142abe-node-exporter-accelerators-collector-config\") pod \"node-exporter-n6wtl\" (UID: \"92294667-9ef8-4c97-8bc2-4cb222142abe\") " pod="openshift-monitoring/node-exporter-n6wtl"
May 25 12:13:34.743250 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.742828    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt5jw\" (UniqueName: \"kubernetes.io/projected/87277b1c-0afb-4fb9-8304-d9cf97d95458-kube-api-access-vt5jw\") pod \"kube-state-metrics-54fd6bc5c4-w7xgj\" (UID: \"87277b1c-0afb-4fb9-8304-d9cf97d95458\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-w7xgj"
May 25 12:13:34.743250 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.742966    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/92294667-9ef8-4c97-8bc2-4cb222142abe-node-exporter-wtmp\") pod \"node-exporter-n6wtl\" (UID: \"92294667-9ef8-4c97-8bc2-4cb222142abe\") " pod="openshift-monitoring/node-exporter-n6wtl"
May 25 12:13:34.754723 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.750933    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/87277b1c-0afb-4fb9-8304-d9cf97d95458-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-w7xgj\" (UID: \"87277b1c-0afb-4fb9-8304-d9cf97d95458\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-w7xgj"
May 25 12:13:34.757783 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.757671    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92294667-9ef8-4c97-8bc2-4cb222142abe-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n6wtl\" (UID: \"92294667-9ef8-4c97-8bc2-4cb222142abe\") " pod="openshift-monitoring/node-exporter-n6wtl"
May 25 12:13:34.764213 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:34.764192    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl6s5\" (UniqueName: \"kubernetes.io/projected/92294667-9ef8-4c97-8bc2-4cb222142abe-kube-api-access-dl6s5\") pod \"node-exporter-n6wtl\" (UID: \"92294667-9ef8-4c97-8bc2-4cb222142abe\") " pod="openshift-monitoring/node-exporter-n6wtl"
May 25 12:13:35.242825 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:35.242787    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/87277b1c-0afb-4fb9-8304-d9cf97d95458-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-w7xgj\" (UID: \"87277b1c-0afb-4fb9-8304-d9cf97d95458\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-w7xgj"
May 25 12:13:35.242825 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:35.242831    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/92294667-9ef8-4c97-8bc2-4cb222142abe-node-exporter-tls\") pod \"node-exporter-n6wtl\" (UID: \"92294667-9ef8-4c97-8bc2-4cb222142abe\") " pod="openshift-monitoring/node-exporter-n6wtl"
May 25 12:13:35.245633 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:35.245606    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/92294667-9ef8-4c97-8bc2-4cb222142abe-node-exporter-tls\") pod \"node-exporter-n6wtl\" (UID: \"92294667-9ef8-4c97-8bc2-4cb222142abe\") " pod="openshift-monitoring/node-exporter-n6wtl"
May 25 12:13:35.245758 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:35.245627    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/87277b1c-0afb-4fb9-8304-d9cf97d95458-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-w7xgj\" (UID: \"87277b1c-0afb-4fb9-8304-d9cf97d95458\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-w7xgj"
May 25 12:13:35.394675 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:35.394644    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-w7xgj"
May 25 12:13:35.402443 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:35.402418    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n6wtl"
May 25 12:13:35.659660 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:13:35.659626    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92294667_9ef8_4c97_8bc2_4cb222142abe.slice/crio-72dd8ac9c553abea2db3b7329db11f1d4f4a66841ed5a4324f6d9243243c430a WatchSource:0}: Error finding container 72dd8ac9c553abea2db3b7329db11f1d4f4a66841ed5a4324f6d9243243c430a: Status 404 returned error can't find the container with id 72dd8ac9c553abea2db3b7329db11f1d4f4a66841ed5a4324f6d9243243c430a
May 25 12:13:35.804889 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:35.804843    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-w7xgj"]
May 25 12:13:35.807241 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:13:35.807199    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87277b1c_0afb_4fb9_8304_d9cf97d95458.slice/crio-7c2dd5c5c0a8d62839fb15a7ba7cd57e49b42e2505a8d6d3446cde57588f8538 WatchSource:0}: Error finding container 7c2dd5c5c0a8d62839fb15a7ba7cd57e49b42e2505a8d6d3446cde57588f8538: Status 404 returned error can't find the container with id 7c2dd5c5c0a8d62839fb15a7ba7cd57e49b42e2505a8d6d3446cde57588f8538
May 25 12:13:35.951363 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:35.951282    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c7fbd8d55-9bjdw" event={"ID":"bd87e21b-6a7a-43e7-991a-c43ec3a112ce","Type":"ContainerStarted","Data":"ccce6793520befae3345c0d69066d4cfac2b40212af0ba11a0d5f37d37612243"}
May 25 12:13:35.952411 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:35.952382    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n6wtl" event={"ID":"92294667-9ef8-4c97-8bc2-4cb222142abe","Type":"ContainerStarted","Data":"72dd8ac9c553abea2db3b7329db11f1d4f4a66841ed5a4324f6d9243243c430a"}
May 25 12:13:35.953363 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:35.953344    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-w7xgj" event={"ID":"87277b1c-0afb-4fb9-8304-d9cf97d95458","Type":"ContainerStarted","Data":"7c2dd5c5c0a8d62839fb15a7ba7cd57e49b42e2505a8d6d3446cde57588f8538"}
May 25 12:13:35.969492 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:35.969452    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-6c7fbd8d55-9bjdw" podStartSLOduration=1.260377027 podStartE2EDuration="3.969440052s" podCreationTimestamp="2026-05-25 12:13:32 +0000 UTC" firstStartedPulling="2026-05-25 12:13:32.993693485 +0000 UTC m=+78.961085101" lastFinishedPulling="2026-05-25 12:13:35.702756492 +0000 UTC m=+81.670148126" observedRunningTime="2026-05-25 12:13:35.968167412 +0000 UTC m=+81.935559063" watchObservedRunningTime="2026-05-25 12:13:35.969440052 +0000 UTC m=+81.936831691"
May 25 12:13:36.957975 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:36.957803    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n6wtl" event={"ID":"92294667-9ef8-4c97-8bc2-4cb222142abe","Type":"ContainerStarted","Data":"99eb02efcc1b7ffb665d47dce7b4c13c392a3da30e53660674372b6afd75220d"}
May 25 12:13:36.960047 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:36.960002    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-w7xgj" event={"ID":"87277b1c-0afb-4fb9-8304-d9cf97d95458","Type":"ContainerStarted","Data":"43069ae2c15647263ecaa350c80d72bd2614c67bee1baa42210eceb9eeb35799"}
May 25 12:13:37.964565 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:37.964511    2563 generic.go:358] "Generic (PLEG): container finished" podID="92294667-9ef8-4c97-8bc2-4cb222142abe" containerID="99eb02efcc1b7ffb665d47dce7b4c13c392a3da30e53660674372b6afd75220d" exitCode=0
May 25 12:13:37.965027 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:37.964597    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n6wtl" event={"ID":"92294667-9ef8-4c97-8bc2-4cb222142abe","Type":"ContainerDied","Data":"99eb02efcc1b7ffb665d47dce7b4c13c392a3da30e53660674372b6afd75220d"}
May 25 12:13:37.966573 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:37.966548    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-w7xgj" event={"ID":"87277b1c-0afb-4fb9-8304-d9cf97d95458","Type":"ContainerStarted","Data":"b5f4ac6619c5ff944d08f9cc55972b71288d76d07a529960b504f9c6c6b646a1"}
May 25 12:13:37.966662 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:37.966581    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-w7xgj" event={"ID":"87277b1c-0afb-4fb9-8304-d9cf97d95458","Type":"ContainerStarted","Data":"81393e0b74447f078f71208a9447038d938720ef5e16bbef4ee8c78897e466e1"}
May 25 12:13:37.997733 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:37.997682    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-w7xgj" podStartSLOduration=2.957573279 podStartE2EDuration="3.997664913s" podCreationTimestamp="2026-05-25 12:13:34 +0000 UTC" firstStartedPulling="2026-05-25 12:13:35.809234021 +0000 UTC m=+81.776625638" lastFinishedPulling="2026-05-25 12:13:36.849325639 +0000 UTC m=+82.816717272" observedRunningTime="2026-05-25 12:13:37.996841984 +0000 UTC m=+83.964233624" watchObservedRunningTime="2026-05-25 12:13:37.997664913 +0000 UTC m=+83.965056552"
May 25 12:13:38.971224 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:38.971188    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n6wtl" event={"ID":"92294667-9ef8-4c97-8bc2-4cb222142abe","Type":"ContainerStarted","Data":"1581c5ac228a31468b14d8e9283e625de33bcdb7dabc428d327949eb9323f22a"}
May 25 12:13:38.971224 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:38.971228    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n6wtl" event={"ID":"92294667-9ef8-4c97-8bc2-4cb222142abe","Type":"ContainerStarted","Data":"6138c5d2288e5f5f31927061124381536d6167d8a97835a9f20c4bb7f90631e6"}
May 25 12:13:39.249701 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:39.249633    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-n6wtl" podStartSLOduration=4.108640474 podStartE2EDuration="5.249614851s" podCreationTimestamp="2026-05-25 12:13:34 +0000 UTC" firstStartedPulling="2026-05-25 12:13:35.661856948 +0000 UTC m=+81.629248582" lastFinishedPulling="2026-05-25 12:13:36.802831328 +0000 UTC m=+82.770222959" observedRunningTime="2026-05-25 12:13:38.988191492 +0000 UTC m=+84.955583131" watchObservedRunningTime="2026-05-25 12:13:39.249614851 +0000 UTC m=+85.217006531"
May 25 12:13:39.250974 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:39.250949    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-c6x4s"]
May 25 12:13:39.254413 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:39.254398    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-c6x4s"
May 25 12:13:39.256208 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:39.256187    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
May 25 12:13:39.256312 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:39.256295    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"default-dockercfg-jpfh4\""
May 25 12:13:39.262428 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:39.262409    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-c6x4s"]
May 25 12:13:39.382987 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:39.382955    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/723fcb86-a890-4564-a706-997b76626043-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-c6x4s\" (UID: \"723fcb86-a890-4564-a706-997b76626043\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-c6x4s"
May 25 12:13:39.483859 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:39.483821    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/723fcb86-a890-4564-a706-997b76626043-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-c6x4s\" (UID: \"723fcb86-a890-4564-a706-997b76626043\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-c6x4s"
May 25 12:13:39.486112 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:39.486093    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/723fcb86-a890-4564-a706-997b76626043-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-c6x4s\" (UID: \"723fcb86-a890-4564-a706-997b76626043\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-c6x4s"
May 25 12:13:39.563380 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:39.563290    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-c6x4s"
May 25 12:13:39.688613 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:39.688559    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-c6x4s"]
May 25 12:13:39.692043 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:13:39.692009    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod723fcb86_a890_4564_a706_997b76626043.slice/crio-55cb61eb3d7deb7ef77dd71cea96014861ec8ec30703d706c1584f2dd6fc9e73 WatchSource:0}: Error finding container 55cb61eb3d7deb7ef77dd71cea96014861ec8ec30703d706c1584f2dd6fc9e73: Status 404 returned error can't find the container with id 55cb61eb3d7deb7ef77dd71cea96014861ec8ec30703d706c1584f2dd6fc9e73
May 25 12:13:39.983400 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:39.983296    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-c6x4s" event={"ID":"723fcb86-a890-4564-a706-997b76626043","Type":"ContainerStarted","Data":"55cb61eb3d7deb7ef77dd71cea96014861ec8ec30703d706c1584f2dd6fc9e73"}
May 25 12:13:40.690011 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.689975    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 25 12:13:40.693446 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.693421    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.695363 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.695341    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
May 25 12:13:40.695482 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.695351    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
May 25 12:13:40.695660 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.695644    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
May 25 12:13:40.695740 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.695667    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
May 25 12:13:40.695740 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.695675    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
May 25 12:13:40.696290 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.696274    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
May 25 12:13:40.696375 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.696289    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
May 25 12:13:40.696375 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.696303    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-gjsxk\""
May 25 12:13:40.696537 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.696505    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
May 25 12:13:40.696606 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.696537    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7l0efrbh8tors\""
May 25 12:13:40.696784 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.696765    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
May 25 12:13:40.696784 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.696776    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
May 25 12:13:40.697073 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.697057    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
May 25 12:13:40.697135 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.697089    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
May 25 12:13:40.699089 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.699072    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
May 25 12:13:40.710301 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.710279    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 25 12:13:40.794039 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.794007    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa43e91d-4493-4558-aaff-84f2462c2876-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.794195 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.794044    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fa43e91d-4493-4558-aaff-84f2462c2876-config-out\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.794195 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.794131    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.794195 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.794162    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa43e91d-4493-4558-aaff-84f2462c2876-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.794195 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.794182    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fa43e91d-4493-4558-aaff-84f2462c2876-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.794371 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.794202    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa43e91d-4493-4558-aaff-84f2462c2876-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.794371 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.794229    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.794371 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.794254    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fa43e91d-4493-4558-aaff-84f2462c2876-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.794371 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.794315    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.794371 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.794350    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fa43e91d-4493-4558-aaff-84f2462c2876-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.794580 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.794446    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-config\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.794580 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.794475    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.794580 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.794530    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.794580 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.794569    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa43e91d-4493-4558-aaff-84f2462c2876-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.794736 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.794605    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.794736 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.794634    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.794736 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.794659    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-web-config\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.794736 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.794681    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msndk\" (UniqueName: \"kubernetes.io/projected/fa43e91d-4493-4558-aaff-84f2462c2876-kube-api-access-msndk\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.895911 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.895874    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fa43e91d-4493-4558-aaff-84f2462c2876-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.896096 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.895918    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-config\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.896096 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.895937    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.896096 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.895965    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.896096 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.895992    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa43e91d-4493-4558-aaff-84f2462c2876-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.896096 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.896019    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.896096 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.896043    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.896096 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.896074    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-web-config\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.896096 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.896098    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-msndk\" (UniqueName: \"kubernetes.io/projected/fa43e91d-4493-4558-aaff-84f2462c2876-kube-api-access-msndk\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.896466 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.896147    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa43e91d-4493-4558-aaff-84f2462c2876-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.896466 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.896176    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fa43e91d-4493-4558-aaff-84f2462c2876-config-out\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.896466 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.896210    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.896466 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.896237    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa43e91d-4493-4558-aaff-84f2462c2876-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.896466 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.896262    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fa43e91d-4493-4558-aaff-84f2462c2876-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.896466 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.896284    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa43e91d-4493-4558-aaff-84f2462c2876-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.896466 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.896317    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.896466 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.896345    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fa43e91d-4493-4558-aaff-84f2462c2876-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.896466 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.896373    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.897843 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.897565    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa43e91d-4493-4558-aaff-84f2462c2876-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.897843 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.897657    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa43e91d-4493-4558-aaff-84f2462c2876-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.898698 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.898302    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fa43e91d-4493-4558-aaff-84f2462c2876-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.899703 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.899046    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa43e91d-4493-4558-aaff-84f2462c2876-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.899703 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.899607    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.899703 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.899645    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fa43e91d-4493-4558-aaff-84f2462c2876-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.899914 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.899836    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa43e91d-4493-4558-aaff-84f2462c2876-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.899971 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.899909    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.900115 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.900071    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-config\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.900557 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.900501    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.901111 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.901063    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.901563 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.901540    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fa43e91d-4493-4558-aaff-84f2462c2876-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.901674 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.901594    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.902145 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.902119    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-web-config\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.902242 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.902183    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.902304 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.902278    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fa43e91d-4493-4558-aaff-84f2462c2876-config-out\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.902354 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.902311    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:40.906865 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:40.906846    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msndk\" (UniqueName: \"kubernetes.io/projected/fa43e91d-4493-4558-aaff-84f2462c2876-kube-api-access-msndk\") pod \"prometheus-k8s-0\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:41.003296 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:41.003267    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:41.455992 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:41.455961    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 25 12:13:41.458530 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:13:41.458492    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa43e91d_4493_4558_aaff_84f2462c2876.slice/crio-414a2d9ed1ec42d25f1533d9d76cbb944c193e451f1409ff1182bef0e08e1c8a WatchSource:0}: Error finding container 414a2d9ed1ec42d25f1533d9d76cbb944c193e451f1409ff1182bef0e08e1c8a: Status 404 returned error can't find the container with id 414a2d9ed1ec42d25f1533d9d76cbb944c193e451f1409ff1182bef0e08e1c8a
May 25 12:13:41.990886 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:41.990834    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-c6x4s" event={"ID":"723fcb86-a890-4564-a706-997b76626043","Type":"ContainerStarted","Data":"f5a96fd756552aad8068e357a21808415332359fc7a18d719d5b68be5678f65e"}
May 25 12:13:41.991153 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:41.991125    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-c6x4s"
May 25 12:13:41.992279 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:41.992226    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fa43e91d-4493-4558-aaff-84f2462c2876","Type":"ContainerStarted","Data":"414a2d9ed1ec42d25f1533d9d76cbb944c193e451f1409ff1182bef0e08e1c8a"}
May 25 12:13:41.996502 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:41.996479    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-c6x4s"
May 25 12:13:42.007691 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:42.007637    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-665c9c8584-c6x4s" podStartSLOduration=1.324486596 podStartE2EDuration="3.007620054s" podCreationTimestamp="2026-05-25 12:13:39 +0000 UTC" firstStartedPulling="2026-05-25 12:13:39.694444764 +0000 UTC m=+85.661836381" lastFinishedPulling="2026-05-25 12:13:41.377578222 +0000 UTC m=+87.344969839" observedRunningTime="2026-05-25 12:13:42.006670494 +0000 UTC m=+87.974062135" watchObservedRunningTime="2026-05-25 12:13:42.007620054 +0000 UTC m=+87.975011694"
May 25 12:13:42.864106 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:42.864069    2563 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c7fbd8d55-9bjdw"
May 25 12:13:42.864106 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:42.864114    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c7fbd8d55-9bjdw"
May 25 12:13:42.868709 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:42.868685    2563 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c7fbd8d55-9bjdw"
May 25 12:13:42.996278 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:42.996246    2563 generic.go:358] "Generic (PLEG): container finished" podID="fa43e91d-4493-4558-aaff-84f2462c2876" containerID="87971080e0ef4b289bd79b5650702298c2b11ae0348d084fce4125d7518a91b5" exitCode=0
May 25 12:13:42.996468 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:42.996336    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fa43e91d-4493-4558-aaff-84f2462c2876","Type":"ContainerDied","Data":"87971080e0ef4b289bd79b5650702298c2b11ae0348d084fce4125d7518a91b5"}
May 25 12:13:43.000464 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:43.000443    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c7fbd8d55-9bjdw"
May 25 12:13:46.006457 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:46.006416    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fa43e91d-4493-4558-aaff-84f2462c2876","Type":"ContainerStarted","Data":"20e72474385ed85b021cafcfc9be78b5af2567958dd56e962eecf5f65073b6b4"}
May 25 12:13:46.006457 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:46.006453    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fa43e91d-4493-4558-aaff-84f2462c2876","Type":"ContainerStarted","Data":"6e96b4d89bef1e25eac6763faeb51e4103e7aeb6387269290a39d359788ba7d4"}
May 25 12:13:47.930449 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:47.930420    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5d8f856f7c-fk5cm"
May 25 12:13:48.017143 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:48.017115    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fa43e91d-4493-4558-aaff-84f2462c2876","Type":"ContainerStarted","Data":"a2966fe1b4ecaff8387ce96bf1b8bede2ff07f4e74325c3ef2b6592c21e0910b"}
May 25 12:13:48.017143 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:48.017147    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fa43e91d-4493-4558-aaff-84f2462c2876","Type":"ContainerStarted","Data":"68f828b5e8c20f1b07abf5556c81b0c57913b685d1e26b83764b99b4d40c3b48"}
May 25 12:13:48.017340 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:48.017161    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fa43e91d-4493-4558-aaff-84f2462c2876","Type":"ContainerStarted","Data":"16e8df2fe462e4e5ecefaf7e1a19b366a10ba7fac67879ab5efcaab1659ccefb"}
May 25 12:13:48.017340 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:48.017170    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fa43e91d-4493-4558-aaff-84f2462c2876","Type":"ContainerStarted","Data":"4e5b814d48c1df8791ac6d1e123ad24247aeb7059446042d88aa12687116c3cf"}
May 25 12:13:48.047212 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:48.047153    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.118626786 podStartE2EDuration="8.047139371s" podCreationTimestamp="2026-05-25 12:13:40 +0000 UTC" firstStartedPulling="2026-05-25 12:13:41.462601503 +0000 UTC m=+87.429993122" lastFinishedPulling="2026-05-25 12:13:47.391114085 +0000 UTC m=+93.358505707" observedRunningTime="2026-05-25 12:13:48.045681308 +0000 UTC m=+94.013072947" watchObservedRunningTime="2026-05-25 12:13:48.047139371 +0000 UTC m=+94.014531009"
May 25 12:13:51.004160 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:51.004123    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:13:52.358359 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:52.358323    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-6947f9fd8d-gmknf"]
May 25 12:13:52.362676 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:52.362653    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6947f9fd8d-gmknf"
May 25 12:13:52.372981 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:52.372957    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6947f9fd8d-gmknf"]
May 25 12:13:52.389220 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:52.389197    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa-console-oauth-config\") pod \"console-6947f9fd8d-gmknf\" (UID: \"2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa\") " pod="openshift-console/console-6947f9fd8d-gmknf"
May 25 12:13:52.389338 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:52.389233    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa-oauth-serving-cert\") pod \"console-6947f9fd8d-gmknf\" (UID: \"2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa\") " pod="openshift-console/console-6947f9fd8d-gmknf"
May 25 12:13:52.389338 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:52.389257    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5gff\" (UniqueName: \"kubernetes.io/projected/2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa-kube-api-access-j5gff\") pod \"console-6947f9fd8d-gmknf\" (UID: \"2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa\") " pod="openshift-console/console-6947f9fd8d-gmknf"
May 25 12:13:52.389338 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:52.389278    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa-console-serving-cert\") pod \"console-6947f9fd8d-gmknf\" (UID: \"2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa\") " pod="openshift-console/console-6947f9fd8d-gmknf"
May 25 12:13:52.389338 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:52.389301    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa-trusted-ca-bundle\") pod \"console-6947f9fd8d-gmknf\" (UID: \"2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa\") " pod="openshift-console/console-6947f9fd8d-gmknf"
May 25 12:13:52.389603 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:52.389370    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa-service-ca\") pod \"console-6947f9fd8d-gmknf\" (UID: \"2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa\") " pod="openshift-console/console-6947f9fd8d-gmknf"
May 25 12:13:52.389603 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:52.389406    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa-console-config\") pod \"console-6947f9fd8d-gmknf\" (UID: \"2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa\") " pod="openshift-console/console-6947f9fd8d-gmknf"
May 25 12:13:52.490762 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:52.490730    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa-console-oauth-config\") pod \"console-6947f9fd8d-gmknf\" (UID: \"2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa\") " pod="openshift-console/console-6947f9fd8d-gmknf"
May 25 12:13:52.490762 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:52.490764    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa-oauth-serving-cert\") pod \"console-6947f9fd8d-gmknf\" (UID: \"2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa\") " pod="openshift-console/console-6947f9fd8d-gmknf"
May 25 12:13:52.490979 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:52.490786    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-j5gff\" (UniqueName: \"kubernetes.io/projected/2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa-kube-api-access-j5gff\") pod \"console-6947f9fd8d-gmknf\" (UID: \"2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa\") " pod="openshift-console/console-6947f9fd8d-gmknf"
May 25 12:13:52.490979 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:52.490805    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa-console-serving-cert\") pod \"console-6947f9fd8d-gmknf\" (UID: \"2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa\") " pod="openshift-console/console-6947f9fd8d-gmknf"
May 25 12:13:52.490979 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:52.490821    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa-trusted-ca-bundle\") pod \"console-6947f9fd8d-gmknf\" (UID: \"2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa\") " pod="openshift-console/console-6947f9fd8d-gmknf"
May 25 12:13:52.490979 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:52.490850    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa-service-ca\") pod \"console-6947f9fd8d-gmknf\" (UID: \"2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa\") " pod="openshift-console/console-6947f9fd8d-gmknf"
May 25 12:13:52.490979 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:52.490905    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa-console-config\") pod \"console-6947f9fd8d-gmknf\" (UID: \"2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa\") " pod="openshift-console/console-6947f9fd8d-gmknf"
May 25 12:13:52.491702 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:52.491676    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa-oauth-serving-cert\") pod \"console-6947f9fd8d-gmknf\" (UID: \"2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa\") " pod="openshift-console/console-6947f9fd8d-gmknf"
May 25 12:13:52.491826 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:52.491791    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa-trusted-ca-bundle\") pod \"console-6947f9fd8d-gmknf\" (UID: \"2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa\") " pod="openshift-console/console-6947f9fd8d-gmknf"
May 25 12:13:52.491880 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:52.491816    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa-service-ca\") pod \"console-6947f9fd8d-gmknf\" (UID: \"2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa\") " pod="openshift-console/console-6947f9fd8d-gmknf"
May 25 12:13:52.491880 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:52.491795    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa-console-config\") pod \"console-6947f9fd8d-gmknf\" (UID: \"2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa\") " pod="openshift-console/console-6947f9fd8d-gmknf"
May 25 12:13:52.493230 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:52.493203    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa-console-oauth-config\") pod \"console-6947f9fd8d-gmknf\" (UID: \"2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa\") " pod="openshift-console/console-6947f9fd8d-gmknf"
May 25 12:13:52.493330 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:52.493313    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa-console-serving-cert\") pod \"console-6947f9fd8d-gmknf\" (UID: \"2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa\") " pod="openshift-console/console-6947f9fd8d-gmknf"
May 25 12:13:52.499106 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:52.499088    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5gff\" (UniqueName: \"kubernetes.io/projected/2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa-kube-api-access-j5gff\") pod \"console-6947f9fd8d-gmknf\" (UID: \"2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa\") " pod="openshift-console/console-6947f9fd8d-gmknf"
May 25 12:13:52.671900 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:52.671816    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6947f9fd8d-gmknf"
May 25 12:13:52.798134 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:52.798099    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6947f9fd8d-gmknf"]
May 25 12:13:52.802560 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:13:52.802508    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bd84ee5_0eef_44fb_b9d4_69a807f2c9aa.slice/crio-4fdac02510b38ca85cf9b3f347eb521f8c48e60b53ad73fb1c2fc9be8c6f5d8e WatchSource:0}: Error finding container 4fdac02510b38ca85cf9b3f347eb521f8c48e60b53ad73fb1c2fc9be8c6f5d8e: Status 404 returned error can't find the container with id 4fdac02510b38ca85cf9b3f347eb521f8c48e60b53ad73fb1c2fc9be8c6f5d8e
May 25 12:13:53.032022 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:53.031985    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6947f9fd8d-gmknf" event={"ID":"2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa","Type":"ContainerStarted","Data":"4991df51a4a37c0a41141f713819cb4c21dbfc29e56694054fc85e8e5e1c0bb4"}
May 25 12:13:53.032189 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:53.032030    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6947f9fd8d-gmknf" event={"ID":"2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa","Type":"ContainerStarted","Data":"4fdac02510b38ca85cf9b3f347eb521f8c48e60b53ad73fb1c2fc9be8c6f5d8e"}
May 25 12:13:53.050003 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:53.049957    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-6947f9fd8d-gmknf" podStartSLOduration=1.049941374 podStartE2EDuration="1.049941374s" podCreationTimestamp="2026-05-25 12:13:52 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:13:53.048882616 +0000 UTC m=+99.016274255" watchObservedRunningTime="2026-05-25 12:13:53.049941374 +0000 UTC m=+99.017333013"
May 25 12:13:54.912380 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:13:54.912351    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-slzvp"
May 25 12:14:02.672411 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:02.672358    2563 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6947f9fd8d-gmknf"
May 25 12:14:02.672949 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:02.672443    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6947f9fd8d-gmknf"
May 25 12:14:02.677226 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:02.677205    2563 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6947f9fd8d-gmknf"
May 25 12:14:03.066707 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:03.066675    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6947f9fd8d-gmknf"
May 25 12:14:03.118468 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:03.118391    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c7fbd8d55-9bjdw"]
May 25 12:14:28.137472 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:28.137405    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-6c7fbd8d55-9bjdw" podUID="bd87e21b-6a7a-43e7-991a-c43ec3a112ce" containerName="console" containerID="cri-o://ccce6793520befae3345c0d69066d4cfac2b40212af0ba11a0d5f37d37612243" gracePeriod=15
May 25 12:14:28.374696 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:28.374675    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6c7fbd8d55-9bjdw_bd87e21b-6a7a-43e7-991a-c43ec3a112ce/console/0.log"
May 25 12:14:28.374813 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:28.374755    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c7fbd8d55-9bjdw"
May 25 12:14:28.488222 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:28.488123    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd87e21b-6a7a-43e7-991a-c43ec3a112ce-service-ca\") pod \"bd87e21b-6a7a-43e7-991a-c43ec3a112ce\" (UID: \"bd87e21b-6a7a-43e7-991a-c43ec3a112ce\") "
May 25 12:14:28.488222 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:28.488164    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bd87e21b-6a7a-43e7-991a-c43ec3a112ce-console-config\") pod \"bd87e21b-6a7a-43e7-991a-c43ec3a112ce\" (UID: \"bd87e21b-6a7a-43e7-991a-c43ec3a112ce\") "
May 25 12:14:28.488455 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:28.488252    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd87e21b-6a7a-43e7-991a-c43ec3a112ce-trusted-ca-bundle\") pod \"bd87e21b-6a7a-43e7-991a-c43ec3a112ce\" (UID: \"bd87e21b-6a7a-43e7-991a-c43ec3a112ce\") "
May 25 12:14:28.488455 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:28.488283    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bd87e21b-6a7a-43e7-991a-c43ec3a112ce-console-oauth-config\") pod \"bd87e21b-6a7a-43e7-991a-c43ec3a112ce\" (UID: \"bd87e21b-6a7a-43e7-991a-c43ec3a112ce\") "
May 25 12:14:28.488455 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:28.488329    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bd87e21b-6a7a-43e7-991a-c43ec3a112ce-console-serving-cert\") pod \"bd87e21b-6a7a-43e7-991a-c43ec3a112ce\" (UID: \"bd87e21b-6a7a-43e7-991a-c43ec3a112ce\") "
May 25 12:14:28.488455 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:28.488352    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bd87e21b-6a7a-43e7-991a-c43ec3a112ce-oauth-serving-cert\") pod \"bd87e21b-6a7a-43e7-991a-c43ec3a112ce\" (UID: \"bd87e21b-6a7a-43e7-991a-c43ec3a112ce\") "
May 25 12:14:28.488455 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:28.488393    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dxvd\" (UniqueName: \"kubernetes.io/projected/bd87e21b-6a7a-43e7-991a-c43ec3a112ce-kube-api-access-7dxvd\") pod \"bd87e21b-6a7a-43e7-991a-c43ec3a112ce\" (UID: \"bd87e21b-6a7a-43e7-991a-c43ec3a112ce\") "
May 25 12:14:28.488745 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:28.488661    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd87e21b-6a7a-43e7-991a-c43ec3a112ce-service-ca" (OuterVolumeSpecName: "service-ca") pod "bd87e21b-6a7a-43e7-991a-c43ec3a112ce" (UID: "bd87e21b-6a7a-43e7-991a-c43ec3a112ce"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:14:28.488745 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:28.488730    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd87e21b-6a7a-43e7-991a-c43ec3a112ce-console-config" (OuterVolumeSpecName: "console-config") pod "bd87e21b-6a7a-43e7-991a-c43ec3a112ce" (UID: "bd87e21b-6a7a-43e7-991a-c43ec3a112ce"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:14:28.488830 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:28.488750    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd87e21b-6a7a-43e7-991a-c43ec3a112ce-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "bd87e21b-6a7a-43e7-991a-c43ec3a112ce" (UID: "bd87e21b-6a7a-43e7-991a-c43ec3a112ce"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:14:28.488830 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:28.488807    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd87e21b-6a7a-43e7-991a-c43ec3a112ce-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bd87e21b-6a7a-43e7-991a-c43ec3a112ce" (UID: "bd87e21b-6a7a-43e7-991a-c43ec3a112ce"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:14:28.490729 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:28.490703    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd87e21b-6a7a-43e7-991a-c43ec3a112ce-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bd87e21b-6a7a-43e7-991a-c43ec3a112ce" (UID: "bd87e21b-6a7a-43e7-991a-c43ec3a112ce"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:14:28.490729 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:28.490714    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd87e21b-6a7a-43e7-991a-c43ec3a112ce-kube-api-access-7dxvd" (OuterVolumeSpecName: "kube-api-access-7dxvd") pod "bd87e21b-6a7a-43e7-991a-c43ec3a112ce" (UID: "bd87e21b-6a7a-43e7-991a-c43ec3a112ce"). InnerVolumeSpecName "kube-api-access-7dxvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:14:28.490836 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:28.490793    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd87e21b-6a7a-43e7-991a-c43ec3a112ce-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bd87e21b-6a7a-43e7-991a-c43ec3a112ce" (UID: "bd87e21b-6a7a-43e7-991a-c43ec3a112ce"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:14:28.589026 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:28.588987    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bd87e21b-6a7a-43e7-991a-c43ec3a112ce-console-serving-cert\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:14:28.589026 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:28.589017    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bd87e21b-6a7a-43e7-991a-c43ec3a112ce-oauth-serving-cert\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:14:28.589026 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:28.589027    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dxvd\" (UniqueName: \"kubernetes.io/projected/bd87e21b-6a7a-43e7-991a-c43ec3a112ce-kube-api-access-7dxvd\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:14:28.589026 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:28.589037    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd87e21b-6a7a-43e7-991a-c43ec3a112ce-service-ca\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:14:28.589284 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:28.589046    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bd87e21b-6a7a-43e7-991a-c43ec3a112ce-console-config\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:14:28.589284 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:28.589054    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd87e21b-6a7a-43e7-991a-c43ec3a112ce-trusted-ca-bundle\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:14:28.589284 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:28.589063    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bd87e21b-6a7a-43e7-991a-c43ec3a112ce-console-oauth-config\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:14:29.134902 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:29.134877    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6c7fbd8d55-9bjdw_bd87e21b-6a7a-43e7-991a-c43ec3a112ce/console/0.log"
May 25 12:14:29.135067 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:29.134914    2563 generic.go:358] "Generic (PLEG): container finished" podID="bd87e21b-6a7a-43e7-991a-c43ec3a112ce" containerID="ccce6793520befae3345c0d69066d4cfac2b40212af0ba11a0d5f37d37612243" exitCode=2
May 25 12:14:29.135067 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:29.134948    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c7fbd8d55-9bjdw" event={"ID":"bd87e21b-6a7a-43e7-991a-c43ec3a112ce","Type":"ContainerDied","Data":"ccce6793520befae3345c0d69066d4cfac2b40212af0ba11a0d5f37d37612243"}
May 25 12:14:29.135067 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:29.134978    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c7fbd8d55-9bjdw"
May 25 12:14:29.135067 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:29.134986    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c7fbd8d55-9bjdw" event={"ID":"bd87e21b-6a7a-43e7-991a-c43ec3a112ce","Type":"ContainerDied","Data":"260bdbe0176ccd03268f792e6620cd305bcef87d5700036a765dcadd53f76fa9"}
May 25 12:14:29.135067 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:29.135003    2563 scope.go:117] "RemoveContainer" containerID="ccce6793520befae3345c0d69066d4cfac2b40212af0ba11a0d5f37d37612243"
May 25 12:14:29.142890 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:29.142761    2563 scope.go:117] "RemoveContainer" containerID="ccce6793520befae3345c0d69066d4cfac2b40212af0ba11a0d5f37d37612243"
May 25 12:14:29.143143 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:14:29.143025    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccce6793520befae3345c0d69066d4cfac2b40212af0ba11a0d5f37d37612243\": container with ID starting with ccce6793520befae3345c0d69066d4cfac2b40212af0ba11a0d5f37d37612243 not found: ID does not exist" containerID="ccce6793520befae3345c0d69066d4cfac2b40212af0ba11a0d5f37d37612243"
May 25 12:14:29.143143 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:29.143049    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccce6793520befae3345c0d69066d4cfac2b40212af0ba11a0d5f37d37612243"} err="failed to get container status \"ccce6793520befae3345c0d69066d4cfac2b40212af0ba11a0d5f37d37612243\": rpc error: code = NotFound desc = could not find container \"ccce6793520befae3345c0d69066d4cfac2b40212af0ba11a0d5f37d37612243\": container with ID starting with ccce6793520befae3345c0d69066d4cfac2b40212af0ba11a0d5f37d37612243 not found: ID does not exist"
May 25 12:14:29.153001 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:29.152975    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c7fbd8d55-9bjdw"]
May 25 12:14:29.157016 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:29.156997    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c7fbd8d55-9bjdw"]
May 25 12:14:30.636593 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:30.636560    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd87e21b-6a7a-43e7-991a-c43ec3a112ce" path="/var/lib/kubelet/pods/bd87e21b-6a7a-43e7-991a-c43ec3a112ce/volumes"
May 25 12:14:41.003600 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:41.003562    2563 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:14:41.023177 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:41.023148    2563 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:14:41.184110 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:41.184083    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:14:59.090142 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.090107    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 25 12:14:59.090586 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.090539    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fa43e91d-4493-4558-aaff-84f2462c2876" containerName="prometheus" containerID="cri-o://6e96b4d89bef1e25eac6763faeb51e4103e7aeb6387269290a39d359788ba7d4" gracePeriod=600
May 25 12:14:59.090663 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.090582    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fa43e91d-4493-4558-aaff-84f2462c2876" containerName="kube-rbac-proxy" containerID="cri-o://68f828b5e8c20f1b07abf5556c81b0c57913b685d1e26b83764b99b4d40c3b48" gracePeriod=600
May 25 12:14:59.090663 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.090617    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fa43e91d-4493-4558-aaff-84f2462c2876" containerName="kube-rbac-proxy-web" containerID="cri-o://16e8df2fe462e4e5ecefaf7e1a19b366a10ba7fac67879ab5efcaab1659ccefb" gracePeriod=600
May 25 12:14:59.090768 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.090599    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fa43e91d-4493-4558-aaff-84f2462c2876" containerName="config-reloader" containerID="cri-o://20e72474385ed85b021cafcfc9be78b5af2567958dd56e962eecf5f65073b6b4" gracePeriod=600
May 25 12:14:59.090768 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.090671    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fa43e91d-4493-4558-aaff-84f2462c2876" containerName="kube-rbac-proxy-thanos" containerID="cri-o://a2966fe1b4ecaff8387ce96bf1b8bede2ff07f4e74325c3ef2b6592c21e0910b" gracePeriod=600
May 25 12:14:59.090856 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.090670    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fa43e91d-4493-4558-aaff-84f2462c2876" containerName="thanos-sidecar" containerID="cri-o://4e5b814d48c1df8791ac6d1e123ad24247aeb7059446042d88aa12687116c3cf" gracePeriod=600
May 25 12:14:59.219225 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.219189    2563 generic.go:358] "Generic (PLEG): container finished" podID="fa43e91d-4493-4558-aaff-84f2462c2876" containerID="a2966fe1b4ecaff8387ce96bf1b8bede2ff07f4e74325c3ef2b6592c21e0910b" exitCode=0
May 25 12:14:59.219225 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.219214    2563 generic.go:358] "Generic (PLEG): container finished" podID="fa43e91d-4493-4558-aaff-84f2462c2876" containerID="68f828b5e8c20f1b07abf5556c81b0c57913b685d1e26b83764b99b4d40c3b48" exitCode=0
May 25 12:14:59.219225 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.219221    2563 generic.go:358] "Generic (PLEG): container finished" podID="fa43e91d-4493-4558-aaff-84f2462c2876" containerID="16e8df2fe462e4e5ecefaf7e1a19b366a10ba7fac67879ab5efcaab1659ccefb" exitCode=0
May 25 12:14:59.219225 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.219229    2563 generic.go:358] "Generic (PLEG): container finished" podID="fa43e91d-4493-4558-aaff-84f2462c2876" containerID="4e5b814d48c1df8791ac6d1e123ad24247aeb7059446042d88aa12687116c3cf" exitCode=0
May 25 12:14:59.219458 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.219242    2563 generic.go:358] "Generic (PLEG): container finished" podID="fa43e91d-4493-4558-aaff-84f2462c2876" containerID="20e72474385ed85b021cafcfc9be78b5af2567958dd56e962eecf5f65073b6b4" exitCode=0
May 25 12:14:59.219458 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.219250    2563 generic.go:358] "Generic (PLEG): container finished" podID="fa43e91d-4493-4558-aaff-84f2462c2876" containerID="6e96b4d89bef1e25eac6763faeb51e4103e7aeb6387269290a39d359788ba7d4" exitCode=0
May 25 12:14:59.219458 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.219250    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fa43e91d-4493-4558-aaff-84f2462c2876","Type":"ContainerDied","Data":"a2966fe1b4ecaff8387ce96bf1b8bede2ff07f4e74325c3ef2b6592c21e0910b"}
May 25 12:14:59.219458 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.219283    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fa43e91d-4493-4558-aaff-84f2462c2876","Type":"ContainerDied","Data":"68f828b5e8c20f1b07abf5556c81b0c57913b685d1e26b83764b99b4d40c3b48"}
May 25 12:14:59.219458 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.219296    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fa43e91d-4493-4558-aaff-84f2462c2876","Type":"ContainerDied","Data":"16e8df2fe462e4e5ecefaf7e1a19b366a10ba7fac67879ab5efcaab1659ccefb"}
May 25 12:14:59.219458 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.219307    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fa43e91d-4493-4558-aaff-84f2462c2876","Type":"ContainerDied","Data":"4e5b814d48c1df8791ac6d1e123ad24247aeb7059446042d88aa12687116c3cf"}
May 25 12:14:59.219458 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.219321    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fa43e91d-4493-4558-aaff-84f2462c2876","Type":"ContainerDied","Data":"20e72474385ed85b021cafcfc9be78b5af2567958dd56e962eecf5f65073b6b4"}
May 25 12:14:59.219458 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.219333    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fa43e91d-4493-4558-aaff-84f2462c2876","Type":"ContainerDied","Data":"6e96b4d89bef1e25eac6763faeb51e4103e7aeb6387269290a39d359788ba7d4"}
May 25 12:14:59.343714 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.343643    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:14:59.436024 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.435992    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-secret-metrics-client-certs\") pod \"fa43e91d-4493-4558-aaff-84f2462c2876\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") "
May 25 12:14:59.436024 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.436025    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fa43e91d-4493-4558-aaff-84f2462c2876-prometheus-k8s-rulefiles-0\") pod \"fa43e91d-4493-4558-aaff-84f2462c2876\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") "
May 25 12:14:59.436213 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.436044    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-web-config\") pod \"fa43e91d-4493-4558-aaff-84f2462c2876\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") "
May 25 12:14:59.436213 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.436074    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-config\") pod \"fa43e91d-4493-4558-aaff-84f2462c2876\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") "
May 25 12:14:59.436299 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.436222    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-secret-grpc-tls\") pod \"fa43e91d-4493-4558-aaff-84f2462c2876\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") "
May 25 12:14:59.436299 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.436266    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-secret-kube-rbac-proxy\") pod \"fa43e91d-4493-4558-aaff-84f2462c2876\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") "
May 25 12:14:59.436389 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.436307    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa43e91d-4493-4558-aaff-84f2462c2876-prometheus-trusted-ca-bundle\") pod \"fa43e91d-4493-4558-aaff-84f2462c2876\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") "
May 25 12:14:59.436389 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.436336    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fa43e91d-4493-4558-aaff-84f2462c2876-tls-assets\") pod \"fa43e91d-4493-4558-aaff-84f2462c2876\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") "
May 25 12:14:59.438328 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.436927    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa43e91d-4493-4558-aaff-84f2462c2876-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "fa43e91d-4493-4558-aaff-84f2462c2876" (UID: "fa43e91d-4493-4558-aaff-84f2462c2876"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:14:59.438328 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.436990    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-thanos-prometheus-http-client-file\") pod \"fa43e91d-4493-4558-aaff-84f2462c2876\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") "
May 25 12:14:59.438328 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.437036    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa43e91d-4493-4558-aaff-84f2462c2876-configmap-serving-certs-ca-bundle\") pod \"fa43e91d-4493-4558-aaff-84f2462c2876\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") "
May 25 12:14:59.438328 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.437063    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-secret-prometheus-k8s-tls\") pod \"fa43e91d-4493-4558-aaff-84f2462c2876\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") "
May 25 12:14:59.438328 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.437092    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa43e91d-4493-4558-aaff-84f2462c2876-configmap-kubelet-serving-ca-bundle\") pod \"fa43e91d-4493-4558-aaff-84f2462c2876\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") "
May 25 12:14:59.438328 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.437127    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fa43e91d-4493-4558-aaff-84f2462c2876-prometheus-k8s-db\") pod \"fa43e91d-4493-4558-aaff-84f2462c2876\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") "
May 25 12:14:59.438328 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.437160    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa43e91d-4493-4558-aaff-84f2462c2876-configmap-metrics-client-ca\") pod \"fa43e91d-4493-4558-aaff-84f2462c2876\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") "
May 25 12:14:59.438328 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.437199    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fa43e91d-4493-4558-aaff-84f2462c2876-config-out\") pod \"fa43e91d-4493-4558-aaff-84f2462c2876\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") "
May 25 12:14:59.438328 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.437244    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"fa43e91d-4493-4558-aaff-84f2462c2876\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") "
May 25 12:14:59.438328 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.437454    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa43e91d-4493-4558-aaff-84f2462c2876-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "fa43e91d-4493-4558-aaff-84f2462c2876" (UID: "fa43e91d-4493-4558-aaff-84f2462c2876"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:14:59.438328 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.437706    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa43e91d-4493-4558-aaff-84f2462c2876-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "fa43e91d-4493-4558-aaff-84f2462c2876" (UID: "fa43e91d-4493-4558-aaff-84f2462c2876"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:14:59.438328 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.437292    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"fa43e91d-4493-4558-aaff-84f2462c2876\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") "
May 25 12:14:59.438328 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.437773    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-msndk\" (UniqueName: \"kubernetes.io/projected/fa43e91d-4493-4558-aaff-84f2462c2876-kube-api-access-msndk\") pod \"fa43e91d-4493-4558-aaff-84f2462c2876\" (UID: \"fa43e91d-4493-4558-aaff-84f2462c2876\") "
May 25 12:14:59.438328 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.437795    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa43e91d-4493-4558-aaff-84f2462c2876-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "fa43e91d-4493-4558-aaff-84f2462c2876" (UID: "fa43e91d-4493-4558-aaff-84f2462c2876"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:14:59.438328 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.438030    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fa43e91d-4493-4558-aaff-84f2462c2876-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:14:59.438328 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.438048    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa43e91d-4493-4558-aaff-84f2462c2876-prometheus-trusted-ca-bundle\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:14:59.438328 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.438065    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa43e91d-4493-4558-aaff-84f2462c2876-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:14:59.439324 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.438082    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa43e91d-4493-4558-aaff-84f2462c2876-configmap-metrics-client-ca\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:14:59.439324 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.438277    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa43e91d-4493-4558-aaff-84f2462c2876-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "fa43e91d-4493-4558-aaff-84f2462c2876" (UID: "fa43e91d-4493-4558-aaff-84f2462c2876"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:14:59.439324 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.438476    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa43e91d-4493-4558-aaff-84f2462c2876-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "fa43e91d-4493-4558-aaff-84f2462c2876" (UID: "fa43e91d-4493-4558-aaff-84f2462c2876"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:14:59.439324 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.438883    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "fa43e91d-4493-4558-aaff-84f2462c2876" (UID: "fa43e91d-4493-4558-aaff-84f2462c2876"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:14:59.439749 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.439590    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "fa43e91d-4493-4558-aaff-84f2462c2876" (UID: "fa43e91d-4493-4558-aaff-84f2462c2876"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:14:59.439749 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.439670    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-config" (OuterVolumeSpecName: "config") pod "fa43e91d-4493-4558-aaff-84f2462c2876" (UID: "fa43e91d-4493-4558-aaff-84f2462c2876"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:14:59.439924 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.439866    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "fa43e91d-4493-4558-aaff-84f2462c2876" (UID: "fa43e91d-4493-4558-aaff-84f2462c2876"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:14:59.440312 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.440282    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "fa43e91d-4493-4558-aaff-84f2462c2876" (UID: "fa43e91d-4493-4558-aaff-84f2462c2876"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:14:59.440502 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.440471    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "fa43e91d-4493-4558-aaff-84f2462c2876" (UID: "fa43e91d-4493-4558-aaff-84f2462c2876"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:14:59.440648 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.440512    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa43e91d-4493-4558-aaff-84f2462c2876-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "fa43e91d-4493-4558-aaff-84f2462c2876" (UID: "fa43e91d-4493-4558-aaff-84f2462c2876"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:14:59.440979 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.440958    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "fa43e91d-4493-4558-aaff-84f2462c2876" (UID: "fa43e91d-4493-4558-aaff-84f2462c2876"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:14:59.441144 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.441127    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa43e91d-4493-4558-aaff-84f2462c2876-config-out" (OuterVolumeSpecName: "config-out") pod "fa43e91d-4493-4558-aaff-84f2462c2876" (UID: "fa43e91d-4493-4558-aaff-84f2462c2876"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:14:59.441699 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.441676    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa43e91d-4493-4558-aaff-84f2462c2876-kube-api-access-msndk" (OuterVolumeSpecName: "kube-api-access-msndk") pod "fa43e91d-4493-4558-aaff-84f2462c2876" (UID: "fa43e91d-4493-4558-aaff-84f2462c2876"). InnerVolumeSpecName "kube-api-access-msndk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:14:59.441883 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.441864    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "fa43e91d-4493-4558-aaff-84f2462c2876" (UID: "fa43e91d-4493-4558-aaff-84f2462c2876"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:14:59.448887 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.448864    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-web-config" (OuterVolumeSpecName: "web-config") pod "fa43e91d-4493-4558-aaff-84f2462c2876" (UID: "fa43e91d-4493-4558-aaff-84f2462c2876"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:14:59.538509 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.538479    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-secret-prometheus-k8s-tls\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:14:59.538509 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.538504    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa43e91d-4493-4558-aaff-84f2462c2876-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:14:59.538509 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.538531    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fa43e91d-4493-4558-aaff-84f2462c2876-prometheus-k8s-db\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:14:59.538727 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.538544    2563 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fa43e91d-4493-4558-aaff-84f2462c2876-config-out\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:14:59.538727 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.538555    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:14:59.538727 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.538565    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:14:59.538727 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.538576    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msndk\" (UniqueName: \"kubernetes.io/projected/fa43e91d-4493-4558-aaff-84f2462c2876-kube-api-access-msndk\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:14:59.538727 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.538584    2563 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-secret-metrics-client-certs\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:14:59.538727 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.538593    2563 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-web-config\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:14:59.538727 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.538601    2563 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-config\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:14:59.538727 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.538609    2563 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-secret-grpc-tls\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:14:59.538727 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.538617    2563 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-secret-kube-rbac-proxy\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:14:59.538727 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.538626    2563 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fa43e91d-4493-4558-aaff-84f2462c2876-tls-assets\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:14:59.538727 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:14:59.538635    2563 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fa43e91d-4493-4558-aaff-84f2462c2876-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:15:00.224614 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.224574    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fa43e91d-4493-4558-aaff-84f2462c2876","Type":"ContainerDied","Data":"414a2d9ed1ec42d25f1533d9d76cbb944c193e451f1409ff1182bef0e08e1c8a"}
May 25 12:15:00.224614 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.224620    2563 scope.go:117] "RemoveContainer" containerID="a2966fe1b4ecaff8387ce96bf1b8bede2ff07f4e74325c3ef2b6592c21e0910b"
May 25 12:15:00.225086 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.224660    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.231612 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.231587    2563 scope.go:117] "RemoveContainer" containerID="68f828b5e8c20f1b07abf5556c81b0c57913b685d1e26b83764b99b4d40c3b48"
May 25 12:15:00.238198 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.238183    2563 scope.go:117] "RemoveContainer" containerID="16e8df2fe462e4e5ecefaf7e1a19b366a10ba7fac67879ab5efcaab1659ccefb"
May 25 12:15:00.244274 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.244254    2563 scope.go:117] "RemoveContainer" containerID="4e5b814d48c1df8791ac6d1e123ad24247aeb7059446042d88aa12687116c3cf"
May 25 12:15:00.248792 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.248773    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 25 12:15:00.250726 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.250701    2563 scope.go:117] "RemoveContainer" containerID="20e72474385ed85b021cafcfc9be78b5af2567958dd56e962eecf5f65073b6b4"
May 25 12:15:00.254272 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.254251    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 25 12:15:00.258110 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.258094    2563 scope.go:117] "RemoveContainer" containerID="6e96b4d89bef1e25eac6763faeb51e4103e7aeb6387269290a39d359788ba7d4"
May 25 12:15:00.264501 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.264485    2563 scope.go:117] "RemoveContainer" containerID="87971080e0ef4b289bd79b5650702298c2b11ae0348d084fce4125d7518a91b5"
May 25 12:15:00.284272 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.284245    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 25 12:15:00.284490 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.284478    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa43e91d-4493-4558-aaff-84f2462c2876" containerName="prometheus"
May 25 12:15:00.284550 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.284492    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa43e91d-4493-4558-aaff-84f2462c2876" containerName="prometheus"
May 25 12:15:00.284550 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.284507    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd87e21b-6a7a-43e7-991a-c43ec3a112ce" containerName="console"
May 25 12:15:00.284550 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.284512    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd87e21b-6a7a-43e7-991a-c43ec3a112ce" containerName="console"
May 25 12:15:00.284550 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.284531    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa43e91d-4493-4558-aaff-84f2462c2876" containerName="thanos-sidecar"
May 25 12:15:00.284550 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.284539    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa43e91d-4493-4558-aaff-84f2462c2876" containerName="thanos-sidecar"
May 25 12:15:00.284550 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.284549    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa43e91d-4493-4558-aaff-84f2462c2876" containerName="kube-rbac-proxy"
May 25 12:15:00.284724 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.284556    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa43e91d-4493-4558-aaff-84f2462c2876" containerName="kube-rbac-proxy"
May 25 12:15:00.284724 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.284564    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa43e91d-4493-4558-aaff-84f2462c2876" containerName="kube-rbac-proxy-web"
May 25 12:15:00.284724 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.284570    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa43e91d-4493-4558-aaff-84f2462c2876" containerName="kube-rbac-proxy-web"
May 25 12:15:00.284724 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.284580    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa43e91d-4493-4558-aaff-84f2462c2876" containerName="init-config-reloader"
May 25 12:15:00.284724 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.284585    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa43e91d-4493-4558-aaff-84f2462c2876" containerName="init-config-reloader"
May 25 12:15:00.284724 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.284591    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa43e91d-4493-4558-aaff-84f2462c2876" containerName="config-reloader"
May 25 12:15:00.284724 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.284595    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa43e91d-4493-4558-aaff-84f2462c2876" containerName="config-reloader"
May 25 12:15:00.284724 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.284602    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa43e91d-4493-4558-aaff-84f2462c2876" containerName="kube-rbac-proxy-thanos"
May 25 12:15:00.284724 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.284607    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa43e91d-4493-4558-aaff-84f2462c2876" containerName="kube-rbac-proxy-thanos"
May 25 12:15:00.284724 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.284648    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="fa43e91d-4493-4558-aaff-84f2462c2876" containerName="kube-rbac-proxy"
May 25 12:15:00.284724 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.284655    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="fa43e91d-4493-4558-aaff-84f2462c2876" containerName="kube-rbac-proxy-thanos"
May 25 12:15:00.284724 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.284661    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="fa43e91d-4493-4558-aaff-84f2462c2876" containerName="prometheus"
May 25 12:15:00.284724 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.284667    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="fa43e91d-4493-4558-aaff-84f2462c2876" containerName="thanos-sidecar"
May 25 12:15:00.284724 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.284674    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="fa43e91d-4493-4558-aaff-84f2462c2876" containerName="kube-rbac-proxy-web"
May 25 12:15:00.284724 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.284682    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="bd87e21b-6a7a-43e7-991a-c43ec3a112ce" containerName="console"
May 25 12:15:00.284724 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.284689    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="fa43e91d-4493-4558-aaff-84f2462c2876" containerName="config-reloader"
May 25 12:15:00.289849 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.289832    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.293364 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.293201    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
May 25 12:15:00.293364 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.293209    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
May 25 12:15:00.293364 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.293227    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
May 25 12:15:00.293364 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.293254    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
May 25 12:15:00.293364 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.293262    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
May 25 12:15:00.293364 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.293275    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
May 25 12:15:00.293364 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.293276    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-gjsxk\""
May 25 12:15:00.293735 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.293719    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
May 25 12:15:00.295809 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.295791    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
May 25 12:15:00.295980 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.295812    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7l0efrbh8tors\""
May 25 12:15:00.296175 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.296153    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
May 25 12:15:00.296272 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.296259    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
May 25 12:15:00.296487 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.296355    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
May 25 12:15:00.301829 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.301811    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
May 25 12:15:00.304699 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.304683    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
May 25 12:15:00.314070 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.314050    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 25 12:15:00.343843 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.343807    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5cef7c45-3024-47d3-88d9-bab6710a1e14-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.343843 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.343845    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5cef7c45-3024-47d3-88d9-bab6710a1e14-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.344072 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.343877    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5cef7c45-3024-47d3-88d9-bab6710a1e14-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.344072 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.343899    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5cef7c45-3024-47d3-88d9-bab6710a1e14-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.344072 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.343921    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5cef7c45-3024-47d3-88d9-bab6710a1e14-config\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.344072 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.343941    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp77k\" (UniqueName: \"kubernetes.io/projected/5cef7c45-3024-47d3-88d9-bab6710a1e14-kube-api-access-zp77k\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.344072 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.343995    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cef7c45-3024-47d3-88d9-bab6710a1e14-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.344072 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.344028    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5cef7c45-3024-47d3-88d9-bab6710a1e14-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.344072 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.344059    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cef7c45-3024-47d3-88d9-bab6710a1e14-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.344360 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.344081    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5cef7c45-3024-47d3-88d9-bab6710a1e14-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.344360 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.344105    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5cef7c45-3024-47d3-88d9-bab6710a1e14-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.344360 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.344155    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5cef7c45-3024-47d3-88d9-bab6710a1e14-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.344360 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.344204    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5cef7c45-3024-47d3-88d9-bab6710a1e14-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.344360 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.344252    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5cef7c45-3024-47d3-88d9-bab6710a1e14-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.344360 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.344270    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5cef7c45-3024-47d3-88d9-bab6710a1e14-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.344360 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.344294    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cef7c45-3024-47d3-88d9-bab6710a1e14-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.344360 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.344336    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5cef7c45-3024-47d3-88d9-bab6710a1e14-config-out\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.344360 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.344355    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5cef7c45-3024-47d3-88d9-bab6710a1e14-web-config\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.445304 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.445218    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5cef7c45-3024-47d3-88d9-bab6710a1e14-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.445304 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.445256    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5cef7c45-3024-47d3-88d9-bab6710a1e14-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.445304 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.445278    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cef7c45-3024-47d3-88d9-bab6710a1e14-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.445304 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.445307    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5cef7c45-3024-47d3-88d9-bab6710a1e14-config-out\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.445595 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.445324    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5cef7c45-3024-47d3-88d9-bab6710a1e14-web-config\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.445595 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.445345    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5cef7c45-3024-47d3-88d9-bab6710a1e14-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.445595 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.445361    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5cef7c45-3024-47d3-88d9-bab6710a1e14-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.445595 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.445382    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5cef7c45-3024-47d3-88d9-bab6710a1e14-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.445595 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.445410    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5cef7c45-3024-47d3-88d9-bab6710a1e14-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.445595 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.445438    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5cef7c45-3024-47d3-88d9-bab6710a1e14-config\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.445595 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.445462    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zp77k\" (UniqueName: \"kubernetes.io/projected/5cef7c45-3024-47d3-88d9-bab6710a1e14-kube-api-access-zp77k\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.445595 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.445488    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cef7c45-3024-47d3-88d9-bab6710a1e14-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.445595 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.445513    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5cef7c45-3024-47d3-88d9-bab6710a1e14-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.445595 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.445566    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cef7c45-3024-47d3-88d9-bab6710a1e14-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.445595 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.445597    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5cef7c45-3024-47d3-88d9-bab6710a1e14-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.446089 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.445628    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5cef7c45-3024-47d3-88d9-bab6710a1e14-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.446089 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.445660    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5cef7c45-3024-47d3-88d9-bab6710a1e14-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.446089 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.445695    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5cef7c45-3024-47d3-88d9-bab6710a1e14-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.446089 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.445697    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5cef7c45-3024-47d3-88d9-bab6710a1e14-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.446646 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.446368    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cef7c45-3024-47d3-88d9-bab6710a1e14-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.448753 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.448571    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5cef7c45-3024-47d3-88d9-bab6710a1e14-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.448753 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.448572    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5cef7c45-3024-47d3-88d9-bab6710a1e14-config\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.448753 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.448678    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5cef7c45-3024-47d3-88d9-bab6710a1e14-config-out\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.448753 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.448695    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5cef7c45-3024-47d3-88d9-bab6710a1e14-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.448753 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.448717    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5cef7c45-3024-47d3-88d9-bab6710a1e14-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.449084 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.448851    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5cef7c45-3024-47d3-88d9-bab6710a1e14-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.449084 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.449059    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5cef7c45-3024-47d3-88d9-bab6710a1e14-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.449341 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.449252    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5cef7c45-3024-47d3-88d9-bab6710a1e14-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.449560 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.449537    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5cef7c45-3024-47d3-88d9-bab6710a1e14-web-config\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.449687 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.449664    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cef7c45-3024-47d3-88d9-bab6710a1e14-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.449997 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.449975    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cef7c45-3024-47d3-88d9-bab6710a1e14-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.450984 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.450964    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5cef7c45-3024-47d3-88d9-bab6710a1e14-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.451089 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.451076    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5cef7c45-3024-47d3-88d9-bab6710a1e14-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.451802 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.451783    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5cef7c45-3024-47d3-88d9-bab6710a1e14-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.452599 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.452579    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5cef7c45-3024-47d3-88d9-bab6710a1e14-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.457064 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.457028    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp77k\" (UniqueName: \"kubernetes.io/projected/5cef7c45-3024-47d3-88d9-bab6710a1e14-kube-api-access-zp77k\") pod \"prometheus-k8s-0\" (UID: \"5cef7c45-3024-47d3-88d9-bab6710a1e14\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.598792 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.598736    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:00.643109 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.643064    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa43e91d-4493-4558-aaff-84f2462c2876" path="/var/lib/kubelet/pods/fa43e91d-4493-4558-aaff-84f2462c2876/volumes"
May 25 12:15:00.732493 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:00.732416    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 25 12:15:00.735341 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:15:00.735310    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cef7c45_3024_47d3_88d9_bab6710a1e14.slice/crio-1fc51a9fde6347ac8afea73731f999ba1d595d0a4d07ea61b8fb7d8dc429becd WatchSource:0}: Error finding container 1fc51a9fde6347ac8afea73731f999ba1d595d0a4d07ea61b8fb7d8dc429becd: Status 404 returned error can't find the container with id 1fc51a9fde6347ac8afea73731f999ba1d595d0a4d07ea61b8fb7d8dc429becd
May 25 12:15:01.229116 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:01.229079    2563 generic.go:358] "Generic (PLEG): container finished" podID="5cef7c45-3024-47d3-88d9-bab6710a1e14" containerID="0b768638bb4ab50478d0a5de063fa68effd4e2916c5d144af919aaa8a6d11764" exitCode=0
May 25 12:15:01.229574 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:01.229126    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5cef7c45-3024-47d3-88d9-bab6710a1e14","Type":"ContainerDied","Data":"0b768638bb4ab50478d0a5de063fa68effd4e2916c5d144af919aaa8a6d11764"}
May 25 12:15:01.229574 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:01.229158    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5cef7c45-3024-47d3-88d9-bab6710a1e14","Type":"ContainerStarted","Data":"1fc51a9fde6347ac8afea73731f999ba1d595d0a4d07ea61b8fb7d8dc429becd"}
May 25 12:15:02.234860 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:02.234824    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5cef7c45-3024-47d3-88d9-bab6710a1e14","Type":"ContainerStarted","Data":"16052cf12f5129b33a0a8619a403928ddd38bc542069367bd6284a3bdf433317"}
May 25 12:15:02.234860 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:02.234858    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5cef7c45-3024-47d3-88d9-bab6710a1e14","Type":"ContainerStarted","Data":"0c63d2729e3e8efc8960f0f87f09d0819a950cfadac7501cf1c473466b35b926"}
May 25 12:15:02.234860 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:02.234867    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5cef7c45-3024-47d3-88d9-bab6710a1e14","Type":"ContainerStarted","Data":"a8a5c75cb5876cd935343c7869e45dedaea8f75c681a3e28a8793228b37534b6"}
May 25 12:15:02.235447 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:02.234876    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5cef7c45-3024-47d3-88d9-bab6710a1e14","Type":"ContainerStarted","Data":"18bf7328916c562f0789e1d5adcdca659eb34696287d3e169cd22d33d5f9d852"}
May 25 12:15:02.235447 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:02.234885    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5cef7c45-3024-47d3-88d9-bab6710a1e14","Type":"ContainerStarted","Data":"f373b3f58ba5f1d59d40b0e6122892fd9222d2846a8ff316bf11a957018469e8"}
May 25 12:15:02.235447 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:02.234894    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5cef7c45-3024-47d3-88d9-bab6710a1e14","Type":"ContainerStarted","Data":"a11d0d2e8a1748c84e71b6166a3301919820802be8c7d6ff2a1f6a625fc2d0ab"}
May 25 12:15:02.263549 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:02.263480    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.263461635 podStartE2EDuration="2.263461635s" podCreationTimestamp="2026-05-25 12:15:00 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:15:02.262241565 +0000 UTC m=+168.229633205" watchObservedRunningTime="2026-05-25 12:15:02.263461635 +0000 UTC m=+168.230853276"
May 25 12:15:03.937723 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:03.937638    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6947f9fd8d-gmknf"]
May 25 12:15:05.599230 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:05.599198    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:15:11.728910 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:11.728874    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-475zv"]
May 25 12:15:11.731664 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:11.731648    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-475zv"
May 25 12:15:11.733429 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:11.733408    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 25 12:15:11.739168 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:11.739141    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-475zv"]
May 25 12:15:11.827979 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:11.827948    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be7c8adf-8b89-4ab0-9f79-b5d4fd568ded-original-pull-secret\") pod \"global-pull-secret-syncer-475zv\" (UID: \"be7c8adf-8b89-4ab0-9f79-b5d4fd568ded\") " pod="kube-system/global-pull-secret-syncer-475zv"
May 25 12:15:11.828166 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:11.827986    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/be7c8adf-8b89-4ab0-9f79-b5d4fd568ded-kubelet-config\") pod \"global-pull-secret-syncer-475zv\" (UID: \"be7c8adf-8b89-4ab0-9f79-b5d4fd568ded\") " pod="kube-system/global-pull-secret-syncer-475zv"
May 25 12:15:11.828166 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:11.828018    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/be7c8adf-8b89-4ab0-9f79-b5d4fd568ded-dbus\") pod \"global-pull-secret-syncer-475zv\" (UID: \"be7c8adf-8b89-4ab0-9f79-b5d4fd568ded\") " pod="kube-system/global-pull-secret-syncer-475zv"
May 25 12:15:11.929195 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:11.929149    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be7c8adf-8b89-4ab0-9f79-b5d4fd568ded-original-pull-secret\") pod \"global-pull-secret-syncer-475zv\" (UID: \"be7c8adf-8b89-4ab0-9f79-b5d4fd568ded\") " pod="kube-system/global-pull-secret-syncer-475zv"
May 25 12:15:11.929360 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:11.929208    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/be7c8adf-8b89-4ab0-9f79-b5d4fd568ded-kubelet-config\") pod \"global-pull-secret-syncer-475zv\" (UID: \"be7c8adf-8b89-4ab0-9f79-b5d4fd568ded\") " pod="kube-system/global-pull-secret-syncer-475zv"
May 25 12:15:11.929360 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:11.929250    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/be7c8adf-8b89-4ab0-9f79-b5d4fd568ded-dbus\") pod \"global-pull-secret-syncer-475zv\" (UID: \"be7c8adf-8b89-4ab0-9f79-b5d4fd568ded\") " pod="kube-system/global-pull-secret-syncer-475zv"
May 25 12:15:11.929360 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:11.929343    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/be7c8adf-8b89-4ab0-9f79-b5d4fd568ded-kubelet-config\") pod \"global-pull-secret-syncer-475zv\" (UID: \"be7c8adf-8b89-4ab0-9f79-b5d4fd568ded\") " pod="kube-system/global-pull-secret-syncer-475zv"
May 25 12:15:11.929499 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:11.929421    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/be7c8adf-8b89-4ab0-9f79-b5d4fd568ded-dbus\") pod \"global-pull-secret-syncer-475zv\" (UID: \"be7c8adf-8b89-4ab0-9f79-b5d4fd568ded\") " pod="kube-system/global-pull-secret-syncer-475zv"
May 25 12:15:11.931390 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:11.931364    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be7c8adf-8b89-4ab0-9f79-b5d4fd568ded-original-pull-secret\") pod \"global-pull-secret-syncer-475zv\" (UID: \"be7c8adf-8b89-4ab0-9f79-b5d4fd568ded\") " pod="kube-system/global-pull-secret-syncer-475zv"
May 25 12:15:12.041644 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:12.041612    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-475zv"
May 25 12:15:12.165548 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:12.165509    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-475zv"]
May 25 12:15:12.168110 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:15:12.168076    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe7c8adf_8b89_4ab0_9f79_b5d4fd568ded.slice/crio-0c18a54d38e4a9aa545a7e568a13276220fb2030db66611188d66d31c1715861 WatchSource:0}: Error finding container 0c18a54d38e4a9aa545a7e568a13276220fb2030db66611188d66d31c1715861: Status 404 returned error can't find the container with id 0c18a54d38e4a9aa545a7e568a13276220fb2030db66611188d66d31c1715861
May 25 12:15:12.262256 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:12.262217    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-475zv" event={"ID":"be7c8adf-8b89-4ab0-9f79-b5d4fd568ded","Type":"ContainerStarted","Data":"0c18a54d38e4a9aa545a7e568a13276220fb2030db66611188d66d31c1715861"}
May 25 12:15:16.275427 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:16.274688    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-475zv" event={"ID":"be7c8adf-8b89-4ab0-9f79-b5d4fd568ded","Type":"ContainerStarted","Data":"2d335886f262e908db1bd3b54a81217a8e325a3ae32841410557d0fa2b18f4cc"}
May 25 12:15:16.289767 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:16.289693    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-475zv" podStartSLOduration=1.330209649 podStartE2EDuration="5.289673955s" podCreationTimestamp="2026-05-25 12:15:11 +0000 UTC" firstStartedPulling="2026-05-25 12:15:12.169749016 +0000 UTC m=+178.137140633" lastFinishedPulling="2026-05-25 12:15:16.129213318 +0000 UTC m=+182.096604939" observedRunningTime="2026-05-25 12:15:16.28900403 +0000 UTC m=+182.256395670" watchObservedRunningTime="2026-05-25 12:15:16.289673955 +0000 UTC m=+182.257065595"
May 25 12:15:28.956535 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:28.956459    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-6947f9fd8d-gmknf" podUID="2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa" containerName="console" containerID="cri-o://4991df51a4a37c0a41141f713819cb4c21dbfc29e56694054fc85e8e5e1c0bb4" gracePeriod=15
May 25 12:15:29.195009 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:29.194985    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6947f9fd8d-gmknf_2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa/console/0.log"
May 25 12:15:29.195130 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:29.195045    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6947f9fd8d-gmknf"
May 25 12:15:29.276063 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:29.276031    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa-console-config\") pod \"2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa\" (UID: \"2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa\") "
May 25 12:15:29.276229 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:29.276088    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5gff\" (UniqueName: \"kubernetes.io/projected/2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa-kube-api-access-j5gff\") pod \"2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa\" (UID: \"2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa\") "
May 25 12:15:29.276229 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:29.276112    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa-oauth-serving-cert\") pod \"2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa\" (UID: \"2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa\") "
May 25 12:15:29.276229 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:29.276154    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa-console-serving-cert\") pod \"2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa\" (UID: \"2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa\") "
May 25 12:15:29.276229 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:29.276177    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa-console-oauth-config\") pod \"2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa\" (UID: \"2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa\") "
May 25 12:15:29.276229 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:29.276192    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa-trusted-ca-bundle\") pod \"2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa\" (UID: \"2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa\") "
May 25 12:15:29.276465 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:29.276237    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa-service-ca\") pod \"2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa\" (UID: \"2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa\") "
May 25 12:15:29.276726 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:29.276505    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa-console-config" (OuterVolumeSpecName: "console-config") pod "2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa" (UID: "2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:15:29.276726 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:29.276696    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa" (UID: "2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:15:29.276856 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:29.276725    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa" (UID: "2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:15:29.276856 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:29.276756    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa-service-ca" (OuterVolumeSpecName: "service-ca") pod "2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa" (UID: "2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:15:29.278417 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:29.278391    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa" (UID: "2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:15:29.278480 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:29.278417    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa-kube-api-access-j5gff" (OuterVolumeSpecName: "kube-api-access-j5gff") pod "2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa" (UID: "2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa"). InnerVolumeSpecName "kube-api-access-j5gff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:15:29.278480 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:29.278453    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa" (UID: "2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:15:29.309949 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:29.309923    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6947f9fd8d-gmknf_2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa/console/0.log"
May 25 12:15:29.310070 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:29.309966    2563 generic.go:358] "Generic (PLEG): container finished" podID="2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa" containerID="4991df51a4a37c0a41141f713819cb4c21dbfc29e56694054fc85e8e5e1c0bb4" exitCode=2
May 25 12:15:29.310070 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:29.310030    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6947f9fd8d-gmknf"
May 25 12:15:29.310070 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:29.310051    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6947f9fd8d-gmknf" event={"ID":"2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa","Type":"ContainerDied","Data":"4991df51a4a37c0a41141f713819cb4c21dbfc29e56694054fc85e8e5e1c0bb4"}
May 25 12:15:29.310176 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:29.310089    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6947f9fd8d-gmknf" event={"ID":"2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa","Type":"ContainerDied","Data":"4fdac02510b38ca85cf9b3f347eb521f8c48e60b53ad73fb1c2fc9be8c6f5d8e"}
May 25 12:15:29.310176 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:29.310105    2563 scope.go:117] "RemoveContainer" containerID="4991df51a4a37c0a41141f713819cb4c21dbfc29e56694054fc85e8e5e1c0bb4"
May 25 12:15:29.317425 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:29.317406    2563 scope.go:117] "RemoveContainer" containerID="4991df51a4a37c0a41141f713819cb4c21dbfc29e56694054fc85e8e5e1c0bb4"
May 25 12:15:29.317721 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:15:29.317700    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4991df51a4a37c0a41141f713819cb4c21dbfc29e56694054fc85e8e5e1c0bb4\": container with ID starting with 4991df51a4a37c0a41141f713819cb4c21dbfc29e56694054fc85e8e5e1c0bb4 not found: ID does not exist" containerID="4991df51a4a37c0a41141f713819cb4c21dbfc29e56694054fc85e8e5e1c0bb4"
May 25 12:15:29.317794 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:29.317733    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4991df51a4a37c0a41141f713819cb4c21dbfc29e56694054fc85e8e5e1c0bb4"} err="failed to get container status \"4991df51a4a37c0a41141f713819cb4c21dbfc29e56694054fc85e8e5e1c0bb4\": rpc error: code = NotFound desc = could not find container \"4991df51a4a37c0a41141f713819cb4c21dbfc29e56694054fc85e8e5e1c0bb4\": container with ID starting with 4991df51a4a37c0a41141f713819cb4c21dbfc29e56694054fc85e8e5e1c0bb4 not found: ID does not exist"
May 25 12:15:29.333062 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:29.333039    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6947f9fd8d-gmknf"]
May 25 12:15:29.342564 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:29.342545    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6947f9fd8d-gmknf"]
May 25 12:15:29.377584 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:29.377554    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa-console-serving-cert\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:15:29.377584 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:29.377581    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa-console-oauth-config\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:15:29.377700 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:29.377593    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa-trusted-ca-bundle\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:15:29.377700 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:29.377603    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa-service-ca\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:15:29.377700 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:29.377613    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa-console-config\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:15:29.377700 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:29.377621    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5gff\" (UniqueName: \"kubernetes.io/projected/2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa-kube-api-access-j5gff\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:15:29.377700 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:29.377629    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa-oauth-serving-cert\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:15:30.636752 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:15:30.636717    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa" path="/var/lib/kubelet/pods/2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa/volumes"
May 25 12:16:00.599817 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:16:00.599762    2563 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:16:00.615243 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:16:00.615216    2563 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:16:01.411152 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:16:01.411121    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
May 25 12:17:14.525293 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:17:14.525260    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw9bb_012c465e-2683-488a-abc1-18ad86529eb3/ovn-acl-logging/0.log"
May 25 12:17:14.525864 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:17:14.525277    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw9bb_012c465e-2683-488a-abc1-18ad86529eb3/ovn-acl-logging/0.log"
May 25 12:17:14.530732 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:17:14.530710    2563 kubelet.go:1628] "Image garbage collection succeeded"
May 25 12:17:51.629641 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:17:51.629603    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6768985688-ws6fx"]
May 25 12:17:51.630080 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:17:51.629870    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa" containerName="console"
May 25 12:17:51.630080 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:17:51.629880    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa" containerName="console"
May 25 12:17:51.630080 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:17:51.629930    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="2bd84ee5-0eef-44fb-b9d4-69a807f2c9aa" containerName="console"
May 25 12:17:51.631987 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:17:51.631969    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6768985688-ws6fx"
May 25 12:17:51.633646 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:17:51.633624    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
May 25 12:17:51.633915 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:17:51.633900    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
May 25 12:17:51.634355 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:17:51.634337    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
May 25 12:17:51.634438 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:17:51.634346    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-wrzfv\""
May 25 12:17:51.638752 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:17:51.638725    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6768985688-ws6fx"]
May 25 12:17:51.767380 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:17:51.767284    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50a56e05-8a6f-4fc0-9fc1-928cd0912875-cert\") pod \"kserve-controller-manager-6768985688-ws6fx\" (UID: \"50a56e05-8a6f-4fc0-9fc1-928cd0912875\") " pod="kserve/kserve-controller-manager-6768985688-ws6fx"
May 25 12:17:51.767380 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:17:51.767334    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gxw4\" (UniqueName: \"kubernetes.io/projected/50a56e05-8a6f-4fc0-9fc1-928cd0912875-kube-api-access-8gxw4\") pod \"kserve-controller-manager-6768985688-ws6fx\" (UID: \"50a56e05-8a6f-4fc0-9fc1-928cd0912875\") " pod="kserve/kserve-controller-manager-6768985688-ws6fx"
May 25 12:17:51.867970 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:17:51.867944    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50a56e05-8a6f-4fc0-9fc1-928cd0912875-cert\") pod \"kserve-controller-manager-6768985688-ws6fx\" (UID: \"50a56e05-8a6f-4fc0-9fc1-928cd0912875\") " pod="kserve/kserve-controller-manager-6768985688-ws6fx"
May 25 12:17:51.868134 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:17:51.867980    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8gxw4\" (UniqueName: \"kubernetes.io/projected/50a56e05-8a6f-4fc0-9fc1-928cd0912875-kube-api-access-8gxw4\") pod \"kserve-controller-manager-6768985688-ws6fx\" (UID: \"50a56e05-8a6f-4fc0-9fc1-928cd0912875\") " pod="kserve/kserve-controller-manager-6768985688-ws6fx"
May 25 12:17:51.870655 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:17:51.870626    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50a56e05-8a6f-4fc0-9fc1-928cd0912875-cert\") pod \"kserve-controller-manager-6768985688-ws6fx\" (UID: \"50a56e05-8a6f-4fc0-9fc1-928cd0912875\") " pod="kserve/kserve-controller-manager-6768985688-ws6fx"
May 25 12:17:51.879280 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:17:51.879255    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gxw4\" (UniqueName: \"kubernetes.io/projected/50a56e05-8a6f-4fc0-9fc1-928cd0912875-kube-api-access-8gxw4\") pod \"kserve-controller-manager-6768985688-ws6fx\" (UID: \"50a56e05-8a6f-4fc0-9fc1-928cd0912875\") " pod="kserve/kserve-controller-manager-6768985688-ws6fx"
May 25 12:17:51.942682 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:17:51.942654    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6768985688-ws6fx"
May 25 12:17:52.061339 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:17:52.061262    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6768985688-ws6fx"]
May 25 12:17:52.063613 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:17:52.063579    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50a56e05_8a6f_4fc0_9fc1_928cd0912875.slice/crio-aa777c1e281896ac6cad54bd12a43c797c5e73508b4f435b9e401d5c7c1eb503 WatchSource:0}: Error finding container aa777c1e281896ac6cad54bd12a43c797c5e73508b4f435b9e401d5c7c1eb503: Status 404 returned error can't find the container with id aa777c1e281896ac6cad54bd12a43c797c5e73508b4f435b9e401d5c7c1eb503
May 25 12:17:52.064867 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:17:52.064851    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 12:17:52.689967 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:17:52.689918    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6768985688-ws6fx" event={"ID":"50a56e05-8a6f-4fc0-9fc1-928cd0912875","Type":"ContainerStarted","Data":"aa777c1e281896ac6cad54bd12a43c797c5e73508b4f435b9e401d5c7c1eb503"}
May 25 12:17:54.696310 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:17:54.696223    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6768985688-ws6fx" event={"ID":"50a56e05-8a6f-4fc0-9fc1-928cd0912875","Type":"ContainerStarted","Data":"79170359c700fdbc3b217817dc3014c5d6f58f0ebced8a9f028f162539d6fcf4"}
May 25 12:17:54.696681 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:17:54.696345    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6768985688-ws6fx"
May 25 12:17:54.711300 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:17:54.711250    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6768985688-ws6fx" podStartSLOduration=1.429091243 podStartE2EDuration="3.711238318s" podCreationTimestamp="2026-05-25 12:17:51 +0000 UTC" firstStartedPulling="2026-05-25 12:17:52.064971643 +0000 UTC m=+338.032363260" lastFinishedPulling="2026-05-25 12:17:54.3471187 +0000 UTC m=+340.314510335" observedRunningTime="2026-05-25 12:17:54.710323284 +0000 UTC m=+340.677714922" watchObservedRunningTime="2026-05-25 12:17:54.711238318 +0000 UTC m=+340.678629957"
May 25 12:18:25.705688 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:18:25.705609    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6768985688-ws6fx"
May 25 12:18:43.469407 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:18:43.469376    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/s3-init-bj9zz"]
May 25 12:18:43.471920 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:18:43.471905    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-bj9zz"
May 25 12:18:43.473620 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:18:43.473597    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
May 25 12:18:43.473620 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:18:43.473616    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-b92kn\""
May 25 12:18:43.478615 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:18:43.478568    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-bj9zz"]
May 25 12:18:43.559247 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:18:43.559211    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmth7\" (UniqueName: \"kubernetes.io/projected/6614a31e-ecf8-4bc8-b8c6-d7ec0a070375-kube-api-access-cmth7\") pod \"s3-init-bj9zz\" (UID: \"6614a31e-ecf8-4bc8-b8c6-d7ec0a070375\") " pod="kserve/s3-init-bj9zz"
May 25 12:18:43.659993 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:18:43.659957    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cmth7\" (UniqueName: \"kubernetes.io/projected/6614a31e-ecf8-4bc8-b8c6-d7ec0a070375-kube-api-access-cmth7\") pod \"s3-init-bj9zz\" (UID: \"6614a31e-ecf8-4bc8-b8c6-d7ec0a070375\") " pod="kserve/s3-init-bj9zz"
May 25 12:18:43.668899 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:18:43.668870    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmth7\" (UniqueName: \"kubernetes.io/projected/6614a31e-ecf8-4bc8-b8c6-d7ec0a070375-kube-api-access-cmth7\") pod \"s3-init-bj9zz\" (UID: \"6614a31e-ecf8-4bc8-b8c6-d7ec0a070375\") " pod="kserve/s3-init-bj9zz"
May 25 12:18:43.796897 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:18:43.796870    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-bj9zz"
May 25 12:18:43.919266 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:18:43.919234    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-bj9zz"]
May 25 12:18:43.922479 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:18:43.922445    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6614a31e_ecf8_4bc8_b8c6_d7ec0a070375.slice/crio-2e4daab03cb23b4fa2cc85e4137665b28100a902a52ee81a87333ef332eb27ce WatchSource:0}: Error finding container 2e4daab03cb23b4fa2cc85e4137665b28100a902a52ee81a87333ef332eb27ce: Status 404 returned error can't find the container with id 2e4daab03cb23b4fa2cc85e4137665b28100a902a52ee81a87333ef332eb27ce
May 25 12:18:44.836553 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:18:44.836487    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bj9zz" event={"ID":"6614a31e-ecf8-4bc8-b8c6-d7ec0a070375","Type":"ContainerStarted","Data":"2e4daab03cb23b4fa2cc85e4137665b28100a902a52ee81a87333ef332eb27ce"}
May 25 12:18:48.850288 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:18:48.850245    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bj9zz" event={"ID":"6614a31e-ecf8-4bc8-b8c6-d7ec0a070375","Type":"ContainerStarted","Data":"d139b5753bca0b87190519759e1c86f60a9241e18b8d5b743849a56026b8c6f6"}
May 25 12:18:48.867254 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:18:48.867206    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/s3-init-bj9zz" podStartSLOduration=1.142289756 podStartE2EDuration="5.867191628s" podCreationTimestamp="2026-05-25 12:18:43 +0000 UTC" firstStartedPulling="2026-05-25 12:18:43.924159761 +0000 UTC m=+389.891551379" lastFinishedPulling="2026-05-25 12:18:48.649061632 +0000 UTC m=+394.616453251" observedRunningTime="2026-05-25 12:18:48.864876735 +0000 UTC m=+394.832268373" watchObservedRunningTime="2026-05-25 12:18:48.867191628 +0000 UTC m=+394.834583266"
May 25 12:18:51.860373 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:18:51.860294    2563 generic.go:358] "Generic (PLEG): container finished" podID="6614a31e-ecf8-4bc8-b8c6-d7ec0a070375" containerID="d139b5753bca0b87190519759e1c86f60a9241e18b8d5b743849a56026b8c6f6" exitCode=0
May 25 12:18:51.860724 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:18:51.860368    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bj9zz" event={"ID":"6614a31e-ecf8-4bc8-b8c6-d7ec0a070375","Type":"ContainerDied","Data":"d139b5753bca0b87190519759e1c86f60a9241e18b8d5b743849a56026b8c6f6"}
May 25 12:18:52.984485 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:18:52.984461    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-bj9zz"
May 25 12:18:53.038849 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:18:53.038817    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmth7\" (UniqueName: \"kubernetes.io/projected/6614a31e-ecf8-4bc8-b8c6-d7ec0a070375-kube-api-access-cmth7\") pod \"6614a31e-ecf8-4bc8-b8c6-d7ec0a070375\" (UID: \"6614a31e-ecf8-4bc8-b8c6-d7ec0a070375\") "
May 25 12:18:53.040949 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:18:53.040918    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6614a31e-ecf8-4bc8-b8c6-d7ec0a070375-kube-api-access-cmth7" (OuterVolumeSpecName: "kube-api-access-cmth7") pod "6614a31e-ecf8-4bc8-b8c6-d7ec0a070375" (UID: "6614a31e-ecf8-4bc8-b8c6-d7ec0a070375"). InnerVolumeSpecName "kube-api-access-cmth7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:18:53.139484 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:18:53.139410    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmth7\" (UniqueName: \"kubernetes.io/projected/6614a31e-ecf8-4bc8-b8c6-d7ec0a070375-kube-api-access-cmth7\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:18:53.866948 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:18:53.866920    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-bj9zz"
May 25 12:18:53.867120 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:18:53.866915    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bj9zz" event={"ID":"6614a31e-ecf8-4bc8-b8c6-d7ec0a070375","Type":"ContainerDied","Data":"2e4daab03cb23b4fa2cc85e4137665b28100a902a52ee81a87333ef332eb27ce"}
May 25 12:18:53.867120 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:18:53.867029    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e4daab03cb23b4fa2cc85e4137665b28100a902a52ee81a87333ef332eb27ce"
May 25 12:19:03.175401 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:03.175361    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4"]
May 25 12:19:03.175845 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:03.175653    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6614a31e-ecf8-4bc8-b8c6-d7ec0a070375" containerName="s3-init"
May 25 12:19:03.175845 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:03.175666    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6614a31e-ecf8-4bc8-b8c6-d7ec0a070375" containerName="s3-init"
May 25 12:19:03.175845 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:03.175720    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="6614a31e-ecf8-4bc8-b8c6-d7ec0a070375" containerName="s3-init"
May 25 12:19:03.179014 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:03.178996    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4"
May 25 12:19:03.180977 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:03.180946    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-b25e8-predictor-serving-cert\""
May 25 12:19:03.181079 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:03.180954    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 12:19:03.181079 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:03.181040    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-79bgq\""
May 25 12:19:03.181079 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:03.181040    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-b25e8-kube-rbac-proxy-sar-config\""
May 25 12:19:03.181221 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:03.180986    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 12:19:03.188499 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:03.188474    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4"]
May 25 12:19:03.317589 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:03.317511    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmk88\" (UniqueName: \"kubernetes.io/projected/e7c39c0c-8a3a-45b5-b369-20cf7b32c434-kube-api-access-lmk88\") pod \"isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4\" (UID: \"e7c39c0c-8a3a-45b5-b369-20cf7b32c434\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4"
May 25 12:19:03.317589 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:03.317601    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7c39c0c-8a3a-45b5-b369-20cf7b32c434-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4\" (UID: \"e7c39c0c-8a3a-45b5-b369-20cf7b32c434\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4"
May 25 12:19:03.317817 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:03.317642    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7c39c0c-8a3a-45b5-b369-20cf7b32c434-proxy-tls\") pod \"isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4\" (UID: \"e7c39c0c-8a3a-45b5-b369-20cf7b32c434\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4"
May 25 12:19:03.317817 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:03.317678    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-raw-sklearn-batcher-b25e8-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7c39c0c-8a3a-45b5-b369-20cf7b32c434-isvc-raw-sklearn-batcher-b25e8-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4\" (UID: \"e7c39c0c-8a3a-45b5-b369-20cf7b32c434\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4"
May 25 12:19:03.418585 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:03.418547    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-raw-sklearn-batcher-b25e8-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7c39c0c-8a3a-45b5-b369-20cf7b32c434-isvc-raw-sklearn-batcher-b25e8-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4\" (UID: \"e7c39c0c-8a3a-45b5-b369-20cf7b32c434\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4"
May 25 12:19:03.418786 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:03.418624    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lmk88\" (UniqueName: \"kubernetes.io/projected/e7c39c0c-8a3a-45b5-b369-20cf7b32c434-kube-api-access-lmk88\") pod \"isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4\" (UID: \"e7c39c0c-8a3a-45b5-b369-20cf7b32c434\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4"
May 25 12:19:03.418786 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:03.418650    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7c39c0c-8a3a-45b5-b369-20cf7b32c434-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4\" (UID: \"e7c39c0c-8a3a-45b5-b369-20cf7b32c434\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4"
May 25 12:19:03.418786 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:03.418702    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7c39c0c-8a3a-45b5-b369-20cf7b32c434-proxy-tls\") pod \"isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4\" (UID: \"e7c39c0c-8a3a-45b5-b369-20cf7b32c434\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4"
May 25 12:19:03.418954 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:19:03.418794    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-serving-cert: secret "isvc-raw-sklearn-batcher-b25e8-predictor-serving-cert" not found
May 25 12:19:03.418954 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:19:03.418861    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7c39c0c-8a3a-45b5-b369-20cf7b32c434-proxy-tls podName:e7c39c0c-8a3a-45b5-b369-20cf7b32c434 nodeName:}" failed. No retries permitted until 2026-05-25 12:19:03.918841801 +0000 UTC m=+409.886233417 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e7c39c0c-8a3a-45b5-b369-20cf7b32c434-proxy-tls") pod "isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4" (UID: "e7c39c0c-8a3a-45b5-b369-20cf7b32c434") : secret "isvc-raw-sklearn-batcher-b25e8-predictor-serving-cert" not found
May 25 12:19:03.419086 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:03.419063    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7c39c0c-8a3a-45b5-b369-20cf7b32c434-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4\" (UID: \"e7c39c0c-8a3a-45b5-b369-20cf7b32c434\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4"
May 25 12:19:03.419338 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:03.419313    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-raw-sklearn-batcher-b25e8-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7c39c0c-8a3a-45b5-b369-20cf7b32c434-isvc-raw-sklearn-batcher-b25e8-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4\" (UID: \"e7c39c0c-8a3a-45b5-b369-20cf7b32c434\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4"
May 25 12:19:03.427512 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:03.427453    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmk88\" (UniqueName: \"kubernetes.io/projected/e7c39c0c-8a3a-45b5-b369-20cf7b32c434-kube-api-access-lmk88\") pod \"isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4\" (UID: \"e7c39c0c-8a3a-45b5-b369-20cf7b32c434\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4"
May 25 12:19:03.923108 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:03.923072    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7c39c0c-8a3a-45b5-b369-20cf7b32c434-proxy-tls\") pod \"isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4\" (UID: \"e7c39c0c-8a3a-45b5-b369-20cf7b32c434\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4"
May 25 12:19:03.925452 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:03.925432    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7c39c0c-8a3a-45b5-b369-20cf7b32c434-proxy-tls\") pod \"isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4\" (UID: \"e7c39c0c-8a3a-45b5-b369-20cf7b32c434\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4"
May 25 12:19:04.091123 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:04.091081    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4"
May 25 12:19:04.214869 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:04.214829    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4"]
May 25 12:19:04.217536 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:19:04.217498    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7c39c0c_8a3a_45b5_b369_20cf7b32c434.slice/crio-86f4d2b0d60540b965461904e5e95733cc89292c48fc136fc275c155ae3b3a5b WatchSource:0}: Error finding container 86f4d2b0d60540b965461904e5e95733cc89292c48fc136fc275c155ae3b3a5b: Status 404 returned error can't find the container with id 86f4d2b0d60540b965461904e5e95733cc89292c48fc136fc275c155ae3b3a5b
May 25 12:19:04.894436 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:04.894396    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4" event={"ID":"e7c39c0c-8a3a-45b5-b369-20cf7b32c434","Type":"ContainerStarted","Data":"86f4d2b0d60540b965461904e5e95733cc89292c48fc136fc275c155ae3b3a5b"}
May 25 12:19:08.906649 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:08.906606    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4" event={"ID":"e7c39c0c-8a3a-45b5-b369-20cf7b32c434","Type":"ContainerStarted","Data":"624217e7f8c3f20a403b40e467f939cc64a5b99504a2c64c1de3e782767ce3f8"}
May 25 12:19:11.916147 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:11.916119    2563 generic.go:358] "Generic (PLEG): container finished" podID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerID="624217e7f8c3f20a403b40e467f939cc64a5b99504a2c64c1de3e782767ce3f8" exitCode=0
May 25 12:19:11.916494 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:11.916189    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4" event={"ID":"e7c39c0c-8a3a-45b5-b369-20cf7b32c434","Type":"ContainerDied","Data":"624217e7f8c3f20a403b40e467f939cc64a5b99504a2c64c1de3e782767ce3f8"}
May 25 12:19:25.960902 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:25.960844    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4" event={"ID":"e7c39c0c-8a3a-45b5-b369-20cf7b32c434","Type":"ContainerStarted","Data":"492459420e36ead56fbac6fe7e759c642e4f12ff03688e203759c10936ee884b"}
May 25 12:19:27.969357 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:27.969276    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4" event={"ID":"e7c39c0c-8a3a-45b5-b369-20cf7b32c434","Type":"ContainerStarted","Data":"162e78c7390a74c0af4ef9d832afe3ecb0515907f66f7ec224c8f85c3154d2c3"}
May 25 12:19:30.982730 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:30.982697    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4" event={"ID":"e7c39c0c-8a3a-45b5-b369-20cf7b32c434","Type":"ContainerStarted","Data":"acb4b35e27c9fea46808c7b39f46373bfad1ee22a81f59ed27129bdbdb1e635e"}
May 25 12:19:30.983164 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:30.982858    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4"
May 25 12:19:31.003212 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:31.003167    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4" podStartSLOduration=1.860820233 podStartE2EDuration="28.003154052s" podCreationTimestamp="2026-05-25 12:19:03 +0000 UTC" firstStartedPulling="2026-05-25 12:19:04.219678012 +0000 UTC m=+410.187069635" lastFinishedPulling="2026-05-25 12:19:30.362011837 +0000 UTC m=+436.329403454" observedRunningTime="2026-05-25 12:19:31.000895765 +0000 UTC m=+436.968287424" watchObservedRunningTime="2026-05-25 12:19:31.003154052 +0000 UTC m=+436.970545690"
May 25 12:19:31.985416 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:31.985373    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4"
May 25 12:19:31.985416 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:31.985415    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4"
May 25 12:19:31.986843 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:31.986804    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4" podUID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
May 25 12:19:31.987533 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:31.987495    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4" podUID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:19:32.988097 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:32.988055    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4" podUID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
May 25 12:19:32.988576 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:32.988426    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4" podUID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:19:32.991721 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:32.991701    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4"
May 25 12:19:33.991048 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:33.990968    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4" podUID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
May 25 12:19:33.991456 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:33.991297    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4" podUID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:19:43.991373 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:43.991327    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4" podUID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
May 25 12:19:43.991841 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:43.991816    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4" podUID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:19:53.991791 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:53.991736    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4" podUID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
May 25 12:19:53.992222 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:19:53.992123    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4" podUID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:20:03.991862 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:03.991806    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4" podUID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
May 25 12:20:03.992343 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:03.992285    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4" podUID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:20:13.991210 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:13.991163    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4" podUID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
May 25 12:20:13.991665 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:13.991594    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4" podUID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:20:23.991004 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:23.990953    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4" podUID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
May 25 12:20:23.991550 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:23.991490    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4" podUID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:20:33.992397 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:33.992357    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4"
May 25 12:20:33.992847 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:33.992577    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4"
May 25 12:20:48.226542 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:48.226479    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4"]
May 25 12:20:48.227031 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:48.226933    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4" podUID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerName="kserve-container" containerID="cri-o://492459420e36ead56fbac6fe7e759c642e4f12ff03688e203759c10936ee884b" gracePeriod=30
May 25 12:20:48.227240 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:48.227155    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4" podUID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerName="kube-rbac-proxy" containerID="cri-o://162e78c7390a74c0af4ef9d832afe3ecb0515907f66f7ec224c8f85c3154d2c3" gracePeriod=30
May 25 12:20:48.227240 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:48.227178    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4" podUID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerName="agent" containerID="cri-o://acb4b35e27c9fea46808c7b39f46373bfad1ee22a81f59ed27129bdbdb1e635e" gracePeriod=30
May 25 12:20:48.323745 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:48.323709    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm"]
May 25 12:20:48.327321 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:48.327294    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm"
May 25 12:20:48.329265 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:48.329245    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-a7098-predictor-serving-cert\""
May 25 12:20:48.329391 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:48.329338    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-a7098-kube-rbac-proxy-sar-config\""
May 25 12:20:48.337904 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:48.336770    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm"]
May 25 12:20:48.370080 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:48.370048    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fe1ba95-5b0a-4156-a667-a264c7dd7e5b-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm\" (UID: \"8fe1ba95-5b0a-4156-a667-a264c7dd7e5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm"
May 25 12:20:48.370314 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:48.370290    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fe1ba95-5b0a-4156-a667-a264c7dd7e5b-proxy-tls\") pod \"isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm\" (UID: \"8fe1ba95-5b0a-4156-a667-a264c7dd7e5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm"
May 25 12:20:48.370456 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:48.370353    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-graph-raw-a7098-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fe1ba95-5b0a-4156-a667-a264c7dd7e5b-isvc-sklearn-graph-raw-a7098-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm\" (UID: \"8fe1ba95-5b0a-4156-a667-a264c7dd7e5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm"
May 25 12:20:48.370540 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:48.370452    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw2mc\" (UniqueName: \"kubernetes.io/projected/8fe1ba95-5b0a-4156-a667-a264c7dd7e5b-kube-api-access-qw2mc\") pod \"isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm\" (UID: \"8fe1ba95-5b0a-4156-a667-a264c7dd7e5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm"
May 25 12:20:48.393365 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:48.393326    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk"]
May 25 12:20:48.396387 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:48.396364    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk"
May 25 12:20:48.398288 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:48.398259    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-a7098-predictor-serving-cert\""
May 25 12:20:48.398288 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:48.398276    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-a7098-kube-rbac-proxy-sar-config\""
May 25 12:20:48.404033 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:48.404007    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk"]
May 25 12:20:48.471889 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:48.471845    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-graph-raw-a7098-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cef298dd-6c2a-4607-9201-7dc8e42a58c0-isvc-xgboost-graph-raw-a7098-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk\" (UID: \"cef298dd-6c2a-4607-9201-7dc8e42a58c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk"
May 25 12:20:48.471889 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:48.471887    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-graph-raw-a7098-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fe1ba95-5b0a-4156-a667-a264c7dd7e5b-isvc-sklearn-graph-raw-a7098-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm\" (UID: \"8fe1ba95-5b0a-4156-a667-a264c7dd7e5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm"
May 25 12:20:48.472138 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:48.471962    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cef298dd-6c2a-4607-9201-7dc8e42a58c0-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk\" (UID: \"cef298dd-6c2a-4607-9201-7dc8e42a58c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk"
May 25 12:20:48.472138 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:48.472045    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qw2mc\" (UniqueName: \"kubernetes.io/projected/8fe1ba95-5b0a-4156-a667-a264c7dd7e5b-kube-api-access-qw2mc\") pod \"isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm\" (UID: \"8fe1ba95-5b0a-4156-a667-a264c7dd7e5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm"
May 25 12:20:48.472138 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:48.472098    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x6ms\" (UniqueName: \"kubernetes.io/projected/cef298dd-6c2a-4607-9201-7dc8e42a58c0-kube-api-access-4x6ms\") pod \"isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk\" (UID: \"cef298dd-6c2a-4607-9201-7dc8e42a58c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk"
May 25 12:20:48.472270 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:48.472159    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fe1ba95-5b0a-4156-a667-a264c7dd7e5b-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm\" (UID: \"8fe1ba95-5b0a-4156-a667-a264c7dd7e5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm"
May 25 12:20:48.472270 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:48.472188    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cef298dd-6c2a-4607-9201-7dc8e42a58c0-proxy-tls\") pod \"isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk\" (UID: \"cef298dd-6c2a-4607-9201-7dc8e42a58c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk"
May 25 12:20:48.472270 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:48.472227    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fe1ba95-5b0a-4156-a667-a264c7dd7e5b-proxy-tls\") pod \"isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm\" (UID: \"8fe1ba95-5b0a-4156-a667-a264c7dd7e5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm"
May 25 12:20:48.472499 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:48.472476    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fe1ba95-5b0a-4156-a667-a264c7dd7e5b-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm\" (UID: \"8fe1ba95-5b0a-4156-a667-a264c7dd7e5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm"
May 25 12:20:48.472766 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:48.472743    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-graph-raw-a7098-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fe1ba95-5b0a-4156-a667-a264c7dd7e5b-isvc-sklearn-graph-raw-a7098-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm\" (UID: \"8fe1ba95-5b0a-4156-a667-a264c7dd7e5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm"
May 25 12:20:48.474840 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:48.474820    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fe1ba95-5b0a-4156-a667-a264c7dd7e5b-proxy-tls\") pod \"isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm\" (UID: \"8fe1ba95-5b0a-4156-a667-a264c7dd7e5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm"
May 25 12:20:48.480893 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:48.480834    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw2mc\" (UniqueName: \"kubernetes.io/projected/8fe1ba95-5b0a-4156-a667-a264c7dd7e5b-kube-api-access-qw2mc\") pod \"isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm\" (UID: \"8fe1ba95-5b0a-4156-a667-a264c7dd7e5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm"
May 25 12:20:48.573093 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:48.573056    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4x6ms\" (UniqueName: \"kubernetes.io/projected/cef298dd-6c2a-4607-9201-7dc8e42a58c0-kube-api-access-4x6ms\") pod \"isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk\" (UID: \"cef298dd-6c2a-4607-9201-7dc8e42a58c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk"
May 25 12:20:48.573296 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:48.573123    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cef298dd-6c2a-4607-9201-7dc8e42a58c0-proxy-tls\") pod \"isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk\" (UID: \"cef298dd-6c2a-4607-9201-7dc8e42a58c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk"
May 25 12:20:48.573296 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:48.573171    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-graph-raw-a7098-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cef298dd-6c2a-4607-9201-7dc8e42a58c0-isvc-xgboost-graph-raw-a7098-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk\" (UID: \"cef298dd-6c2a-4607-9201-7dc8e42a58c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk"
May 25 12:20:48.573296 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:48.573211    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cef298dd-6c2a-4607-9201-7dc8e42a58c0-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk\" (UID: \"cef298dd-6c2a-4607-9201-7dc8e42a58c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk"
May 25 12:20:48.573457 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:20:48.573317    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-serving-cert: secret "isvc-xgboost-graph-raw-a7098-predictor-serving-cert" not found
May 25 12:20:48.573457 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:20:48.573388    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cef298dd-6c2a-4607-9201-7dc8e42a58c0-proxy-tls podName:cef298dd-6c2a-4607-9201-7dc8e42a58c0 nodeName:}" failed. No retries permitted until 2026-05-25 12:20:49.073366638 +0000 UTC m=+515.040758268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/cef298dd-6c2a-4607-9201-7dc8e42a58c0-proxy-tls") pod "isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk" (UID: "cef298dd-6c2a-4607-9201-7dc8e42a58c0") : secret "isvc-xgboost-graph-raw-a7098-predictor-serving-cert" not found
May 25 12:20:48.573698 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:48.573675    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cef298dd-6c2a-4607-9201-7dc8e42a58c0-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk\" (UID: \"cef298dd-6c2a-4607-9201-7dc8e42a58c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk"
May 25 12:20:48.573890 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:48.573872    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-graph-raw-a7098-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cef298dd-6c2a-4607-9201-7dc8e42a58c0-isvc-xgboost-graph-raw-a7098-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk\" (UID: \"cef298dd-6c2a-4607-9201-7dc8e42a58c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk"
May 25 12:20:48.581884 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:48.581854    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x6ms\" (UniqueName: \"kubernetes.io/projected/cef298dd-6c2a-4607-9201-7dc8e42a58c0-kube-api-access-4x6ms\") pod \"isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk\" (UID: \"cef298dd-6c2a-4607-9201-7dc8e42a58c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk"
May 25 12:20:48.645110 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:48.645079    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm"
May 25 12:20:48.778675 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:48.778649    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm"]
May 25 12:20:48.780419 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:20:48.780393    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fe1ba95_5b0a_4156_a667_a264c7dd7e5b.slice/crio-421a391a401c45b06e097fa97e70c9bae5ba34593d333cd0fac1f61717351ff0 WatchSource:0}: Error finding container 421a391a401c45b06e097fa97e70c9bae5ba34593d333cd0fac1f61717351ff0: Status 404 returned error can't find the container with id 421a391a401c45b06e097fa97e70c9bae5ba34593d333cd0fac1f61717351ff0
May 25 12:20:49.078694 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:49.078615    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cef298dd-6c2a-4607-9201-7dc8e42a58c0-proxy-tls\") pod \"isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk\" (UID: \"cef298dd-6c2a-4607-9201-7dc8e42a58c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk"
May 25 12:20:49.080912 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:49.080885    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cef298dd-6c2a-4607-9201-7dc8e42a58c0-proxy-tls\") pod \"isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk\" (UID: \"cef298dd-6c2a-4607-9201-7dc8e42a58c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk"
May 25 12:20:49.208563 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:49.208497    2563 generic.go:358] "Generic (PLEG): container finished" podID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerID="162e78c7390a74c0af4ef9d832afe3ecb0515907f66f7ec224c8f85c3154d2c3" exitCode=2
May 25 12:20:49.208737 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:49.208576    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4" event={"ID":"e7c39c0c-8a3a-45b5-b369-20cf7b32c434","Type":"ContainerDied","Data":"162e78c7390a74c0af4ef9d832afe3ecb0515907f66f7ec224c8f85c3154d2c3"}
May 25 12:20:49.209983 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:49.209961    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm" event={"ID":"8fe1ba95-5b0a-4156-a667-a264c7dd7e5b","Type":"ContainerStarted","Data":"8c354d25405efa568e436b73d819292b46b3504dcd04dcb0ed68a302ac5d0de4"}
May 25 12:20:49.210104 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:49.209988    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm" event={"ID":"8fe1ba95-5b0a-4156-a667-a264c7dd7e5b","Type":"ContainerStarted","Data":"421a391a401c45b06e097fa97e70c9bae5ba34593d333cd0fac1f61717351ff0"}
May 25 12:20:49.308128 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:49.308096    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk"
May 25 12:20:49.433367 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:49.433343    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk"]
May 25 12:20:49.435894 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:20:49.435864    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcef298dd_6c2a_4607_9201_7dc8e42a58c0.slice/crio-1af898cdf391f262afd082cc7609e622752486aa2ac5db41e11cca3383fd5e14 WatchSource:0}: Error finding container 1af898cdf391f262afd082cc7609e622752486aa2ac5db41e11cca3383fd5e14: Status 404 returned error can't find the container with id 1af898cdf391f262afd082cc7609e622752486aa2ac5db41e11cca3383fd5e14
May 25 12:20:50.213927 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:50.213892    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk" event={"ID":"cef298dd-6c2a-4607-9201-7dc8e42a58c0","Type":"ContainerStarted","Data":"6c25db827f5c05265344beb2a54971d58c406b21a1561094e2c4201156ae6c0c"}
May 25 12:20:50.213927 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:50.213931    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk" event={"ID":"cef298dd-6c2a-4607-9201-7dc8e42a58c0","Type":"ContainerStarted","Data":"1af898cdf391f262afd082cc7609e622752486aa2ac5db41e11cca3383fd5e14"}
May 25 12:20:52.988268 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:52.988167    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4" podUID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
May 25 12:20:53.224425 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:53.224387    2563 generic.go:358] "Generic (PLEG): container finished" podID="8fe1ba95-5b0a-4156-a667-a264c7dd7e5b" containerID="8c354d25405efa568e436b73d819292b46b3504dcd04dcb0ed68a302ac5d0de4" exitCode=0
May 25 12:20:53.224596 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:53.224466    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm" event={"ID":"8fe1ba95-5b0a-4156-a667-a264c7dd7e5b","Type":"ContainerDied","Data":"8c354d25405efa568e436b73d819292b46b3504dcd04dcb0ed68a302ac5d0de4"}
May 25 12:20:53.225830 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:53.225801    2563 generic.go:358] "Generic (PLEG): container finished" podID="cef298dd-6c2a-4607-9201-7dc8e42a58c0" containerID="6c25db827f5c05265344beb2a54971d58c406b21a1561094e2c4201156ae6c0c" exitCode=0
May 25 12:20:53.225966 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:53.225889    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk" event={"ID":"cef298dd-6c2a-4607-9201-7dc8e42a58c0","Type":"ContainerDied","Data":"6c25db827f5c05265344beb2a54971d58c406b21a1561094e2c4201156ae6c0c"}
May 25 12:20:53.228211 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:53.228188    2563 generic.go:358] "Generic (PLEG): container finished" podID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerID="492459420e36ead56fbac6fe7e759c642e4f12ff03688e203759c10936ee884b" exitCode=0
May 25 12:20:53.228287 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:53.228217    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4" event={"ID":"e7c39c0c-8a3a-45b5-b369-20cf7b32c434","Type":"ContainerDied","Data":"492459420e36ead56fbac6fe7e759c642e4f12ff03688e203759c10936ee884b"}
May 25 12:20:53.991232 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:53.991144    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4" podUID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
May 25 12:20:53.991731 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:53.991496    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4" podUID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:20:54.233984 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:54.233871    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm" event={"ID":"8fe1ba95-5b0a-4156-a667-a264c7dd7e5b","Type":"ContainerStarted","Data":"1875b035b7e66c2bc58796082b5a934bb74743fc640304e0c80f75661b3c4189"}
May 25 12:20:54.233984 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:54.233913    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm" event={"ID":"8fe1ba95-5b0a-4156-a667-a264c7dd7e5b","Type":"ContainerStarted","Data":"e5cc5c68458309ed848541ccb2dc119404cdcd5cdd65d0ffeba9956f57cde891"}
May 25 12:20:54.234586 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:54.234489    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm"
May 25 12:20:54.234586 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:54.234557    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm"
May 25 12:20:54.235950 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:54.235922    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm" podUID="8fe1ba95-5b0a-4156-a667-a264c7dd7e5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
May 25 12:20:54.253011 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:54.252927    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm" podStartSLOduration=6.252893145 podStartE2EDuration="6.252893145s" podCreationTimestamp="2026-05-25 12:20:48 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:20:54.252267214 +0000 UTC m=+520.219658854" watchObservedRunningTime="2026-05-25 12:20:54.252893145 +0000 UTC m=+520.220284786"
May 25 12:20:55.238686 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:55.238427    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm" podUID="8fe1ba95-5b0a-4156-a667-a264c7dd7e5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
May 25 12:20:56.242485 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:56.242432    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm" podUID="8fe1ba95-5b0a-4156-a667-a264c7dd7e5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
May 25 12:20:57.989168 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:20:57.989117    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4" podUID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
May 25 12:21:01.248636 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:01.248603    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm"
May 25 12:21:01.249393 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:01.249348    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm" podUID="8fe1ba95-5b0a-4156-a667-a264c7dd7e5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
May 25 12:21:02.989094 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:02.989045    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4" podUID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
May 25 12:21:02.989534 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:02.989179    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4"
May 25 12:21:03.991621 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:03.991491    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4" podUID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
May 25 12:21:03.992055 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:03.991939    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4" podUID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:21:07.988348 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:07.988302    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4" podUID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
May 25 12:21:11.249388 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:11.249355    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm" podUID="8fe1ba95-5b0a-4156-a667-a264c7dd7e5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
May 25 12:21:11.294175 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:11.294141    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk" event={"ID":"cef298dd-6c2a-4607-9201-7dc8e42a58c0","Type":"ContainerStarted","Data":"7ac7abc2aa17131caba5de318e1a84cbbe99cfaddd0b5537f11963c0d282b9fb"}
May 25 12:21:11.294334 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:11.294182    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk" event={"ID":"cef298dd-6c2a-4607-9201-7dc8e42a58c0","Type":"ContainerStarted","Data":"080b58adab648fcf5797d8d41daca77d91a8c62c4d89f7ea2ba930ba27dd6285"}
May 25 12:21:11.294456 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:11.294431    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk"
May 25 12:21:11.312513 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:11.312456    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk" podStartSLOduration=5.550526847 podStartE2EDuration="23.312439363s" podCreationTimestamp="2026-05-25 12:20:48 +0000 UTC" firstStartedPulling="2026-05-25 12:20:53.227168507 +0000 UTC m=+519.194560133" lastFinishedPulling="2026-05-25 12:21:10.989081029 +0000 UTC m=+536.956472649" observedRunningTime="2026-05-25 12:21:11.311592209 +0000 UTC m=+537.278983848" watchObservedRunningTime="2026-05-25 12:21:11.312439363 +0000 UTC m=+537.279831004"
May 25 12:21:12.297834 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:12.297799    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk"
May 25 12:21:12.298921 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:12.298895    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk" podUID="cef298dd-6c2a-4607-9201-7dc8e42a58c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
May 25 12:21:12.988959 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:12.988915    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4" podUID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
May 25 12:21:13.300529 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:13.300479    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk" podUID="cef298dd-6c2a-4607-9201-7dc8e42a58c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
May 25 12:21:13.991478 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:13.991430    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4" podUID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
May 25 12:21:13.991686 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:13.991629    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4"
May 25 12:21:13.991753 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:13.991733    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4" podUID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:21:13.991839 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:13.991826    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4"
May 25 12:21:17.988934 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:17.988887    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4" podUID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
May 25 12:21:18.304603 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:18.304574    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk"
May 25 12:21:18.305173 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:18.305148    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk" podUID="cef298dd-6c2a-4607-9201-7dc8e42a58c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
May 25 12:21:18.883660 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:18.883639    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4"
May 25 12:21:19.042941 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:19.042866    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7c39c0c-8a3a-45b5-b369-20cf7b32c434-kserve-provision-location\") pod \"e7c39c0c-8a3a-45b5-b369-20cf7b32c434\" (UID: \"e7c39c0c-8a3a-45b5-b369-20cf7b32c434\") "
May 25 12:21:19.042941 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:19.042900    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7c39c0c-8a3a-45b5-b369-20cf7b32c434-proxy-tls\") pod \"e7c39c0c-8a3a-45b5-b369-20cf7b32c434\" (UID: \"e7c39c0c-8a3a-45b5-b369-20cf7b32c434\") "
May 25 12:21:19.042941 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:19.042941    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-raw-sklearn-batcher-b25e8-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7c39c0c-8a3a-45b5-b369-20cf7b32c434-isvc-raw-sklearn-batcher-b25e8-kube-rbac-proxy-sar-config\") pod \"e7c39c0c-8a3a-45b5-b369-20cf7b32c434\" (UID: \"e7c39c0c-8a3a-45b5-b369-20cf7b32c434\") "
May 25 12:21:19.043478 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:19.042977    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmk88\" (UniqueName: \"kubernetes.io/projected/e7c39c0c-8a3a-45b5-b369-20cf7b32c434-kube-api-access-lmk88\") pod \"e7c39c0c-8a3a-45b5-b369-20cf7b32c434\" (UID: \"e7c39c0c-8a3a-45b5-b369-20cf7b32c434\") "
May 25 12:21:19.043478 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:19.043207    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7c39c0c-8a3a-45b5-b369-20cf7b32c434-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e7c39c0c-8a3a-45b5-b369-20cf7b32c434" (UID: "e7c39c0c-8a3a-45b5-b369-20cf7b32c434"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:21:19.043478 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:19.043278    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7c39c0c-8a3a-45b5-b369-20cf7b32c434-isvc-raw-sklearn-batcher-b25e8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-raw-sklearn-batcher-b25e8-kube-rbac-proxy-sar-config") pod "e7c39c0c-8a3a-45b5-b369-20cf7b32c434" (UID: "e7c39c0c-8a3a-45b5-b369-20cf7b32c434"). InnerVolumeSpecName "isvc-raw-sklearn-batcher-b25e8-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:21:19.045084 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:19.045060    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7c39c0c-8a3a-45b5-b369-20cf7b32c434-kube-api-access-lmk88" (OuterVolumeSpecName: "kube-api-access-lmk88") pod "e7c39c0c-8a3a-45b5-b369-20cf7b32c434" (UID: "e7c39c0c-8a3a-45b5-b369-20cf7b32c434"). InnerVolumeSpecName "kube-api-access-lmk88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:21:19.045147 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:19.045104    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7c39c0c-8a3a-45b5-b369-20cf7b32c434-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e7c39c0c-8a3a-45b5-b369-20cf7b32c434" (UID: "e7c39c0c-8a3a-45b5-b369-20cf7b32c434"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:21:19.143614 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:19.143582    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-raw-sklearn-batcher-b25e8-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7c39c0c-8a3a-45b5-b369-20cf7b32c434-isvc-raw-sklearn-batcher-b25e8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:21:19.143614 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:19.143608    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmk88\" (UniqueName: \"kubernetes.io/projected/e7c39c0c-8a3a-45b5-b369-20cf7b32c434-kube-api-access-lmk88\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:21:19.143614 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:19.143619    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7c39c0c-8a3a-45b5-b369-20cf7b32c434-kserve-provision-location\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:21:19.143813 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:19.143629    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7c39c0c-8a3a-45b5-b369-20cf7b32c434-proxy-tls\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:21:19.320928 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:19.320832    2563 generic.go:358] "Generic (PLEG): container finished" podID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerID="acb4b35e27c9fea46808c7b39f46373bfad1ee22a81f59ed27129bdbdb1e635e" exitCode=0
May 25 12:21:19.320928 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:19.320884    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4" event={"ID":"e7c39c0c-8a3a-45b5-b369-20cf7b32c434","Type":"ContainerDied","Data":"acb4b35e27c9fea46808c7b39f46373bfad1ee22a81f59ed27129bdbdb1e635e"}
May 25 12:21:19.320928 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:19.320923    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4" event={"ID":"e7c39c0c-8a3a-45b5-b369-20cf7b32c434","Type":"ContainerDied","Data":"86f4d2b0d60540b965461904e5e95733cc89292c48fc136fc275c155ae3b3a5b"}
May 25 12:21:19.321203 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:19.320928    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4"
May 25 12:21:19.321203 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:19.320945    2563 scope.go:117] "RemoveContainer" containerID="acb4b35e27c9fea46808c7b39f46373bfad1ee22a81f59ed27129bdbdb1e635e"
May 25 12:21:19.330285 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:19.330181    2563 scope.go:117] "RemoveContainer" containerID="162e78c7390a74c0af4ef9d832afe3ecb0515907f66f7ec224c8f85c3154d2c3"
May 25 12:21:19.336960 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:19.336939    2563 scope.go:117] "RemoveContainer" containerID="492459420e36ead56fbac6fe7e759c642e4f12ff03688e203759c10936ee884b"
May 25 12:21:19.343833 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:19.343814    2563 scope.go:117] "RemoveContainer" containerID="624217e7f8c3f20a403b40e467f939cc64a5b99504a2c64c1de3e782767ce3f8"
May 25 12:21:19.345219 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:19.345196    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4"]
May 25 12:21:19.349270 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:19.349245    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b25e8-predictor-7865b99f89-wf5w4"]
May 25 12:21:19.351034 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:19.351018    2563 scope.go:117] "RemoveContainer" containerID="acb4b35e27c9fea46808c7b39f46373bfad1ee22a81f59ed27129bdbdb1e635e"
May 25 12:21:19.351287 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:21:19.351269    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acb4b35e27c9fea46808c7b39f46373bfad1ee22a81f59ed27129bdbdb1e635e\": container with ID starting with acb4b35e27c9fea46808c7b39f46373bfad1ee22a81f59ed27129bdbdb1e635e not found: ID does not exist" containerID="acb4b35e27c9fea46808c7b39f46373bfad1ee22a81f59ed27129bdbdb1e635e"
May 25 12:21:19.351342 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:19.351297    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acb4b35e27c9fea46808c7b39f46373bfad1ee22a81f59ed27129bdbdb1e635e"} err="failed to get container status \"acb4b35e27c9fea46808c7b39f46373bfad1ee22a81f59ed27129bdbdb1e635e\": rpc error: code = NotFound desc = could not find container \"acb4b35e27c9fea46808c7b39f46373bfad1ee22a81f59ed27129bdbdb1e635e\": container with ID starting with acb4b35e27c9fea46808c7b39f46373bfad1ee22a81f59ed27129bdbdb1e635e not found: ID does not exist"
May 25 12:21:19.351342 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:19.351316    2563 scope.go:117] "RemoveContainer" containerID="162e78c7390a74c0af4ef9d832afe3ecb0515907f66f7ec224c8f85c3154d2c3"
May 25 12:21:19.351512 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:21:19.351493    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"162e78c7390a74c0af4ef9d832afe3ecb0515907f66f7ec224c8f85c3154d2c3\": container with ID starting with 162e78c7390a74c0af4ef9d832afe3ecb0515907f66f7ec224c8f85c3154d2c3 not found: ID does not exist" containerID="162e78c7390a74c0af4ef9d832afe3ecb0515907f66f7ec224c8f85c3154d2c3"
May 25 12:21:19.351616 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:19.351532    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"162e78c7390a74c0af4ef9d832afe3ecb0515907f66f7ec224c8f85c3154d2c3"} err="failed to get container status \"162e78c7390a74c0af4ef9d832afe3ecb0515907f66f7ec224c8f85c3154d2c3\": rpc error: code = NotFound desc = could not find container \"162e78c7390a74c0af4ef9d832afe3ecb0515907f66f7ec224c8f85c3154d2c3\": container with ID starting with 162e78c7390a74c0af4ef9d832afe3ecb0515907f66f7ec224c8f85c3154d2c3 not found: ID does not exist"
May 25 12:21:19.351616 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:19.351558    2563 scope.go:117] "RemoveContainer" containerID="492459420e36ead56fbac6fe7e759c642e4f12ff03688e203759c10936ee884b"
May 25 12:21:19.351782 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:21:19.351767    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"492459420e36ead56fbac6fe7e759c642e4f12ff03688e203759c10936ee884b\": container with ID starting with 492459420e36ead56fbac6fe7e759c642e4f12ff03688e203759c10936ee884b not found: ID does not exist" containerID="492459420e36ead56fbac6fe7e759c642e4f12ff03688e203759c10936ee884b"
May 25 12:21:19.351821 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:19.351785    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"492459420e36ead56fbac6fe7e759c642e4f12ff03688e203759c10936ee884b"} err="failed to get container status \"492459420e36ead56fbac6fe7e759c642e4f12ff03688e203759c10936ee884b\": rpc error: code = NotFound desc = could not find container \"492459420e36ead56fbac6fe7e759c642e4f12ff03688e203759c10936ee884b\": container with ID starting with 492459420e36ead56fbac6fe7e759c642e4f12ff03688e203759c10936ee884b not found: ID does not exist"
May 25 12:21:19.351821 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:19.351798    2563 scope.go:117] "RemoveContainer" containerID="624217e7f8c3f20a403b40e467f939cc64a5b99504a2c64c1de3e782767ce3f8"
May 25 12:21:19.351976 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:21:19.351963    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"624217e7f8c3f20a403b40e467f939cc64a5b99504a2c64c1de3e782767ce3f8\": container with ID starting with 624217e7f8c3f20a403b40e467f939cc64a5b99504a2c64c1de3e782767ce3f8 not found: ID does not exist" containerID="624217e7f8c3f20a403b40e467f939cc64a5b99504a2c64c1de3e782767ce3f8"
May 25 12:21:19.352011 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:19.351978    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"624217e7f8c3f20a403b40e467f939cc64a5b99504a2c64c1de3e782767ce3f8"} err="failed to get container status \"624217e7f8c3f20a403b40e467f939cc64a5b99504a2c64c1de3e782767ce3f8\": rpc error: code = NotFound desc = could not find container \"624217e7f8c3f20a403b40e467f939cc64a5b99504a2c64c1de3e782767ce3f8\": container with ID starting with 624217e7f8c3f20a403b40e467f939cc64a5b99504a2c64c1de3e782767ce3f8 not found: ID does not exist"
May 25 12:21:20.637804 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:20.637767    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" path="/var/lib/kubelet/pods/e7c39c0c-8a3a-45b5-b369-20cf7b32c434/volumes"
May 25 12:21:21.249530 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:21.249479    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm" podUID="8fe1ba95-5b0a-4156-a667-a264c7dd7e5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
May 25 12:21:28.305084 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:28.305043    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk" podUID="cef298dd-6c2a-4607-9201-7dc8e42a58c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
May 25 12:21:31.250216 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:31.250175    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm" podUID="8fe1ba95-5b0a-4156-a667-a264c7dd7e5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
May 25 12:21:38.305317 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:38.305277    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk" podUID="cef298dd-6c2a-4607-9201-7dc8e42a58c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
May 25 12:21:41.249195 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:41.249158    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm" podUID="8fe1ba95-5b0a-4156-a667-a264c7dd7e5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
May 25 12:21:48.305667 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:48.305624    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk" podUID="cef298dd-6c2a-4607-9201-7dc8e42a58c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
May 25 12:21:51.249914 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:51.249878    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm" podUID="8fe1ba95-5b0a-4156-a667-a264c7dd7e5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
May 25 12:21:58.305618 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:21:58.305576    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk" podUID="cef298dd-6c2a-4607-9201-7dc8e42a58c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
May 25 12:22:01.250331 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:01.250299    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm"
May 25 12:22:08.305932 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:08.305891    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk" podUID="cef298dd-6c2a-4607-9201-7dc8e42a58c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
May 25 12:22:14.544613 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:14.544585    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw9bb_012c465e-2683-488a-abc1-18ad86529eb3/ovn-acl-logging/0.log"
May 25 12:22:14.546885 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:14.546860    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw9bb_012c465e-2683-488a-abc1-18ad86529eb3/ovn-acl-logging/0.log"
May 25 12:22:18.306378 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:18.306348    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk"
May 25 12:22:38.588706 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:38.588627    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm"]
May 25 12:22:38.589215 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:38.589005    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm" podUID="8fe1ba95-5b0a-4156-a667-a264c7dd7e5b" containerName="kserve-container" containerID="cri-o://e5cc5c68458309ed848541ccb2dc119404cdcd5cdd65d0ffeba9956f57cde891" gracePeriod=30
May 25 12:22:38.589215 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:38.589041    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm" podUID="8fe1ba95-5b0a-4156-a667-a264c7dd7e5b" containerName="kube-rbac-proxy" containerID="cri-o://1875b035b7e66c2bc58796082b5a934bb74743fc640304e0c80f75661b3c4189" gracePeriod=30
May 25 12:22:38.641281 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:38.641245    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7"]
May 25 12:22:38.641681 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:38.641662    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerName="kserve-container"
May 25 12:22:38.641788 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:38.641683    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerName="kserve-container"
May 25 12:22:38.641788 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:38.641704    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerName="storage-initializer"
May 25 12:22:38.641788 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:38.641714    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerName="storage-initializer"
May 25 12:22:38.641788 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:38.641725    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerName="agent"
May 25 12:22:38.641788 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:38.641734    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerName="agent"
May 25 12:22:38.641788 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:38.641762    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerName="kube-rbac-proxy"
May 25 12:22:38.641788 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:38.641770    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerName="kube-rbac-proxy"
May 25 12:22:38.642137 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:38.641832    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerName="kube-rbac-proxy"
May 25 12:22:38.642137 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:38.641847    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerName="kserve-container"
May 25 12:22:38.642137 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:38.641860    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="e7c39c0c-8a3a-45b5-b369-20cf7b32c434" containerName="agent"
May 25 12:22:38.644263 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:38.644236    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7"
May 25 12:22:38.646184 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:38.646159    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-94e63-predictor-serving-cert\""
May 25 12:22:38.646184 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:38.646177    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-94e63-kube-rbac-proxy-sar-config\""
May 25 12:22:38.651672 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:38.651651    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7"]
May 25 12:22:38.745089 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:38.745059    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk"]
May 25 12:22:38.745423 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:38.745366    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk" podUID="cef298dd-6c2a-4607-9201-7dc8e42a58c0" containerName="kserve-container" containerID="cri-o://080b58adab648fcf5797d8d41daca77d91a8c62c4d89f7ea2ba930ba27dd6285" gracePeriod=30
May 25 12:22:38.745699 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:38.745392    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk" podUID="cef298dd-6c2a-4607-9201-7dc8e42a58c0" containerName="kube-rbac-proxy" containerID="cri-o://7ac7abc2aa17131caba5de318e1a84cbbe99cfaddd0b5537f11963c0d282b9fb" gracePeriod=30
May 25 12:22:38.755744 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:38.755721    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87c07aa3-95d1-402c-9fde-901df344d3a2-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7\" (UID: \"87c07aa3-95d1-402c-9fde-901df344d3a2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7"
May 25 12:22:38.756074 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:38.755760    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnf9c\" (UniqueName: \"kubernetes.io/projected/87c07aa3-95d1-402c-9fde-901df344d3a2-kube-api-access-qnf9c\") pod \"isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7\" (UID: \"87c07aa3-95d1-402c-9fde-901df344d3a2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7"
May 25 12:22:38.756074 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:38.755835    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-graph-raw-hpa-94e63-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87c07aa3-95d1-402c-9fde-901df344d3a2-isvc-sklearn-graph-raw-hpa-94e63-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7\" (UID: \"87c07aa3-95d1-402c-9fde-901df344d3a2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7"
May 25 12:22:38.756074 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:38.755879    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87c07aa3-95d1-402c-9fde-901df344d3a2-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7\" (UID: \"87c07aa3-95d1-402c-9fde-901df344d3a2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7"
May 25 12:22:38.789647 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:38.789618    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8"]
May 25 12:22:38.791975 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:38.791958    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8"
May 25 12:22:38.793753 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:38.793730    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-94e63-predictor-serving-cert\""
May 25 12:22:38.793846 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:38.793761    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-94e63-kube-rbac-proxy-sar-config\""
May 25 12:22:38.802381 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:38.802354    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8"]
May 25 12:22:38.857240 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:38.857175    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qnf9c\" (UniqueName: \"kubernetes.io/projected/87c07aa3-95d1-402c-9fde-901df344d3a2-kube-api-access-qnf9c\") pod \"isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7\" (UID: \"87c07aa3-95d1-402c-9fde-901df344d3a2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7"
May 25 12:22:38.857240 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:38.857218    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-graph-raw-hpa-94e63-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87c07aa3-95d1-402c-9fde-901df344d3a2-isvc-sklearn-graph-raw-hpa-94e63-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7\" (UID: \"87c07aa3-95d1-402c-9fde-901df344d3a2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7"
May 25 12:22:38.857423 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:38.857242    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87c07aa3-95d1-402c-9fde-901df344d3a2-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7\" (UID: \"87c07aa3-95d1-402c-9fde-901df344d3a2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7"
May 25 12:22:38.857423 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:38.857316    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87c07aa3-95d1-402c-9fde-901df344d3a2-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7\" (UID: \"87c07aa3-95d1-402c-9fde-901df344d3a2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7"
May 25 12:22:38.857560 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:22:38.857429    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-serving-cert: secret "isvc-sklearn-graph-raw-hpa-94e63-predictor-serving-cert" not found
May 25 12:22:38.857560 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:22:38.857493    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87c07aa3-95d1-402c-9fde-901df344d3a2-proxy-tls podName:87c07aa3-95d1-402c-9fde-901df344d3a2 nodeName:}" failed. No retries permitted until 2026-05-25 12:22:39.357471652 +0000 UTC m=+625.324863281 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/87c07aa3-95d1-402c-9fde-901df344d3a2-proxy-tls") pod "isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7" (UID: "87c07aa3-95d1-402c-9fde-901df344d3a2") : secret "isvc-sklearn-graph-raw-hpa-94e63-predictor-serving-cert" not found
May 25 12:22:38.857781 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:38.857752    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87c07aa3-95d1-402c-9fde-901df344d3a2-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7\" (UID: \"87c07aa3-95d1-402c-9fde-901df344d3a2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7"
May 25 12:22:38.858010 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:38.857986    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-graph-raw-hpa-94e63-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87c07aa3-95d1-402c-9fde-901df344d3a2-isvc-sklearn-graph-raw-hpa-94e63-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7\" (UID: \"87c07aa3-95d1-402c-9fde-901df344d3a2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7"
May 25 12:22:38.865487 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:38.865462    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnf9c\" (UniqueName: \"kubernetes.io/projected/87c07aa3-95d1-402c-9fde-901df344d3a2-kube-api-access-qnf9c\") pod \"isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7\" (UID: \"87c07aa3-95d1-402c-9fde-901df344d3a2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7"
May 25 12:22:38.958697 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:38.958659    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmsr4\" (UniqueName: \"kubernetes.io/projected/ad22a875-9798-4221-ab20-ff7c796f50df-kube-api-access-lmsr4\") pod \"isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8\" (UID: \"ad22a875-9798-4221-ab20-ff7c796f50df\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8"
May 25 12:22:38.958861 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:38.958709    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad22a875-9798-4221-ab20-ff7c796f50df-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8\" (UID: \"ad22a875-9798-4221-ab20-ff7c796f50df\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8"
May 25 12:22:38.958861 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:38.958730    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad22a875-9798-4221-ab20-ff7c796f50df-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8\" (UID: \"ad22a875-9798-4221-ab20-ff7c796f50df\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8"
May 25 12:22:38.958941 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:38.958851    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-graph-raw-hpa-94e63-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad22a875-9798-4221-ab20-ff7c796f50df-isvc-xgboost-graph-raw-hpa-94e63-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8\" (UID: \"ad22a875-9798-4221-ab20-ff7c796f50df\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8"
May 25 12:22:39.059262 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:39.059225    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lmsr4\" (UniqueName: \"kubernetes.io/projected/ad22a875-9798-4221-ab20-ff7c796f50df-kube-api-access-lmsr4\") pod \"isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8\" (UID: \"ad22a875-9798-4221-ab20-ff7c796f50df\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8"
May 25 12:22:39.059429 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:39.059274    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad22a875-9798-4221-ab20-ff7c796f50df-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8\" (UID: \"ad22a875-9798-4221-ab20-ff7c796f50df\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8"
May 25 12:22:39.059429 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:39.059301    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad22a875-9798-4221-ab20-ff7c796f50df-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8\" (UID: \"ad22a875-9798-4221-ab20-ff7c796f50df\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8"
May 25 12:22:39.059429 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:39.059369    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-graph-raw-hpa-94e63-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad22a875-9798-4221-ab20-ff7c796f50df-isvc-xgboost-graph-raw-hpa-94e63-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8\" (UID: \"ad22a875-9798-4221-ab20-ff7c796f50df\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8"
May 25 12:22:39.059597 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:22:39.059481    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-serving-cert: secret "isvc-xgboost-graph-raw-hpa-94e63-predictor-serving-cert" not found
May 25 12:22:39.059597 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:22:39.059575    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad22a875-9798-4221-ab20-ff7c796f50df-proxy-tls podName:ad22a875-9798-4221-ab20-ff7c796f50df nodeName:}" failed. No retries permitted until 2026-05-25 12:22:39.559555274 +0000 UTC m=+625.526946898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ad22a875-9798-4221-ab20-ff7c796f50df-proxy-tls") pod "isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8" (UID: "ad22a875-9798-4221-ab20-ff7c796f50df") : secret "isvc-xgboost-graph-raw-hpa-94e63-predictor-serving-cert" not found
May 25 12:22:39.059741 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:39.059721    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad22a875-9798-4221-ab20-ff7c796f50df-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8\" (UID: \"ad22a875-9798-4221-ab20-ff7c796f50df\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8"
May 25 12:22:39.059992 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:39.059975    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-graph-raw-hpa-94e63-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad22a875-9798-4221-ab20-ff7c796f50df-isvc-xgboost-graph-raw-hpa-94e63-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8\" (UID: \"ad22a875-9798-4221-ab20-ff7c796f50df\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8"
May 25 12:22:39.070140 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:39.070114    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmsr4\" (UniqueName: \"kubernetes.io/projected/ad22a875-9798-4221-ab20-ff7c796f50df-kube-api-access-lmsr4\") pod \"isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8\" (UID: \"ad22a875-9798-4221-ab20-ff7c796f50df\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8"
May 25 12:22:39.362585 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:39.362553    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87c07aa3-95d1-402c-9fde-901df344d3a2-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7\" (UID: \"87c07aa3-95d1-402c-9fde-901df344d3a2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7"
May 25 12:22:39.364981 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:39.364953    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87c07aa3-95d1-402c-9fde-901df344d3a2-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7\" (UID: \"87c07aa3-95d1-402c-9fde-901df344d3a2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7"
May 25 12:22:39.551015 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:39.550982    2563 generic.go:358] "Generic (PLEG): container finished" podID="8fe1ba95-5b0a-4156-a667-a264c7dd7e5b" containerID="1875b035b7e66c2bc58796082b5a934bb74743fc640304e0c80f75661b3c4189" exitCode=2
May 25 12:22:39.551194 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:39.551057    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm" event={"ID":"8fe1ba95-5b0a-4156-a667-a264c7dd7e5b","Type":"ContainerDied","Data":"1875b035b7e66c2bc58796082b5a934bb74743fc640304e0c80f75661b3c4189"}
May 25 12:22:39.552803 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:39.552782    2563 generic.go:358] "Generic (PLEG): container finished" podID="cef298dd-6c2a-4607-9201-7dc8e42a58c0" containerID="7ac7abc2aa17131caba5de318e1a84cbbe99cfaddd0b5537f11963c0d282b9fb" exitCode=2
May 25 12:22:39.552912 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:39.552818    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk" event={"ID":"cef298dd-6c2a-4607-9201-7dc8e42a58c0","Type":"ContainerDied","Data":"7ac7abc2aa17131caba5de318e1a84cbbe99cfaddd0b5537f11963c0d282b9fb"}
May 25 12:22:39.559004 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:39.558980    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7"
May 25 12:22:39.566367 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:39.566340    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad22a875-9798-4221-ab20-ff7c796f50df-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8\" (UID: \"ad22a875-9798-4221-ab20-ff7c796f50df\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8"
May 25 12:22:39.568776 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:39.568756    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad22a875-9798-4221-ab20-ff7c796f50df-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8\" (UID: \"ad22a875-9798-4221-ab20-ff7c796f50df\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8"
May 25 12:22:39.689203 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:39.689059    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7"]
May 25 12:22:39.691803 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:22:39.691772    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87c07aa3_95d1_402c_9fde_901df344d3a2.slice/crio-8904772ac3114f19b72efb3560c622595fcee11e639483f871bcd6e03593c1f2 WatchSource:0}: Error finding container 8904772ac3114f19b72efb3560c622595fcee11e639483f871bcd6e03593c1f2: Status 404 returned error can't find the container with id 8904772ac3114f19b72efb3560c622595fcee11e639483f871bcd6e03593c1f2
May 25 12:22:39.702670 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:39.702633    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8"
May 25 12:22:39.833644 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:39.833606    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8"]
May 25 12:22:39.836758 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:22:39.836732    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad22a875_9798_4221_ab20_ff7c796f50df.slice/crio-d894542e708f195c7945587d6d72c394aeeac7c2a24dbf2c1173db70677383a0 WatchSource:0}: Error finding container d894542e708f195c7945587d6d72c394aeeac7c2a24dbf2c1173db70677383a0: Status 404 returned error can't find the container with id d894542e708f195c7945587d6d72c394aeeac7c2a24dbf2c1173db70677383a0
May 25 12:22:40.557225 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:40.557187    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8" event={"ID":"ad22a875-9798-4221-ab20-ff7c796f50df","Type":"ContainerStarted","Data":"16aed49997e21b963eeada1282864901f261f8a13ba4521ed2346c8ef1965b4f"}
May 25 12:22:40.557225 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:40.557224    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8" event={"ID":"ad22a875-9798-4221-ab20-ff7c796f50df","Type":"ContainerStarted","Data":"d894542e708f195c7945587d6d72c394aeeac7c2a24dbf2c1173db70677383a0"}
May 25 12:22:40.558421 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:40.558401    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7" event={"ID":"87c07aa3-95d1-402c-9fde-901df344d3a2","Type":"ContainerStarted","Data":"11b82069b2b1c95b50734c9253df8dd59ba0e59bb1205a2796438f5c13926b06"}
May 25 12:22:40.558510 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:40.558425    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7" event={"ID":"87c07aa3-95d1-402c-9fde-901df344d3a2","Type":"ContainerStarted","Data":"8904772ac3114f19b72efb3560c622595fcee11e639483f871bcd6e03593c1f2"}
May 25 12:22:41.242731 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:41.242687    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm" podUID="8fe1ba95-5b0a-4156-a667-a264c7dd7e5b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
May 25 12:22:41.249491 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:41.249463    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm" podUID="8fe1ba95-5b0a-4156-a667-a264c7dd7e5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
May 25 12:22:42.382802 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:42.382777    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk"
May 25 12:22:42.491640 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:42.491613    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4x6ms\" (UniqueName: \"kubernetes.io/projected/cef298dd-6c2a-4607-9201-7dc8e42a58c0-kube-api-access-4x6ms\") pod \"cef298dd-6c2a-4607-9201-7dc8e42a58c0\" (UID: \"cef298dd-6c2a-4607-9201-7dc8e42a58c0\") "
May 25 12:22:42.491785 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:42.491671    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-graph-raw-a7098-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cef298dd-6c2a-4607-9201-7dc8e42a58c0-isvc-xgboost-graph-raw-a7098-kube-rbac-proxy-sar-config\") pod \"cef298dd-6c2a-4607-9201-7dc8e42a58c0\" (UID: \"cef298dd-6c2a-4607-9201-7dc8e42a58c0\") "
May 25 12:22:42.491785 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:42.491715    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cef298dd-6c2a-4607-9201-7dc8e42a58c0-kserve-provision-location\") pod \"cef298dd-6c2a-4607-9201-7dc8e42a58c0\" (UID: \"cef298dd-6c2a-4607-9201-7dc8e42a58c0\") "
May 25 12:22:42.491894 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:42.491872    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cef298dd-6c2a-4607-9201-7dc8e42a58c0-proxy-tls\") pod \"cef298dd-6c2a-4607-9201-7dc8e42a58c0\" (UID: \"cef298dd-6c2a-4607-9201-7dc8e42a58c0\") "
May 25 12:22:42.492082 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:42.492057    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef298dd-6c2a-4607-9201-7dc8e42a58c0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cef298dd-6c2a-4607-9201-7dc8e42a58c0" (UID: "cef298dd-6c2a-4607-9201-7dc8e42a58c0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:22:42.492178 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:42.492080    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cef298dd-6c2a-4607-9201-7dc8e42a58c0-isvc-xgboost-graph-raw-a7098-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-graph-raw-a7098-kube-rbac-proxy-sar-config") pod "cef298dd-6c2a-4607-9201-7dc8e42a58c0" (UID: "cef298dd-6c2a-4607-9201-7dc8e42a58c0"). InnerVolumeSpecName "isvc-xgboost-graph-raw-a7098-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:22:42.493710 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:42.493687    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cef298dd-6c2a-4607-9201-7dc8e42a58c0-kube-api-access-4x6ms" (OuterVolumeSpecName: "kube-api-access-4x6ms") pod "cef298dd-6c2a-4607-9201-7dc8e42a58c0" (UID: "cef298dd-6c2a-4607-9201-7dc8e42a58c0"). InnerVolumeSpecName "kube-api-access-4x6ms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:22:42.493841 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:42.493821    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cef298dd-6c2a-4607-9201-7dc8e42a58c0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cef298dd-6c2a-4607-9201-7dc8e42a58c0" (UID: "cef298dd-6c2a-4607-9201-7dc8e42a58c0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:22:42.566606 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:42.566566    2563 generic.go:358] "Generic (PLEG): container finished" podID="cef298dd-6c2a-4607-9201-7dc8e42a58c0" containerID="080b58adab648fcf5797d8d41daca77d91a8c62c4d89f7ea2ba930ba27dd6285" exitCode=0
May 25 12:22:42.566759 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:42.566643    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk" event={"ID":"cef298dd-6c2a-4607-9201-7dc8e42a58c0","Type":"ContainerDied","Data":"080b58adab648fcf5797d8d41daca77d91a8c62c4d89f7ea2ba930ba27dd6285"}
May 25 12:22:42.566759 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:42.566661    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk"
May 25 12:22:42.566759 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:42.566682    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk" event={"ID":"cef298dd-6c2a-4607-9201-7dc8e42a58c0","Type":"ContainerDied","Data":"1af898cdf391f262afd082cc7609e622752486aa2ac5db41e11cca3383fd5e14"}
May 25 12:22:42.566759 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:42.566697    2563 scope.go:117] "RemoveContainer" containerID="7ac7abc2aa17131caba5de318e1a84cbbe99cfaddd0b5537f11963c0d282b9fb"
May 25 12:22:42.574512 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:42.574496    2563 scope.go:117] "RemoveContainer" containerID="080b58adab648fcf5797d8d41daca77d91a8c62c4d89f7ea2ba930ba27dd6285"
May 25 12:22:42.581254 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:42.581238    2563 scope.go:117] "RemoveContainer" containerID="6c25db827f5c05265344beb2a54971d58c406b21a1561094e2c4201156ae6c0c"
May 25 12:22:42.588074 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:42.588055    2563 scope.go:117] "RemoveContainer" containerID="7ac7abc2aa17131caba5de318e1a84cbbe99cfaddd0b5537f11963c0d282b9fb"
May 25 12:22:42.588342 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:22:42.588320    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ac7abc2aa17131caba5de318e1a84cbbe99cfaddd0b5537f11963c0d282b9fb\": container with ID starting with 7ac7abc2aa17131caba5de318e1a84cbbe99cfaddd0b5537f11963c0d282b9fb not found: ID does not exist" containerID="7ac7abc2aa17131caba5de318e1a84cbbe99cfaddd0b5537f11963c0d282b9fb"
May 25 12:22:42.588399 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:42.588345    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ac7abc2aa17131caba5de318e1a84cbbe99cfaddd0b5537f11963c0d282b9fb"} err="failed to get container status \"7ac7abc2aa17131caba5de318e1a84cbbe99cfaddd0b5537f11963c0d282b9fb\": rpc error: code = NotFound desc = could not find container \"7ac7abc2aa17131caba5de318e1a84cbbe99cfaddd0b5537f11963c0d282b9fb\": container with ID starting with 7ac7abc2aa17131caba5de318e1a84cbbe99cfaddd0b5537f11963c0d282b9fb not found: ID does not exist"
May 25 12:22:42.588399 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:42.588372    2563 scope.go:117] "RemoveContainer" containerID="080b58adab648fcf5797d8d41daca77d91a8c62c4d89f7ea2ba930ba27dd6285"
May 25 12:22:42.588683 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:22:42.588668    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"080b58adab648fcf5797d8d41daca77d91a8c62c4d89f7ea2ba930ba27dd6285\": container with ID starting with 080b58adab648fcf5797d8d41daca77d91a8c62c4d89f7ea2ba930ba27dd6285 not found: ID does not exist" containerID="080b58adab648fcf5797d8d41daca77d91a8c62c4d89f7ea2ba930ba27dd6285"
May 25 12:22:42.588735 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:42.588687    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"080b58adab648fcf5797d8d41daca77d91a8c62c4d89f7ea2ba930ba27dd6285"} err="failed to get container status \"080b58adab648fcf5797d8d41daca77d91a8c62c4d89f7ea2ba930ba27dd6285\": rpc error: code = NotFound desc = could not find container \"080b58adab648fcf5797d8d41daca77d91a8c62c4d89f7ea2ba930ba27dd6285\": container with ID starting with 080b58adab648fcf5797d8d41daca77d91a8c62c4d89f7ea2ba930ba27dd6285 not found: ID does not exist"
May 25 12:22:42.588735 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:42.588700    2563 scope.go:117] "RemoveContainer" containerID="6c25db827f5c05265344beb2a54971d58c406b21a1561094e2c4201156ae6c0c"
May 25 12:22:42.588878 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:42.588861    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk"]
May 25 12:22:42.588924 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:22:42.588913    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c25db827f5c05265344beb2a54971d58c406b21a1561094e2c4201156ae6c0c\": container with ID starting with 6c25db827f5c05265344beb2a54971d58c406b21a1561094e2c4201156ae6c0c not found: ID does not exist" containerID="6c25db827f5c05265344beb2a54971d58c406b21a1561094e2c4201156ae6c0c"
May 25 12:22:42.588956 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:42.588930    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c25db827f5c05265344beb2a54971d58c406b21a1561094e2c4201156ae6c0c"} err="failed to get container status \"6c25db827f5c05265344beb2a54971d58c406b21a1561094e2c4201156ae6c0c\": rpc error: code = NotFound desc = could not find container \"6c25db827f5c05265344beb2a54971d58c406b21a1561094e2c4201156ae6c0c\": container with ID starting with 6c25db827f5c05265344beb2a54971d58c406b21a1561094e2c4201156ae6c0c not found: ID does not exist"
May 25 12:22:42.592855 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:42.592836    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7098-predictor-8f8798c7-qvtkk"]
May 25 12:22:42.593047 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:42.593029    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cef298dd-6c2a-4607-9201-7dc8e42a58c0-proxy-tls\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:22:42.593096 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:42.593053    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x6ms\" (UniqueName: \"kubernetes.io/projected/cef298dd-6c2a-4607-9201-7dc8e42a58c0-kube-api-access-4x6ms\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:22:42.593096 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:42.593069    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-graph-raw-a7098-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cef298dd-6c2a-4607-9201-7dc8e42a58c0-isvc-xgboost-graph-raw-a7098-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:22:42.593096 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:42.593081    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cef298dd-6c2a-4607-9201-7dc8e42a58c0-kserve-provision-location\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:22:42.636951 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:42.636925    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cef298dd-6c2a-4607-9201-7dc8e42a58c0" path="/var/lib/kubelet/pods/cef298dd-6c2a-4607-9201-7dc8e42a58c0/volumes"
May 25 12:22:43.118899 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:43.118878    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm"
May 25 12:22:43.301731 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:43.301694    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fe1ba95-5b0a-4156-a667-a264c7dd7e5b-kserve-provision-location\") pod \"8fe1ba95-5b0a-4156-a667-a264c7dd7e5b\" (UID: \"8fe1ba95-5b0a-4156-a667-a264c7dd7e5b\") "
May 25 12:22:43.301908 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:43.301747    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-graph-raw-a7098-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fe1ba95-5b0a-4156-a667-a264c7dd7e5b-isvc-sklearn-graph-raw-a7098-kube-rbac-proxy-sar-config\") pod \"8fe1ba95-5b0a-4156-a667-a264c7dd7e5b\" (UID: \"8fe1ba95-5b0a-4156-a667-a264c7dd7e5b\") "
May 25 12:22:43.301908 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:43.301823    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qw2mc\" (UniqueName: \"kubernetes.io/projected/8fe1ba95-5b0a-4156-a667-a264c7dd7e5b-kube-api-access-qw2mc\") pod \"8fe1ba95-5b0a-4156-a667-a264c7dd7e5b\" (UID: \"8fe1ba95-5b0a-4156-a667-a264c7dd7e5b\") "
May 25 12:22:43.301908 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:43.301866    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fe1ba95-5b0a-4156-a667-a264c7dd7e5b-proxy-tls\") pod \"8fe1ba95-5b0a-4156-a667-a264c7dd7e5b\" (UID: \"8fe1ba95-5b0a-4156-a667-a264c7dd7e5b\") "
May 25 12:22:43.302087 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:43.302056    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fe1ba95-5b0a-4156-a667-a264c7dd7e5b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8fe1ba95-5b0a-4156-a667-a264c7dd7e5b" (UID: "8fe1ba95-5b0a-4156-a667-a264c7dd7e5b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:22:43.302135 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:43.302120    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fe1ba95-5b0a-4156-a667-a264c7dd7e5b-isvc-sklearn-graph-raw-a7098-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-graph-raw-a7098-kube-rbac-proxy-sar-config") pod "8fe1ba95-5b0a-4156-a667-a264c7dd7e5b" (UID: "8fe1ba95-5b0a-4156-a667-a264c7dd7e5b"). InnerVolumeSpecName "isvc-sklearn-graph-raw-a7098-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:22:43.303927 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:43.303902    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fe1ba95-5b0a-4156-a667-a264c7dd7e5b-kube-api-access-qw2mc" (OuterVolumeSpecName: "kube-api-access-qw2mc") pod "8fe1ba95-5b0a-4156-a667-a264c7dd7e5b" (UID: "8fe1ba95-5b0a-4156-a667-a264c7dd7e5b"). InnerVolumeSpecName "kube-api-access-qw2mc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:22:43.304001 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:43.303968    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fe1ba95-5b0a-4156-a667-a264c7dd7e5b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8fe1ba95-5b0a-4156-a667-a264c7dd7e5b" (UID: "8fe1ba95-5b0a-4156-a667-a264c7dd7e5b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:22:43.402768 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:43.402741    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw2mc\" (UniqueName: \"kubernetes.io/projected/8fe1ba95-5b0a-4156-a667-a264c7dd7e5b-kube-api-access-qw2mc\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:22:43.402768 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:43.402770    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fe1ba95-5b0a-4156-a667-a264c7dd7e5b-proxy-tls\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:22:43.403073 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:43.402780    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fe1ba95-5b0a-4156-a667-a264c7dd7e5b-kserve-provision-location\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:22:43.403073 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:43.402790    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-graph-raw-a7098-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fe1ba95-5b0a-4156-a667-a264c7dd7e5b-isvc-sklearn-graph-raw-a7098-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:22:43.572392 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:43.572355    2563 generic.go:358] "Generic (PLEG): container finished" podID="87c07aa3-95d1-402c-9fde-901df344d3a2" containerID="11b82069b2b1c95b50734c9253df8dd59ba0e59bb1205a2796438f5c13926b06" exitCode=0
May 25 12:22:43.572569 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:43.572429    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7" event={"ID":"87c07aa3-95d1-402c-9fde-901df344d3a2","Type":"ContainerDied","Data":"11b82069b2b1c95b50734c9253df8dd59ba0e59bb1205a2796438f5c13926b06"}
May 25 12:22:43.574125 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:43.574101    2563 generic.go:358] "Generic (PLEG): container finished" podID="8fe1ba95-5b0a-4156-a667-a264c7dd7e5b" containerID="e5cc5c68458309ed848541ccb2dc119404cdcd5cdd65d0ffeba9956f57cde891" exitCode=0
May 25 12:22:43.574260 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:43.574173    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm"
May 25 12:22:43.574375 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:43.574174    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm" event={"ID":"8fe1ba95-5b0a-4156-a667-a264c7dd7e5b","Type":"ContainerDied","Data":"e5cc5c68458309ed848541ccb2dc119404cdcd5cdd65d0ffeba9956f57cde891"}
May 25 12:22:43.574375 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:43.574297    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm" event={"ID":"8fe1ba95-5b0a-4156-a667-a264c7dd7e5b","Type":"ContainerDied","Data":"421a391a401c45b06e097fa97e70c9bae5ba34593d333cd0fac1f61717351ff0"}
May 25 12:22:43.574375 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:43.574321    2563 scope.go:117] "RemoveContainer" containerID="1875b035b7e66c2bc58796082b5a934bb74743fc640304e0c80f75661b3c4189"
May 25 12:22:43.575569 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:43.575547    2563 generic.go:358] "Generic (PLEG): container finished" podID="ad22a875-9798-4221-ab20-ff7c796f50df" containerID="16aed49997e21b963eeada1282864901f261f8a13ba4521ed2346c8ef1965b4f" exitCode=0
May 25 12:22:43.575658 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:43.575580    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8" event={"ID":"ad22a875-9798-4221-ab20-ff7c796f50df","Type":"ContainerDied","Data":"16aed49997e21b963eeada1282864901f261f8a13ba4521ed2346c8ef1965b4f"}
May 25 12:22:43.586481 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:43.586462    2563 scope.go:117] "RemoveContainer" containerID="e5cc5c68458309ed848541ccb2dc119404cdcd5cdd65d0ffeba9956f57cde891"
May 25 12:22:43.595424 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:43.595314    2563 scope.go:117] "RemoveContainer" containerID="8c354d25405efa568e436b73d819292b46b3504dcd04dcb0ed68a302ac5d0de4"
May 25 12:22:43.604846 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:43.604752    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm"]
May 25 12:22:43.606162 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:43.606141    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7098-predictor-6c4dc45985-6shnm"]
May 25 12:22:43.611923 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:43.611876    2563 scope.go:117] "RemoveContainer" containerID="1875b035b7e66c2bc58796082b5a934bb74743fc640304e0c80f75661b3c4189"
May 25 12:22:43.612416 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:22:43.612395    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1875b035b7e66c2bc58796082b5a934bb74743fc640304e0c80f75661b3c4189\": container with ID starting with 1875b035b7e66c2bc58796082b5a934bb74743fc640304e0c80f75661b3c4189 not found: ID does not exist" containerID="1875b035b7e66c2bc58796082b5a934bb74743fc640304e0c80f75661b3c4189"
May 25 12:22:43.612625 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:43.612427    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1875b035b7e66c2bc58796082b5a934bb74743fc640304e0c80f75661b3c4189"} err="failed to get container status \"1875b035b7e66c2bc58796082b5a934bb74743fc640304e0c80f75661b3c4189\": rpc error: code = NotFound desc = could not find container \"1875b035b7e66c2bc58796082b5a934bb74743fc640304e0c80f75661b3c4189\": container with ID starting with 1875b035b7e66c2bc58796082b5a934bb74743fc640304e0c80f75661b3c4189 not found: ID does not exist"
May 25 12:22:43.612625 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:43.612452    2563 scope.go:117] "RemoveContainer" containerID="e5cc5c68458309ed848541ccb2dc119404cdcd5cdd65d0ffeba9956f57cde891"
May 25 12:22:43.612801 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:22:43.612777    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5cc5c68458309ed848541ccb2dc119404cdcd5cdd65d0ffeba9956f57cde891\": container with ID starting with e5cc5c68458309ed848541ccb2dc119404cdcd5cdd65d0ffeba9956f57cde891 not found: ID does not exist" containerID="e5cc5c68458309ed848541ccb2dc119404cdcd5cdd65d0ffeba9956f57cde891"
May 25 12:22:43.612843 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:43.612813    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5cc5c68458309ed848541ccb2dc119404cdcd5cdd65d0ffeba9956f57cde891"} err="failed to get container status \"e5cc5c68458309ed848541ccb2dc119404cdcd5cdd65d0ffeba9956f57cde891\": rpc error: code = NotFound desc = could not find container \"e5cc5c68458309ed848541ccb2dc119404cdcd5cdd65d0ffeba9956f57cde891\": container with ID starting with e5cc5c68458309ed848541ccb2dc119404cdcd5cdd65d0ffeba9956f57cde891 not found: ID does not exist"
May 25 12:22:43.612843 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:43.612837    2563 scope.go:117] "RemoveContainer" containerID="8c354d25405efa568e436b73d819292b46b3504dcd04dcb0ed68a302ac5d0de4"
May 25 12:22:43.613155 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:22:43.613105    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c354d25405efa568e436b73d819292b46b3504dcd04dcb0ed68a302ac5d0de4\": container with ID starting with 8c354d25405efa568e436b73d819292b46b3504dcd04dcb0ed68a302ac5d0de4 not found: ID does not exist" containerID="8c354d25405efa568e436b73d819292b46b3504dcd04dcb0ed68a302ac5d0de4"
May 25 12:22:43.613155 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:43.613130    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c354d25405efa568e436b73d819292b46b3504dcd04dcb0ed68a302ac5d0de4"} err="failed to get container status \"8c354d25405efa568e436b73d819292b46b3504dcd04dcb0ed68a302ac5d0de4\": rpc error: code = NotFound desc = could not find container \"8c354d25405efa568e436b73d819292b46b3504dcd04dcb0ed68a302ac5d0de4\": container with ID starting with 8c354d25405efa568e436b73d819292b46b3504dcd04dcb0ed68a302ac5d0de4 not found: ID does not exist"
May 25 12:22:44.580503 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:44.580468    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7" event={"ID":"87c07aa3-95d1-402c-9fde-901df344d3a2","Type":"ContainerStarted","Data":"26cd7a3d1ba59c6ce204e3c33a233d804e9a9cf2cdfb5ba0fab0f542efc58e63"}
May 25 12:22:44.580952 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:44.580510    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7" event={"ID":"87c07aa3-95d1-402c-9fde-901df344d3a2","Type":"ContainerStarted","Data":"9d537a39b11741ba1617eaf48add8074efd75ea337a91c369b85f3fc22bb7bfb"}
May 25 12:22:44.580952 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:44.580866    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7"
May 25 12:22:44.580952 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:44.580898    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7"
May 25 12:22:44.582330 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:44.582294    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7" podUID="87c07aa3-95d1-402c-9fde-901df344d3a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
May 25 12:22:44.583101 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:44.583080    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8" event={"ID":"ad22a875-9798-4221-ab20-ff7c796f50df","Type":"ContainerStarted","Data":"51cfd34a4b38c187df3c387f26b80aa3ddc07233580433f10435f712a359f88a"}
May 25 12:22:44.583176 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:44.583104    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8" event={"ID":"ad22a875-9798-4221-ab20-ff7c796f50df","Type":"ContainerStarted","Data":"15d93f716d54ab65f271deda5eecf78272ad81290c5b20d0880115dbc22ca16c"}
May 25 12:22:44.583304 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:44.583289    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8"
May 25 12:22:44.598604 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:44.598561    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7" podStartSLOduration=6.598545827 podStartE2EDuration="6.598545827s" podCreationTimestamp="2026-05-25 12:22:38 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:22:44.596472851 +0000 UTC m=+630.563864493" watchObservedRunningTime="2026-05-25 12:22:44.598545827 +0000 UTC m=+630.565937465"
May 25 12:22:44.612896 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:44.612848    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8" podStartSLOduration=6.612831614 podStartE2EDuration="6.612831614s" podCreationTimestamp="2026-05-25 12:22:38 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:22:44.61170215 +0000 UTC m=+630.579093801" watchObservedRunningTime="2026-05-25 12:22:44.612831614 +0000 UTC m=+630.580223254"
May 25 12:22:44.636994 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:44.636968    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fe1ba95-5b0a-4156-a667-a264c7dd7e5b" path="/var/lib/kubelet/pods/8fe1ba95-5b0a-4156-a667-a264c7dd7e5b/volumes"
May 25 12:22:45.586569 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:45.586505    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7" podUID="87c07aa3-95d1-402c-9fde-901df344d3a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
May 25 12:22:45.586999 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:45.586615    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8"
May 25 12:22:45.587623 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:45.587601    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8" podUID="ad22a875-9798-4221-ab20-ff7c796f50df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
May 25 12:22:46.589579 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:46.589539    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8" podUID="ad22a875-9798-4221-ab20-ff7c796f50df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
May 25 12:22:50.591138 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:50.591107    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7"
May 25 12:22:50.591762 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:50.591733    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7" podUID="87c07aa3-95d1-402c-9fde-901df344d3a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
May 25 12:22:51.593350 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:51.593310    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8"
May 25 12:22:51.593953 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:22:51.593923    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8" podUID="ad22a875-9798-4221-ab20-ff7c796f50df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
May 25 12:23:00.592673 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:23:00.592628    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7" podUID="87c07aa3-95d1-402c-9fde-901df344d3a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
May 25 12:23:01.594257 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:23:01.594210    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8" podUID="ad22a875-9798-4221-ab20-ff7c796f50df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
May 25 12:23:10.592254 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:23:10.592213    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7" podUID="87c07aa3-95d1-402c-9fde-901df344d3a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
May 25 12:23:11.593844 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:23:11.593803    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8" podUID="ad22a875-9798-4221-ab20-ff7c796f50df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
May 25 12:23:20.591870 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:23:20.591827    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7" podUID="87c07aa3-95d1-402c-9fde-901df344d3a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
May 25 12:23:21.594197 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:23:21.594159    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8" podUID="ad22a875-9798-4221-ab20-ff7c796f50df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
May 25 12:23:30.591871 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:23:30.591826    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7" podUID="87c07aa3-95d1-402c-9fde-901df344d3a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
May 25 12:23:31.594313 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:23:31.594276    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8" podUID="ad22a875-9798-4221-ab20-ff7c796f50df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
May 25 12:23:40.591843 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:23:40.591802    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7" podUID="87c07aa3-95d1-402c-9fde-901df344d3a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
May 25 12:23:41.593891 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:23:41.593853    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8" podUID="ad22a875-9798-4221-ab20-ff7c796f50df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
May 25 12:23:50.592215 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:23:50.592180    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7"
May 25 12:23:51.594677 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:23:51.594639    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8"
May 25 12:24:18.952185 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:18.952104    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7"]
May 25 12:24:18.952670 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:18.952504    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7" podUID="87c07aa3-95d1-402c-9fde-901df344d3a2" containerName="kserve-container" containerID="cri-o://9d537a39b11741ba1617eaf48add8074efd75ea337a91c369b85f3fc22bb7bfb" gracePeriod=30
May 25 12:24:18.952670 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:18.952570    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7" podUID="87c07aa3-95d1-402c-9fde-901df344d3a2" containerName="kube-rbac-proxy" containerID="cri-o://26cd7a3d1ba59c6ce204e3c33a233d804e9a9cf2cdfb5ba0fab0f542efc58e63" gracePeriod=30
May 25 12:24:19.063365 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:19.063330    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8"]
May 25 12:24:19.063798 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:19.063772    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8" podUID="ad22a875-9798-4221-ab20-ff7c796f50df" containerName="kserve-container" containerID="cri-o://15d93f716d54ab65f271deda5eecf78272ad81290c5b20d0880115dbc22ca16c" gracePeriod=30
May 25 12:24:19.063895 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:19.063812    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8" podUID="ad22a875-9798-4221-ab20-ff7c796f50df" containerName="kube-rbac-proxy" containerID="cri-o://51cfd34a4b38c187df3c387f26b80aa3ddc07233580433f10435f712a359f88a" gracePeriod=30
May 25 12:24:19.843041 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:19.843005    2563 generic.go:358] "Generic (PLEG): container finished" podID="ad22a875-9798-4221-ab20-ff7c796f50df" containerID="51cfd34a4b38c187df3c387f26b80aa3ddc07233580433f10435f712a359f88a" exitCode=2
May 25 12:24:19.843219 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:19.843077    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8" event={"ID":"ad22a875-9798-4221-ab20-ff7c796f50df","Type":"ContainerDied","Data":"51cfd34a4b38c187df3c387f26b80aa3ddc07233580433f10435f712a359f88a"}
May 25 12:24:19.844760 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:19.844740    2563 generic.go:358] "Generic (PLEG): container finished" podID="87c07aa3-95d1-402c-9fde-901df344d3a2" containerID="26cd7a3d1ba59c6ce204e3c33a233d804e9a9cf2cdfb5ba0fab0f542efc58e63" exitCode=2
May 25 12:24:19.844823 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:19.844774    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7" event={"ID":"87c07aa3-95d1-402c-9fde-901df344d3a2","Type":"ContainerDied","Data":"26cd7a3d1ba59c6ce204e3c33a233d804e9a9cf2cdfb5ba0fab0f542efc58e63"}
May 25 12:24:20.586954 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:20.586911    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7" podUID="87c07aa3-95d1-402c-9fde-901df344d3a2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
May 25 12:24:20.592241 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:20.592211    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7" podUID="87c07aa3-95d1-402c-9fde-901df344d3a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
May 25 12:24:21.590737 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:21.590695    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8" podUID="ad22a875-9798-4221-ab20-ff7c796f50df" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
May 25 12:24:21.593987 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:21.593963    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8" podUID="ad22a875-9798-4221-ab20-ff7c796f50df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
May 25 12:24:22.703681 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:22.703653    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8"
May 25 12:24:22.801304 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:22.801215    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad22a875-9798-4221-ab20-ff7c796f50df-proxy-tls\") pod \"ad22a875-9798-4221-ab20-ff7c796f50df\" (UID: \"ad22a875-9798-4221-ab20-ff7c796f50df\") "
May 25 12:24:22.801304 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:22.801277    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad22a875-9798-4221-ab20-ff7c796f50df-kserve-provision-location\") pod \"ad22a875-9798-4221-ab20-ff7c796f50df\" (UID: \"ad22a875-9798-4221-ab20-ff7c796f50df\") "
May 25 12:24:22.801304 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:22.801306    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmsr4\" (UniqueName: \"kubernetes.io/projected/ad22a875-9798-4221-ab20-ff7c796f50df-kube-api-access-lmsr4\") pod \"ad22a875-9798-4221-ab20-ff7c796f50df\" (UID: \"ad22a875-9798-4221-ab20-ff7c796f50df\") "
May 25 12:24:22.801603 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:22.801345    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-graph-raw-hpa-94e63-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad22a875-9798-4221-ab20-ff7c796f50df-isvc-xgboost-graph-raw-hpa-94e63-kube-rbac-proxy-sar-config\") pod \"ad22a875-9798-4221-ab20-ff7c796f50df\" (UID: \"ad22a875-9798-4221-ab20-ff7c796f50df\") "
May 25 12:24:22.801720 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:22.801692    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad22a875-9798-4221-ab20-ff7c796f50df-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ad22a875-9798-4221-ab20-ff7c796f50df" (UID: "ad22a875-9798-4221-ab20-ff7c796f50df"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:24:22.801764 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:22.801729    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad22a875-9798-4221-ab20-ff7c796f50df-isvc-xgboost-graph-raw-hpa-94e63-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-graph-raw-hpa-94e63-kube-rbac-proxy-sar-config") pod "ad22a875-9798-4221-ab20-ff7c796f50df" (UID: "ad22a875-9798-4221-ab20-ff7c796f50df"). InnerVolumeSpecName "isvc-xgboost-graph-raw-hpa-94e63-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:24:22.803422 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:22.803400    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad22a875-9798-4221-ab20-ff7c796f50df-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ad22a875-9798-4221-ab20-ff7c796f50df" (UID: "ad22a875-9798-4221-ab20-ff7c796f50df"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:24:22.803497 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:22.803461    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad22a875-9798-4221-ab20-ff7c796f50df-kube-api-access-lmsr4" (OuterVolumeSpecName: "kube-api-access-lmsr4") pod "ad22a875-9798-4221-ab20-ff7c796f50df" (UID: "ad22a875-9798-4221-ab20-ff7c796f50df"). InnerVolumeSpecName "kube-api-access-lmsr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:24:22.859497 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:22.859460    2563 generic.go:358] "Generic (PLEG): container finished" podID="ad22a875-9798-4221-ab20-ff7c796f50df" containerID="15d93f716d54ab65f271deda5eecf78272ad81290c5b20d0880115dbc22ca16c" exitCode=0
May 25 12:24:22.859669 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:22.859562    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8" event={"ID":"ad22a875-9798-4221-ab20-ff7c796f50df","Type":"ContainerDied","Data":"15d93f716d54ab65f271deda5eecf78272ad81290c5b20d0880115dbc22ca16c"}
May 25 12:24:22.859669 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:22.859594    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8"
May 25 12:24:22.859669 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:22.859603    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8" event={"ID":"ad22a875-9798-4221-ab20-ff7c796f50df","Type":"ContainerDied","Data":"d894542e708f195c7945587d6d72c394aeeac7c2a24dbf2c1173db70677383a0"}
May 25 12:24:22.859669 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:22.859619    2563 scope.go:117] "RemoveContainer" containerID="51cfd34a4b38c187df3c387f26b80aa3ddc07233580433f10435f712a359f88a"
May 25 12:24:22.867783 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:22.867761    2563 scope.go:117] "RemoveContainer" containerID="15d93f716d54ab65f271deda5eecf78272ad81290c5b20d0880115dbc22ca16c"
May 25 12:24:22.874444 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:22.874429    2563 scope.go:117] "RemoveContainer" containerID="16aed49997e21b963eeada1282864901f261f8a13ba4521ed2346c8ef1965b4f"
May 25 12:24:22.881097 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:22.881081    2563 scope.go:117] "RemoveContainer" containerID="51cfd34a4b38c187df3c387f26b80aa3ddc07233580433f10435f712a359f88a"
May 25 12:24:22.881377 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:24:22.881356    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51cfd34a4b38c187df3c387f26b80aa3ddc07233580433f10435f712a359f88a\": container with ID starting with 51cfd34a4b38c187df3c387f26b80aa3ddc07233580433f10435f712a359f88a not found: ID does not exist" containerID="51cfd34a4b38c187df3c387f26b80aa3ddc07233580433f10435f712a359f88a"
May 25 12:24:22.881489 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:22.881384    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51cfd34a4b38c187df3c387f26b80aa3ddc07233580433f10435f712a359f88a"} err="failed to get container status \"51cfd34a4b38c187df3c387f26b80aa3ddc07233580433f10435f712a359f88a\": rpc error: code = NotFound desc = could not find container \"51cfd34a4b38c187df3c387f26b80aa3ddc07233580433f10435f712a359f88a\": container with ID starting with 51cfd34a4b38c187df3c387f26b80aa3ddc07233580433f10435f712a359f88a not found: ID does not exist"
May 25 12:24:22.881489 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:22.881401    2563 scope.go:117] "RemoveContainer" containerID="15d93f716d54ab65f271deda5eecf78272ad81290c5b20d0880115dbc22ca16c"
May 25 12:24:22.881729 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:24:22.881711    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15d93f716d54ab65f271deda5eecf78272ad81290c5b20d0880115dbc22ca16c\": container with ID starting with 15d93f716d54ab65f271deda5eecf78272ad81290c5b20d0880115dbc22ca16c not found: ID does not exist" containerID="15d93f716d54ab65f271deda5eecf78272ad81290c5b20d0880115dbc22ca16c"
May 25 12:24:22.881793 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:22.881739    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15d93f716d54ab65f271deda5eecf78272ad81290c5b20d0880115dbc22ca16c"} err="failed to get container status \"15d93f716d54ab65f271deda5eecf78272ad81290c5b20d0880115dbc22ca16c\": rpc error: code = NotFound desc = could not find container \"15d93f716d54ab65f271deda5eecf78272ad81290c5b20d0880115dbc22ca16c\": container with ID starting with 15d93f716d54ab65f271deda5eecf78272ad81290c5b20d0880115dbc22ca16c not found: ID does not exist"
May 25 12:24:22.881793 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:22.881761    2563 scope.go:117] "RemoveContainer" containerID="16aed49997e21b963eeada1282864901f261f8a13ba4521ed2346c8ef1965b4f"
May 25 12:24:22.882023 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:24:22.882007    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16aed49997e21b963eeada1282864901f261f8a13ba4521ed2346c8ef1965b4f\": container with ID starting with 16aed49997e21b963eeada1282864901f261f8a13ba4521ed2346c8ef1965b4f not found: ID does not exist" containerID="16aed49997e21b963eeada1282864901f261f8a13ba4521ed2346c8ef1965b4f"
May 25 12:24:22.882088 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:22.882033    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16aed49997e21b963eeada1282864901f261f8a13ba4521ed2346c8ef1965b4f"} err="failed to get container status \"16aed49997e21b963eeada1282864901f261f8a13ba4521ed2346c8ef1965b4f\": rpc error: code = NotFound desc = could not find container \"16aed49997e21b963eeada1282864901f261f8a13ba4521ed2346c8ef1965b4f\": container with ID starting with 16aed49997e21b963eeada1282864901f261f8a13ba4521ed2346c8ef1965b4f not found: ID does not exist"
May 25 12:24:22.882088 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:22.882073    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8"]
May 25 12:24:22.887036 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:22.887014    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-94e63-predictor-6c76597f6f-m8vv8"]
May 25 12:24:22.901856 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:22.901830    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmsr4\" (UniqueName: \"kubernetes.io/projected/ad22a875-9798-4221-ab20-ff7c796f50df-kube-api-access-lmsr4\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:24:22.901957 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:22.901859    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-graph-raw-hpa-94e63-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad22a875-9798-4221-ab20-ff7c796f50df-isvc-xgboost-graph-raw-hpa-94e63-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:24:22.901957 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:22.901871    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad22a875-9798-4221-ab20-ff7c796f50df-proxy-tls\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:24:22.901957 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:22.901882    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad22a875-9798-4221-ab20-ff7c796f50df-kserve-provision-location\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:24:23.275800 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:23.275770    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7"
May 25 12:24:23.407022 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:23.406941    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87c07aa3-95d1-402c-9fde-901df344d3a2-proxy-tls\") pod \"87c07aa3-95d1-402c-9fde-901df344d3a2\" (UID: \"87c07aa3-95d1-402c-9fde-901df344d3a2\") "
May 25 12:24:23.407022 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:23.406994    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnf9c\" (UniqueName: \"kubernetes.io/projected/87c07aa3-95d1-402c-9fde-901df344d3a2-kube-api-access-qnf9c\") pod \"87c07aa3-95d1-402c-9fde-901df344d3a2\" (UID: \"87c07aa3-95d1-402c-9fde-901df344d3a2\") "
May 25 12:24:23.407022 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:23.407021    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-graph-raw-hpa-94e63-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87c07aa3-95d1-402c-9fde-901df344d3a2-isvc-sklearn-graph-raw-hpa-94e63-kube-rbac-proxy-sar-config\") pod \"87c07aa3-95d1-402c-9fde-901df344d3a2\" (UID: \"87c07aa3-95d1-402c-9fde-901df344d3a2\") "
May 25 12:24:23.407310 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:23.407048    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87c07aa3-95d1-402c-9fde-901df344d3a2-kserve-provision-location\") pod \"87c07aa3-95d1-402c-9fde-901df344d3a2\" (UID: \"87c07aa3-95d1-402c-9fde-901df344d3a2\") "
May 25 12:24:23.407368 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:23.407352    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87c07aa3-95d1-402c-9fde-901df344d3a2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "87c07aa3-95d1-402c-9fde-901df344d3a2" (UID: "87c07aa3-95d1-402c-9fde-901df344d3a2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:24:23.407416 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:23.407362    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87c07aa3-95d1-402c-9fde-901df344d3a2-isvc-sklearn-graph-raw-hpa-94e63-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-graph-raw-hpa-94e63-kube-rbac-proxy-sar-config") pod "87c07aa3-95d1-402c-9fde-901df344d3a2" (UID: "87c07aa3-95d1-402c-9fde-901df344d3a2"). InnerVolumeSpecName "isvc-sklearn-graph-raw-hpa-94e63-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:24:23.409009 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:23.408984    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87c07aa3-95d1-402c-9fde-901df344d3a2-kube-api-access-qnf9c" (OuterVolumeSpecName: "kube-api-access-qnf9c") pod "87c07aa3-95d1-402c-9fde-901df344d3a2" (UID: "87c07aa3-95d1-402c-9fde-901df344d3a2"). InnerVolumeSpecName "kube-api-access-qnf9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:24:23.409091 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:23.409012    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87c07aa3-95d1-402c-9fde-901df344d3a2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "87c07aa3-95d1-402c-9fde-901df344d3a2" (UID: "87c07aa3-95d1-402c-9fde-901df344d3a2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:24:23.508076 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:23.508037    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87c07aa3-95d1-402c-9fde-901df344d3a2-proxy-tls\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:24:23.508076 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:23.508066    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnf9c\" (UniqueName: \"kubernetes.io/projected/87c07aa3-95d1-402c-9fde-901df344d3a2-kube-api-access-qnf9c\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:24:23.508076 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:23.508078    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-graph-raw-hpa-94e63-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87c07aa3-95d1-402c-9fde-901df344d3a2-isvc-sklearn-graph-raw-hpa-94e63-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:24:23.508076 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:23.508089    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87c07aa3-95d1-402c-9fde-901df344d3a2-kserve-provision-location\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:24:23.864901 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:23.864858    2563 generic.go:358] "Generic (PLEG): container finished" podID="87c07aa3-95d1-402c-9fde-901df344d3a2" containerID="9d537a39b11741ba1617eaf48add8074efd75ea337a91c369b85f3fc22bb7bfb" exitCode=0
May 25 12:24:23.865301 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:23.864900    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7" event={"ID":"87c07aa3-95d1-402c-9fde-901df344d3a2","Type":"ContainerDied","Data":"9d537a39b11741ba1617eaf48add8074efd75ea337a91c369b85f3fc22bb7bfb"}
May 25 12:24:23.865301 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:23.864925    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7"
May 25 12:24:23.865301 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:23.864945    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7" event={"ID":"87c07aa3-95d1-402c-9fde-901df344d3a2","Type":"ContainerDied","Data":"8904772ac3114f19b72efb3560c622595fcee11e639483f871bcd6e03593c1f2"}
May 25 12:24:23.865301 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:23.864963    2563 scope.go:117] "RemoveContainer" containerID="26cd7a3d1ba59c6ce204e3c33a233d804e9a9cf2cdfb5ba0fab0f542efc58e63"
May 25 12:24:23.875024 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:23.875009    2563 scope.go:117] "RemoveContainer" containerID="9d537a39b11741ba1617eaf48add8074efd75ea337a91c369b85f3fc22bb7bfb"
May 25 12:24:23.881733 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:23.881715    2563 scope.go:117] "RemoveContainer" containerID="11b82069b2b1c95b50734c9253df8dd59ba0e59bb1205a2796438f5c13926b06"
May 25 12:24:23.888395 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:23.888380    2563 scope.go:117] "RemoveContainer" containerID="26cd7a3d1ba59c6ce204e3c33a233d804e9a9cf2cdfb5ba0fab0f542efc58e63"
May 25 12:24:23.888743 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:24:23.888715    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26cd7a3d1ba59c6ce204e3c33a233d804e9a9cf2cdfb5ba0fab0f542efc58e63\": container with ID starting with 26cd7a3d1ba59c6ce204e3c33a233d804e9a9cf2cdfb5ba0fab0f542efc58e63 not found: ID does not exist" containerID="26cd7a3d1ba59c6ce204e3c33a233d804e9a9cf2cdfb5ba0fab0f542efc58e63"
May 25 12:24:23.888824 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:23.888760    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7"]
May 25 12:24:23.888824 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:23.888754    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26cd7a3d1ba59c6ce204e3c33a233d804e9a9cf2cdfb5ba0fab0f542efc58e63"} err="failed to get container status \"26cd7a3d1ba59c6ce204e3c33a233d804e9a9cf2cdfb5ba0fab0f542efc58e63\": rpc error: code = NotFound desc = could not find container \"26cd7a3d1ba59c6ce204e3c33a233d804e9a9cf2cdfb5ba0fab0f542efc58e63\": container with ID starting with 26cd7a3d1ba59c6ce204e3c33a233d804e9a9cf2cdfb5ba0fab0f542efc58e63 not found: ID does not exist"
May 25 12:24:23.888824 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:23.888809    2563 scope.go:117] "RemoveContainer" containerID="9d537a39b11741ba1617eaf48add8074efd75ea337a91c369b85f3fc22bb7bfb"
May 25 12:24:23.889038 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:24:23.889013    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d537a39b11741ba1617eaf48add8074efd75ea337a91c369b85f3fc22bb7bfb\": container with ID starting with 9d537a39b11741ba1617eaf48add8074efd75ea337a91c369b85f3fc22bb7bfb not found: ID does not exist" containerID="9d537a39b11741ba1617eaf48add8074efd75ea337a91c369b85f3fc22bb7bfb"
May 25 12:24:23.889117 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:23.889049    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d537a39b11741ba1617eaf48add8074efd75ea337a91c369b85f3fc22bb7bfb"} err="failed to get container status \"9d537a39b11741ba1617eaf48add8074efd75ea337a91c369b85f3fc22bb7bfb\": rpc error: code = NotFound desc = could not find container \"9d537a39b11741ba1617eaf48add8074efd75ea337a91c369b85f3fc22bb7bfb\": container with ID starting with 9d537a39b11741ba1617eaf48add8074efd75ea337a91c369b85f3fc22bb7bfb not found: ID does not exist"
May 25 12:24:23.889117 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:23.889072    2563 scope.go:117] "RemoveContainer" containerID="11b82069b2b1c95b50734c9253df8dd59ba0e59bb1205a2796438f5c13926b06"
May 25 12:24:23.889326 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:24:23.889307    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11b82069b2b1c95b50734c9253df8dd59ba0e59bb1205a2796438f5c13926b06\": container with ID starting with 11b82069b2b1c95b50734c9253df8dd59ba0e59bb1205a2796438f5c13926b06 not found: ID does not exist" containerID="11b82069b2b1c95b50734c9253df8dd59ba0e59bb1205a2796438f5c13926b06"
May 25 12:24:23.889371 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:23.889342    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11b82069b2b1c95b50734c9253df8dd59ba0e59bb1205a2796438f5c13926b06"} err="failed to get container status \"11b82069b2b1c95b50734c9253df8dd59ba0e59bb1205a2796438f5c13926b06\": rpc error: code = NotFound desc = could not find container \"11b82069b2b1c95b50734c9253df8dd59ba0e59bb1205a2796438f5c13926b06\": container with ID starting with 11b82069b2b1c95b50734c9253df8dd59ba0e59bb1205a2796438f5c13926b06 not found: ID does not exist"
May 25 12:24:23.895497 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:23.895476    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-94e63-predictor-9d474f649-nlpz7"]
May 25 12:24:24.637316 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:24.637279    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87c07aa3-95d1-402c-9fde-901df344d3a2" path="/var/lib/kubelet/pods/87c07aa3-95d1-402c-9fde-901df344d3a2/volumes"
May 25 12:24:24.637794 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:24.637781    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad22a875-9798-4221-ab20-ff7c796f50df" path="/var/lib/kubelet/pods/ad22a875-9798-4221-ab20-ff7c796f50df/volumes"
May 25 12:24:39.075589 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.075547    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8"]
May 25 12:24:39.076049 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.075975    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cef298dd-6c2a-4607-9201-7dc8e42a58c0" containerName="kserve-container"
May 25 12:24:39.076049 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.075995    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cef298dd-6c2a-4607-9201-7dc8e42a58c0" containerName="kserve-container"
May 25 12:24:39.076049 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.076005    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87c07aa3-95d1-402c-9fde-901df344d3a2" containerName="kserve-container"
May 25 12:24:39.076049 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.076013    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="87c07aa3-95d1-402c-9fde-901df344d3a2" containerName="kserve-container"
May 25 12:24:39.076049 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.076024    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fe1ba95-5b0a-4156-a667-a264c7dd7e5b" containerName="storage-initializer"
May 25 12:24:39.076049 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.076033    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fe1ba95-5b0a-4156-a667-a264c7dd7e5b" containerName="storage-initializer"
May 25 12:24:39.076049 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.076041    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cef298dd-6c2a-4607-9201-7dc8e42a58c0" containerName="kube-rbac-proxy"
May 25 12:24:39.076049 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.076049    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cef298dd-6c2a-4607-9201-7dc8e42a58c0" containerName="kube-rbac-proxy"
May 25 12:24:39.076480 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.076062    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fe1ba95-5b0a-4156-a667-a264c7dd7e5b" containerName="kserve-container"
May 25 12:24:39.076480 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.076070    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fe1ba95-5b0a-4156-a667-a264c7dd7e5b" containerName="kserve-container"
May 25 12:24:39.076480 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.076081    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87c07aa3-95d1-402c-9fde-901df344d3a2" containerName="kube-rbac-proxy"
May 25 12:24:39.076480 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.076090    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="87c07aa3-95d1-402c-9fde-901df344d3a2" containerName="kube-rbac-proxy"
May 25 12:24:39.076480 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.076109    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad22a875-9798-4221-ab20-ff7c796f50df" containerName="kube-rbac-proxy"
May 25 12:24:39.076480 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.076117    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad22a875-9798-4221-ab20-ff7c796f50df" containerName="kube-rbac-proxy"
May 25 12:24:39.076480 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.076130    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cef298dd-6c2a-4607-9201-7dc8e42a58c0" containerName="storage-initializer"
May 25 12:24:39.076480 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.076138    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cef298dd-6c2a-4607-9201-7dc8e42a58c0" containerName="storage-initializer"
May 25 12:24:39.076480 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.076147    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fe1ba95-5b0a-4156-a667-a264c7dd7e5b" containerName="kube-rbac-proxy"
May 25 12:24:39.076480 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.076155    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fe1ba95-5b0a-4156-a667-a264c7dd7e5b" containerName="kube-rbac-proxy"
May 25 12:24:39.076480 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.076166    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad22a875-9798-4221-ab20-ff7c796f50df" containerName="storage-initializer"
May 25 12:24:39.076480 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.076174    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad22a875-9798-4221-ab20-ff7c796f50df" containerName="storage-initializer"
May 25 12:24:39.076480 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.076186    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad22a875-9798-4221-ab20-ff7c796f50df" containerName="kserve-container"
May 25 12:24:39.076480 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.076194    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad22a875-9798-4221-ab20-ff7c796f50df" containerName="kserve-container"
May 25 12:24:39.076480 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.076208    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87c07aa3-95d1-402c-9fde-901df344d3a2" containerName="storage-initializer"
May 25 12:24:39.076480 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.076215    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="87c07aa3-95d1-402c-9fde-901df344d3a2" containerName="storage-initializer"
May 25 12:24:39.076480 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.076286    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="87c07aa3-95d1-402c-9fde-901df344d3a2" containerName="kube-rbac-proxy"
May 25 12:24:39.076480 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.076300    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="cef298dd-6c2a-4607-9201-7dc8e42a58c0" containerName="kube-rbac-proxy"
May 25 12:24:39.076480 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.076309    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="ad22a875-9798-4221-ab20-ff7c796f50df" containerName="kserve-container"
May 25 12:24:39.076480 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.076320    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="cef298dd-6c2a-4607-9201-7dc8e42a58c0" containerName="kserve-container"
May 25 12:24:39.076480 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.076329    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="87c07aa3-95d1-402c-9fde-901df344d3a2" containerName="kserve-container"
May 25 12:24:39.076480 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.076342    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="8fe1ba95-5b0a-4156-a667-a264c7dd7e5b" containerName="kserve-container"
May 25 12:24:39.076480 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.076352    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="8fe1ba95-5b0a-4156-a667-a264c7dd7e5b" containerName="kube-rbac-proxy"
May 25 12:24:39.076480 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.076362    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="ad22a875-9798-4221-ab20-ff7c796f50df" containerName="kube-rbac-proxy"
May 25 12:24:39.079787 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.079766    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8"
May 25 12:24:39.081669 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.081643    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-0f5ec-predictor-serving-cert\""
May 25 12:24:39.081669 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.081661    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 12:24:39.081865 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.081691    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-0f5ec-kube-rbac-proxy-sar-config\""
May 25 12:24:39.081865 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.081711    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 12:24:39.081865 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.081726    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-79bgq\""
May 25 12:24:39.091899 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.091878    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8"]
May 25 12:24:39.239840 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.239800    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-logger-raw-0f5ec-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7af9f91a-431e-4411-bf6e-1c41f6ceeaa7-isvc-logger-raw-0f5ec-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8\" (UID: \"7af9f91a-431e-4411-bf6e-1c41f6ceeaa7\") " pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8"
May 25 12:24:39.240013 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.239855    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7af9f91a-431e-4411-bf6e-1c41f6ceeaa7-kserve-provision-location\") pod \"isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8\" (UID: \"7af9f91a-431e-4411-bf6e-1c41f6ceeaa7\") " pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8"
May 25 12:24:39.240013 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.239912    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7af9f91a-431e-4411-bf6e-1c41f6ceeaa7-proxy-tls\") pod \"isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8\" (UID: \"7af9f91a-431e-4411-bf6e-1c41f6ceeaa7\") " pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8"
May 25 12:24:39.240013 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.239948    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vlcw\" (UniqueName: \"kubernetes.io/projected/7af9f91a-431e-4411-bf6e-1c41f6ceeaa7-kube-api-access-8vlcw\") pod \"isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8\" (UID: \"7af9f91a-431e-4411-bf6e-1c41f6ceeaa7\") " pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8"
May 25 12:24:39.341217 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.341142    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8vlcw\" (UniqueName: \"kubernetes.io/projected/7af9f91a-431e-4411-bf6e-1c41f6ceeaa7-kube-api-access-8vlcw\") pod \"isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8\" (UID: \"7af9f91a-431e-4411-bf6e-1c41f6ceeaa7\") " pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8"
May 25 12:24:39.341217 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.341193    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-logger-raw-0f5ec-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7af9f91a-431e-4411-bf6e-1c41f6ceeaa7-isvc-logger-raw-0f5ec-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8\" (UID: \"7af9f91a-431e-4411-bf6e-1c41f6ceeaa7\") " pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8"
May 25 12:24:39.341358 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.341234    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7af9f91a-431e-4411-bf6e-1c41f6ceeaa7-kserve-provision-location\") pod \"isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8\" (UID: \"7af9f91a-431e-4411-bf6e-1c41f6ceeaa7\") " pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8"
May 25 12:24:39.341358 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.341276    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7af9f91a-431e-4411-bf6e-1c41f6ceeaa7-proxy-tls\") pod \"isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8\" (UID: \"7af9f91a-431e-4411-bf6e-1c41f6ceeaa7\") " pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8"
May 25 12:24:39.341674 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.341654    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7af9f91a-431e-4411-bf6e-1c41f6ceeaa7-kserve-provision-location\") pod \"isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8\" (UID: \"7af9f91a-431e-4411-bf6e-1c41f6ceeaa7\") " pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8"
May 25 12:24:39.341999 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.341970    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-logger-raw-0f5ec-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7af9f91a-431e-4411-bf6e-1c41f6ceeaa7-isvc-logger-raw-0f5ec-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8\" (UID: \"7af9f91a-431e-4411-bf6e-1c41f6ceeaa7\") " pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8"
May 25 12:24:39.343868 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.343847    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7af9f91a-431e-4411-bf6e-1c41f6ceeaa7-proxy-tls\") pod \"isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8\" (UID: \"7af9f91a-431e-4411-bf6e-1c41f6ceeaa7\") " pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8"
May 25 12:24:39.359942 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.359916    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vlcw\" (UniqueName: \"kubernetes.io/projected/7af9f91a-431e-4411-bf6e-1c41f6ceeaa7-kube-api-access-8vlcw\") pod \"isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8\" (UID: \"7af9f91a-431e-4411-bf6e-1c41f6ceeaa7\") " pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8"
May 25 12:24:39.390684 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.390654    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8"
May 25 12:24:39.519803 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.519781    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8"]
May 25 12:24:39.522435 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:24:39.522399    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7af9f91a_431e_4411_bf6e_1c41f6ceeaa7.slice/crio-e1ac02b4fec1a8729accade4a0e9ff390ee543c46a6a6a3fde63787d56b8d8ec WatchSource:0}: Error finding container e1ac02b4fec1a8729accade4a0e9ff390ee543c46a6a6a3fde63787d56b8d8ec: Status 404 returned error can't find the container with id e1ac02b4fec1a8729accade4a0e9ff390ee543c46a6a6a3fde63787d56b8d8ec
May 25 12:24:39.524330 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.524312    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 12:24:39.910819 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.910778    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8" event={"ID":"7af9f91a-431e-4411-bf6e-1c41f6ceeaa7","Type":"ContainerStarted","Data":"f50cfdc96b568d3afc62759b3f35c0ac69ca3f3cd4caff3fc1e48c72c72aa34f"}
May 25 12:24:39.910819 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:39.910825    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8" event={"ID":"7af9f91a-431e-4411-bf6e-1c41f6ceeaa7","Type":"ContainerStarted","Data":"e1ac02b4fec1a8729accade4a0e9ff390ee543c46a6a6a3fde63787d56b8d8ec"}
May 25 12:24:43.922895 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:43.922862    2563 generic.go:358] "Generic (PLEG): container finished" podID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerID="f50cfdc96b568d3afc62759b3f35c0ac69ca3f3cd4caff3fc1e48c72c72aa34f" exitCode=0
May 25 12:24:43.923267 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:43.922936    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8" event={"ID":"7af9f91a-431e-4411-bf6e-1c41f6ceeaa7","Type":"ContainerDied","Data":"f50cfdc96b568d3afc62759b3f35c0ac69ca3f3cd4caff3fc1e48c72c72aa34f"}
May 25 12:24:44.927899 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:44.927863    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8" event={"ID":"7af9f91a-431e-4411-bf6e-1c41f6ceeaa7","Type":"ContainerStarted","Data":"a222a8fbbc98e4d950fc96609a67da64c4bff00fe318d93de8290ba3c7409839"}
May 25 12:24:44.927899 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:44.927906    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8" event={"ID":"7af9f91a-431e-4411-bf6e-1c41f6ceeaa7","Type":"ContainerStarted","Data":"dcde1234b153be1165e3381ea698820fe035d11e8a33f37c599fb673f5093d4e"}
May 25 12:24:44.928302 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:44.927916    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8" event={"ID":"7af9f91a-431e-4411-bf6e-1c41f6ceeaa7","Type":"ContainerStarted","Data":"81118c8fc171e99643baccd15372943082f29e85a17b24b8b09585e66d15e3eb"}
May 25 12:24:44.928302 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:44.928161    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8"
May 25 12:24:44.928302 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:44.928191    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8"
May 25 12:24:44.929592 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:44.929564    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
May 25 12:24:44.949198 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:44.949150    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8" podStartSLOduration=5.949137552 podStartE2EDuration="5.949137552s" podCreationTimestamp="2026-05-25 12:24:39 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:24:44.948843399 +0000 UTC m=+750.916235039" watchObservedRunningTime="2026-05-25 12:24:44.949137552 +0000 UTC m=+750.916529192"
May 25 12:24:45.931271 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:45.931236    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
May 25 12:24:45.931687 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:45.931306    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8"
May 25 12:24:45.932198 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:45.932173    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:24:46.933619 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:46.933572    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
May 25 12:24:46.934071 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:46.934050    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:24:51.937754 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:51.937725    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8"
May 25 12:24:51.938332 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:51.938305    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
May 25 12:24:51.938529 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:24:51.938493    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:25:01.938440 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:25:01.938394    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
May 25 12:25:01.938897 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:25:01.938876    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:25:11.938811 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:25:11.938768    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
May 25 12:25:11.939273 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:25:11.939251    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:25:21.938767 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:25:21.938721    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
May 25 12:25:21.939191 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:25:21.939173    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:25:31.939004 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:25:31.938950    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
May 25 12:25:31.939548 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:25:31.939397    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:25:41.938238 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:25:41.938138    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
May 25 12:25:41.938768 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:25:41.938566    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:25:51.938846 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:25:51.938818    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8"
May 25 12:25:51.939281 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:25:51.939118    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8"
May 25 12:26:04.217860 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:04.217828    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8"]
May 25 12:26:04.218305 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:04.218187    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerName="kserve-container" containerID="cri-o://81118c8fc171e99643baccd15372943082f29e85a17b24b8b09585e66d15e3eb" gracePeriod=30
May 25 12:26:04.218305 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:04.218230    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerName="agent" containerID="cri-o://a222a8fbbc98e4d950fc96609a67da64c4bff00fe318d93de8290ba3c7409839" gracePeriod=30
May 25 12:26:04.218423 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:04.218230    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerName="kube-rbac-proxy" containerID="cri-o://dcde1234b153be1165e3381ea698820fe035d11e8a33f37c599fb673f5093d4e" gracePeriod=30
May 25 12:26:04.270105 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:04.270069    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w"]
May 25 12:26:04.273642 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:04.273621    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w"
May 25 12:26:04.275661 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:04.275639    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-4a8bd-kube-rbac-proxy-sar-config\""
May 25 12:26:04.275762 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:04.275718    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-4a8bd-predictor-serving-cert\""
May 25 12:26:04.284313 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:04.284290    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w"]
May 25 12:26:04.390849 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:04.390812    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-scale-raw-4a8bd-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c2670ee-0cf0-4b98-873d-182daa4ce2c4-isvc-sklearn-scale-raw-4a8bd-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w\" (UID: \"2c2670ee-0cf0-4b98-873d-182daa4ce2c4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w"
May 25 12:26:04.391028 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:04.390881    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgfb2\" (UniqueName: \"kubernetes.io/projected/2c2670ee-0cf0-4b98-873d-182daa4ce2c4-kube-api-access-fgfb2\") pod \"isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w\" (UID: \"2c2670ee-0cf0-4b98-873d-182daa4ce2c4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w"
May 25 12:26:04.391028 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:04.390900    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c2670ee-0cf0-4b98-873d-182daa4ce2c4-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w\" (UID: \"2c2670ee-0cf0-4b98-873d-182daa4ce2c4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w"
May 25 12:26:04.391028 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:04.390926    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c2670ee-0cf0-4b98-873d-182daa4ce2c4-proxy-tls\") pod \"isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w\" (UID: \"2c2670ee-0cf0-4b98-873d-182daa4ce2c4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w"
May 25 12:26:04.491687 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:04.491659    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c2670ee-0cf0-4b98-873d-182daa4ce2c4-proxy-tls\") pod \"isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w\" (UID: \"2c2670ee-0cf0-4b98-873d-182daa4ce2c4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w"
May 25 12:26:04.491859 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:04.491712    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-scale-raw-4a8bd-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c2670ee-0cf0-4b98-873d-182daa4ce2c4-isvc-sklearn-scale-raw-4a8bd-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w\" (UID: \"2c2670ee-0cf0-4b98-873d-182daa4ce2c4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w"
May 25 12:26:04.491859 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:04.491762    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fgfb2\" (UniqueName: \"kubernetes.io/projected/2c2670ee-0cf0-4b98-873d-182daa4ce2c4-kube-api-access-fgfb2\") pod \"isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w\" (UID: \"2c2670ee-0cf0-4b98-873d-182daa4ce2c4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w"
May 25 12:26:04.491859 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:04.491781    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c2670ee-0cf0-4b98-873d-182daa4ce2c4-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w\" (UID: \"2c2670ee-0cf0-4b98-873d-182daa4ce2c4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w"
May 25 12:26:04.491859 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:26:04.491844    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-serving-cert: secret "isvc-sklearn-scale-raw-4a8bd-predictor-serving-cert" not found
May 25 12:26:04.492068 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:26:04.491922    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c2670ee-0cf0-4b98-873d-182daa4ce2c4-proxy-tls podName:2c2670ee-0cf0-4b98-873d-182daa4ce2c4 nodeName:}" failed. No retries permitted until 2026-05-25 12:26:04.991898979 +0000 UTC m=+830.959290597 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2c2670ee-0cf0-4b98-873d-182daa4ce2c4-proxy-tls") pod "isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w" (UID: "2c2670ee-0cf0-4b98-873d-182daa4ce2c4") : secret "isvc-sklearn-scale-raw-4a8bd-predictor-serving-cert" not found
May 25 12:26:04.492138 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:04.492121    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c2670ee-0cf0-4b98-873d-182daa4ce2c4-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w\" (UID: \"2c2670ee-0cf0-4b98-873d-182daa4ce2c4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w"
May 25 12:26:04.492439 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:04.492419    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-scale-raw-4a8bd-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c2670ee-0cf0-4b98-873d-182daa4ce2c4-isvc-sklearn-scale-raw-4a8bd-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w\" (UID: \"2c2670ee-0cf0-4b98-873d-182daa4ce2c4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w"
May 25 12:26:04.502605 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:04.502580    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgfb2\" (UniqueName: \"kubernetes.io/projected/2c2670ee-0cf0-4b98-873d-182daa4ce2c4-kube-api-access-fgfb2\") pod \"isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w\" (UID: \"2c2670ee-0cf0-4b98-873d-182daa4ce2c4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w"
May 25 12:26:04.995772 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:04.995733    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c2670ee-0cf0-4b98-873d-182daa4ce2c4-proxy-tls\") pod \"isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w\" (UID: \"2c2670ee-0cf0-4b98-873d-182daa4ce2c4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w"
May 25 12:26:04.998165 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:04.998132    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c2670ee-0cf0-4b98-873d-182daa4ce2c4-proxy-tls\") pod \"isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w\" (UID: \"2c2670ee-0cf0-4b98-873d-182daa4ce2c4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w"
May 25 12:26:05.147313 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:05.147274    2563 generic.go:358] "Generic (PLEG): container finished" podID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerID="dcde1234b153be1165e3381ea698820fe035d11e8a33f37c599fb673f5093d4e" exitCode=2
May 25 12:26:05.147313 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:05.147316    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8" event={"ID":"7af9f91a-431e-4411-bf6e-1c41f6ceeaa7","Type":"ContainerDied","Data":"dcde1234b153be1165e3381ea698820fe035d11e8a33f37c599fb673f5093d4e"}
May 25 12:26:05.186303 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:05.186268    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w"
May 25 12:26:05.314430 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:05.314156    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w"]
May 25 12:26:05.317312 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:26:05.317278    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c2670ee_0cf0_4b98_873d_182daa4ce2c4.slice/crio-1c7612b2bdc6f339c789d03f32767a4fa6dd467a4ec0f43ef9d85337fcac881f WatchSource:0}: Error finding container 1c7612b2bdc6f339c789d03f32767a4fa6dd467a4ec0f43ef9d85337fcac881f: Status 404 returned error can't find the container with id 1c7612b2bdc6f339c789d03f32767a4fa6dd467a4ec0f43ef9d85337fcac881f
May 25 12:26:06.151948 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:06.151907    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w" event={"ID":"2c2670ee-0cf0-4b98-873d-182daa4ce2c4","Type":"ContainerStarted","Data":"8ed057c6ed33d0be76cc2ea5a88a19d72ef623aa4652d6deb930401f62ba797d"}
May 25 12:26:06.151948 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:06.151949    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w" event={"ID":"2c2670ee-0cf0-4b98-873d-182daa4ce2c4","Type":"ContainerStarted","Data":"1c7612b2bdc6f339c789d03f32767a4fa6dd467a4ec0f43ef9d85337fcac881f"}
May 25 12:26:06.933774 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:06.933728    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
May 25 12:26:09.163018 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:09.162987    2563 generic.go:358] "Generic (PLEG): container finished" podID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerID="81118c8fc171e99643baccd15372943082f29e85a17b24b8b09585e66d15e3eb" exitCode=0
May 25 12:26:09.163388 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:09.163046    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8" event={"ID":"7af9f91a-431e-4411-bf6e-1c41f6ceeaa7","Type":"ContainerDied","Data":"81118c8fc171e99643baccd15372943082f29e85a17b24b8b09585e66d15e3eb"}
May 25 12:26:10.167209 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:10.167177    2563 generic.go:358] "Generic (PLEG): container finished" podID="2c2670ee-0cf0-4b98-873d-182daa4ce2c4" containerID="8ed057c6ed33d0be76cc2ea5a88a19d72ef623aa4652d6deb930401f62ba797d" exitCode=0
May 25 12:26:10.167625 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:10.167257    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w" event={"ID":"2c2670ee-0cf0-4b98-873d-182daa4ce2c4","Type":"ContainerDied","Data":"8ed057c6ed33d0be76cc2ea5a88a19d72ef623aa4652d6deb930401f62ba797d"}
May 25 12:26:11.171748 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:11.171693    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w" event={"ID":"2c2670ee-0cf0-4b98-873d-182daa4ce2c4","Type":"ContainerStarted","Data":"d3435d39fce5807ca88ff905eecb013a7798ced579b6743daa246a056ef14dd8"}
May 25 12:26:11.172100 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:11.171756    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w" event={"ID":"2c2670ee-0cf0-4b98-873d-182daa4ce2c4","Type":"ContainerStarted","Data":"8301882e4b4b9c40562c4f6f8bc8ee2fa9a9505080b2ade9373a7d44e9f34fc4"}
May 25 12:26:11.172100 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:11.171954    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w"
May 25 12:26:11.190862 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:11.190815    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w" podStartSLOduration=7.190802695 podStartE2EDuration="7.190802695s" podCreationTimestamp="2026-05-25 12:26:04 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:26:11.189154197 +0000 UTC m=+837.156545836" watchObservedRunningTime="2026-05-25 12:26:11.190802695 +0000 UTC m=+837.158194333"
May 25 12:26:11.933732 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:11.933688    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
May 25 12:26:11.939059 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:11.939031    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
May 25 12:26:11.939455 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:11.939427    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:26:12.174509 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:12.174472    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w"
May 25 12:26:12.175591 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:12.175563    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w" podUID="2c2670ee-0cf0-4b98-873d-182daa4ce2c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
May 25 12:26:13.176818 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:13.176781    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w" podUID="2c2670ee-0cf0-4b98-873d-182daa4ce2c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
May 25 12:26:16.934180 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:16.934131    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
May 25 12:26:16.934559 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:16.934288    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8"
May 25 12:26:18.180889 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:18.180862    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w"
May 25 12:26:18.181317 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:18.181291    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w" podUID="2c2670ee-0cf0-4b98-873d-182daa4ce2c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
May 25 12:26:21.934154 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:21.934104    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
May 25 12:26:21.938471 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:21.938442    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
May 25 12:26:21.938819 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:21.938796    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:26:26.934430 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:26.934386    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
May 25 12:26:28.181535 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:28.181477    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w" podUID="2c2670ee-0cf0-4b98-873d-182daa4ce2c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
May 25 12:26:31.934053 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:31.934005    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
May 25 12:26:31.938343 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:31.938309    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
May 25 12:26:31.938474 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:31.938447    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8"
May 25 12:26:31.938748 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:31.938727    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:26:31.938837 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:31.938826    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8"
May 25 12:26:34.239201 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:34.239174    2563 generic.go:358] "Generic (PLEG): container finished" podID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerID="a222a8fbbc98e4d950fc96609a67da64c4bff00fe318d93de8290ba3c7409839" exitCode=0
May 25 12:26:34.239473 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:34.239232    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8" event={"ID":"7af9f91a-431e-4411-bf6e-1c41f6ceeaa7","Type":"ContainerDied","Data":"a222a8fbbc98e4d950fc96609a67da64c4bff00fe318d93de8290ba3c7409839"}
May 25 12:26:34.368188 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:34.368168    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8"
May 25 12:26:34.443543 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:34.443488    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-logger-raw-0f5ec-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7af9f91a-431e-4411-bf6e-1c41f6ceeaa7-isvc-logger-raw-0f5ec-kube-rbac-proxy-sar-config\") pod \"7af9f91a-431e-4411-bf6e-1c41f6ceeaa7\" (UID: \"7af9f91a-431e-4411-bf6e-1c41f6ceeaa7\") "
May 25 12:26:34.443712 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:34.443558    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vlcw\" (UniqueName: \"kubernetes.io/projected/7af9f91a-431e-4411-bf6e-1c41f6ceeaa7-kube-api-access-8vlcw\") pod \"7af9f91a-431e-4411-bf6e-1c41f6ceeaa7\" (UID: \"7af9f91a-431e-4411-bf6e-1c41f6ceeaa7\") "
May 25 12:26:34.443712 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:34.443609    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7af9f91a-431e-4411-bf6e-1c41f6ceeaa7-kserve-provision-location\") pod \"7af9f91a-431e-4411-bf6e-1c41f6ceeaa7\" (UID: \"7af9f91a-431e-4411-bf6e-1c41f6ceeaa7\") "
May 25 12:26:34.443712 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:34.443638    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7af9f91a-431e-4411-bf6e-1c41f6ceeaa7-proxy-tls\") pod \"7af9f91a-431e-4411-bf6e-1c41f6ceeaa7\" (UID: \"7af9f91a-431e-4411-bf6e-1c41f6ceeaa7\") "
May 25 12:26:34.443906 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:34.443877    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7af9f91a-431e-4411-bf6e-1c41f6ceeaa7-isvc-logger-raw-0f5ec-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-logger-raw-0f5ec-kube-rbac-proxy-sar-config") pod "7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" (UID: "7af9f91a-431e-4411-bf6e-1c41f6ceeaa7"). InnerVolumeSpecName "isvc-logger-raw-0f5ec-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:26:34.444009 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:34.443985    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af9f91a-431e-4411-bf6e-1c41f6ceeaa7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" (UID: "7af9f91a-431e-4411-bf6e-1c41f6ceeaa7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:26:34.445598 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:34.445577    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7af9f91a-431e-4411-bf6e-1c41f6ceeaa7-kube-api-access-8vlcw" (OuterVolumeSpecName: "kube-api-access-8vlcw") pod "7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" (UID: "7af9f91a-431e-4411-bf6e-1c41f6ceeaa7"). InnerVolumeSpecName "kube-api-access-8vlcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:26:34.445689 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:34.445647    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7af9f91a-431e-4411-bf6e-1c41f6ceeaa7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" (UID: "7af9f91a-431e-4411-bf6e-1c41f6ceeaa7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:26:34.544648 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:34.544619    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7af9f91a-431e-4411-bf6e-1c41f6ceeaa7-kserve-provision-location\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:26:34.544648 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:34.544644    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7af9f91a-431e-4411-bf6e-1c41f6ceeaa7-proxy-tls\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:26:34.544648 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:34.544653    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-logger-raw-0f5ec-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7af9f91a-431e-4411-bf6e-1c41f6ceeaa7-isvc-logger-raw-0f5ec-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:26:34.544869 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:34.544664    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vlcw\" (UniqueName: \"kubernetes.io/projected/7af9f91a-431e-4411-bf6e-1c41f6ceeaa7-kube-api-access-8vlcw\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:26:35.243951 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:35.243915    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8" event={"ID":"7af9f91a-431e-4411-bf6e-1c41f6ceeaa7","Type":"ContainerDied","Data":"e1ac02b4fec1a8729accade4a0e9ff390ee543c46a6a6a3fde63787d56b8d8ec"}
May 25 12:26:35.244408 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:35.243964    2563 scope.go:117] "RemoveContainer" containerID="a222a8fbbc98e4d950fc96609a67da64c4bff00fe318d93de8290ba3c7409839"
May 25 12:26:35.244408 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:35.243930    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8"
May 25 12:26:35.251399 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:35.251381    2563 scope.go:117] "RemoveContainer" containerID="dcde1234b153be1165e3381ea698820fe035d11e8a33f37c599fb673f5093d4e"
May 25 12:26:35.258025 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:35.258007    2563 scope.go:117] "RemoveContainer" containerID="81118c8fc171e99643baccd15372943082f29e85a17b24b8b09585e66d15e3eb"
May 25 12:26:35.263318 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:35.263294    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8"]
May 25 12:26:35.265586 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:35.265569    2563 scope.go:117] "RemoveContainer" containerID="f50cfdc96b568d3afc62759b3f35c0ac69ca3f3cd4caff3fc1e48c72c72aa34f"
May 25 12:26:35.267346 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:35.267325    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-0f5ec-predictor-6c8bf97597-m7lr8"]
May 25 12:26:36.637895 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:36.637861    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" path="/var/lib/kubelet/pods/7af9f91a-431e-4411-bf6e-1c41f6ceeaa7/volumes"
May 25 12:26:38.181389 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:38.181350    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w" podUID="2c2670ee-0cf0-4b98-873d-182daa4ce2c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
May 25 12:26:48.181449 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:48.181409    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w" podUID="2c2670ee-0cf0-4b98-873d-182daa4ce2c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
May 25 12:26:58.182007 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:26:58.181967    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w" podUID="2c2670ee-0cf0-4b98-873d-182daa4ce2c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
May 25 12:27:08.182071 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:27:08.181990    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w" podUID="2c2670ee-0cf0-4b98-873d-182daa4ce2c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
May 25 12:27:14.563303 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:27:14.563277    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw9bb_012c465e-2683-488a-abc1-18ad86529eb3/ovn-acl-logging/0.log"
May 25 12:27:14.566276 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:27:14.566257    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw9bb_012c465e-2683-488a-abc1-18ad86529eb3/ovn-acl-logging/0.log"
May 25 12:27:18.182164 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:27:18.182123    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w" podUID="2c2670ee-0cf0-4b98-873d-182daa4ce2c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
May 25 12:27:28.181562 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:27:28.181493    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w" podUID="2c2670ee-0cf0-4b98-873d-182daa4ce2c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
May 25 12:27:38.181543 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:27:38.181480    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w" podUID="2c2670ee-0cf0-4b98-873d-182daa4ce2c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
May 25 12:27:44.640165 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:27:44.640118    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w" podUID="2c2670ee-0cf0-4b98-873d-182daa4ce2c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
May 25 12:27:54.636280 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:27:54.636235    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w" podUID="2c2670ee-0cf0-4b98-873d-182daa4ce2c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
May 25 12:28:04.635963 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:04.635924    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w" podUID="2c2670ee-0cf0-4b98-873d-182daa4ce2c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
May 25 12:28:14.636147 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:14.636100    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w" podUID="2c2670ee-0cf0-4b98-873d-182daa4ce2c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
May 25 12:28:24.641323 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:24.641293    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w"
May 25 12:28:34.449201 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:34.449110    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w"]
May 25 12:28:34.449687 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:34.449546    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w" podUID="2c2670ee-0cf0-4b98-873d-182daa4ce2c4" containerName="kserve-container" containerID="cri-o://8301882e4b4b9c40562c4f6f8bc8ee2fa9a9505080b2ade9373a7d44e9f34fc4" gracePeriod=30
May 25 12:28:34.449687 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:34.449611    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w" podUID="2c2670ee-0cf0-4b98-873d-182daa4ce2c4" containerName="kube-rbac-proxy" containerID="cri-o://d3435d39fce5807ca88ff905eecb013a7798ced579b6743daa246a056ef14dd8" gracePeriod=30
May 25 12:28:34.549199 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:34.549163    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm"]
May 25 12:28:34.549476 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:34.549463    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerName="kserve-container"
May 25 12:28:34.549476 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:34.549477    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerName="kserve-container"
May 25 12:28:34.549599 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:34.549489    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerName="kube-rbac-proxy"
May 25 12:28:34.549599 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:34.549496    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerName="kube-rbac-proxy"
May 25 12:28:34.549599 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:34.549504    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerName="storage-initializer"
May 25 12:28:34.549599 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:34.549514    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerName="storage-initializer"
May 25 12:28:34.549599 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:34.549547    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerName="agent"
May 25 12:28:34.549599 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:34.549556    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerName="agent"
May 25 12:28:34.549832 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:34.549613    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerName="kube-rbac-proxy"
May 25 12:28:34.549832 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:34.549626    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerName="kserve-container"
May 25 12:28:34.549832 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:34.549635    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="7af9f91a-431e-4411-bf6e-1c41f6ceeaa7" containerName="agent"
May 25 12:28:34.552920 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:34.552900    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm"
May 25 12:28:34.554763 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:34.554743    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-10f269-kube-rbac-proxy-sar-config\""
May 25 12:28:34.554877 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:34.554749    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-10f269-predictor-serving-cert\""
May 25 12:28:34.561807 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:34.561783    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm"]
May 25 12:28:34.583326 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:34.583293    2563 generic.go:358] "Generic (PLEG): container finished" podID="2c2670ee-0cf0-4b98-873d-182daa4ce2c4" containerID="d3435d39fce5807ca88ff905eecb013a7798ced579b6743daa246a056ef14dd8" exitCode=2
May 25 12:28:34.583487 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:34.583363    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w" event={"ID":"2c2670ee-0cf0-4b98-873d-182daa4ce2c4","Type":"ContainerDied","Data":"d3435d39fce5807ca88ff905eecb013a7798ced579b6743daa246a056ef14dd8"}
May 25 12:28:34.635997 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:34.635963    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w" podUID="2c2670ee-0cf0-4b98-873d-182daa4ce2c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
May 25 12:28:34.675640 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:34.675606    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f14ed01-2147-4647-b02a-17683660e3f2-proxy-tls\") pod \"isvc-primary-10f269-predictor-5bbff9f956-2jzjm\" (UID: \"2f14ed01-2147-4647-b02a-17683660e3f2\") " pod="kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm"
May 25 12:28:34.675766 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:34.675645    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9nh5\" (UniqueName: \"kubernetes.io/projected/2f14ed01-2147-4647-b02a-17683660e3f2-kube-api-access-v9nh5\") pod \"isvc-primary-10f269-predictor-5bbff9f956-2jzjm\" (UID: \"2f14ed01-2147-4647-b02a-17683660e3f2\") " pod="kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm"
May 25 12:28:34.675766 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:34.675718    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f14ed01-2147-4647-b02a-17683660e3f2-kserve-provision-location\") pod \"isvc-primary-10f269-predictor-5bbff9f956-2jzjm\" (UID: \"2f14ed01-2147-4647-b02a-17683660e3f2\") " pod="kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm"
May 25 12:28:34.675839 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:34.675765    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-primary-10f269-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f14ed01-2147-4647-b02a-17683660e3f2-isvc-primary-10f269-kube-rbac-proxy-sar-config\") pod \"isvc-primary-10f269-predictor-5bbff9f956-2jzjm\" (UID: \"2f14ed01-2147-4647-b02a-17683660e3f2\") " pod="kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm"
May 25 12:28:34.776790 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:34.776754    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f14ed01-2147-4647-b02a-17683660e3f2-kserve-provision-location\") pod \"isvc-primary-10f269-predictor-5bbff9f956-2jzjm\" (UID: \"2f14ed01-2147-4647-b02a-17683660e3f2\") " pod="kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm"
May 25 12:28:34.776969 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:34.776799    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-primary-10f269-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f14ed01-2147-4647-b02a-17683660e3f2-isvc-primary-10f269-kube-rbac-proxy-sar-config\") pod \"isvc-primary-10f269-predictor-5bbff9f956-2jzjm\" (UID: \"2f14ed01-2147-4647-b02a-17683660e3f2\") " pod="kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm"
May 25 12:28:34.776969 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:34.776855    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f14ed01-2147-4647-b02a-17683660e3f2-proxy-tls\") pod \"isvc-primary-10f269-predictor-5bbff9f956-2jzjm\" (UID: \"2f14ed01-2147-4647-b02a-17683660e3f2\") " pod="kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm"
May 25 12:28:34.776969 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:34.776872    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-v9nh5\" (UniqueName: \"kubernetes.io/projected/2f14ed01-2147-4647-b02a-17683660e3f2-kube-api-access-v9nh5\") pod \"isvc-primary-10f269-predictor-5bbff9f956-2jzjm\" (UID: \"2f14ed01-2147-4647-b02a-17683660e3f2\") " pod="kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm"
May 25 12:28:34.777193 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:34.777169    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f14ed01-2147-4647-b02a-17683660e3f2-kserve-provision-location\") pod \"isvc-primary-10f269-predictor-5bbff9f956-2jzjm\" (UID: \"2f14ed01-2147-4647-b02a-17683660e3f2\") " pod="kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm"
May 25 12:28:34.777561 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:34.777507    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-primary-10f269-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f14ed01-2147-4647-b02a-17683660e3f2-isvc-primary-10f269-kube-rbac-proxy-sar-config\") pod \"isvc-primary-10f269-predictor-5bbff9f956-2jzjm\" (UID: \"2f14ed01-2147-4647-b02a-17683660e3f2\") " pod="kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm"
May 25 12:28:34.779261 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:34.779242    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f14ed01-2147-4647-b02a-17683660e3f2-proxy-tls\") pod \"isvc-primary-10f269-predictor-5bbff9f956-2jzjm\" (UID: \"2f14ed01-2147-4647-b02a-17683660e3f2\") " pod="kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm"
May 25 12:28:34.786472 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:34.786453    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9nh5\" (UniqueName: \"kubernetes.io/projected/2f14ed01-2147-4647-b02a-17683660e3f2-kube-api-access-v9nh5\") pod \"isvc-primary-10f269-predictor-5bbff9f956-2jzjm\" (UID: \"2f14ed01-2147-4647-b02a-17683660e3f2\") " pod="kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm"
May 25 12:28:34.864555 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:34.864501    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm"
May 25 12:28:34.992820 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:34.992785    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm"]
May 25 12:28:34.995809 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:28:34.995780    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f14ed01_2147_4647_b02a_17683660e3f2.slice/crio-1f02273d2166cd8dbc54fbc5143b5995a3aef97e9947376905260864403f8d97 WatchSource:0}: Error finding container 1f02273d2166cd8dbc54fbc5143b5995a3aef97e9947376905260864403f8d97: Status 404 returned error can't find the container with id 1f02273d2166cd8dbc54fbc5143b5995a3aef97e9947376905260864403f8d97
May 25 12:28:35.587166 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:35.587132    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm" event={"ID":"2f14ed01-2147-4647-b02a-17683660e3f2","Type":"ContainerStarted","Data":"1c454aa126d0319e87751305f2559be62bfdef85704cfb52e049d2a9065d2154"}
May 25 12:28:35.587166 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:35.587169    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm" event={"ID":"2f14ed01-2147-4647-b02a-17683660e3f2","Type":"ContainerStarted","Data":"1f02273d2166cd8dbc54fbc5143b5995a3aef97e9947376905260864403f8d97"}
May 25 12:28:38.177229 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:38.177187    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w" podUID="2c2670ee-0cf0-4b98-873d-182daa4ce2c4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
May 25 12:28:39.598768 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:39.598732    2563 generic.go:358] "Generic (PLEG): container finished" podID="2f14ed01-2147-4647-b02a-17683660e3f2" containerID="1c454aa126d0319e87751305f2559be62bfdef85704cfb52e049d2a9065d2154" exitCode=0
May 25 12:28:39.599202 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:39.598803    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm" event={"ID":"2f14ed01-2147-4647-b02a-17683660e3f2","Type":"ContainerDied","Data":"1c454aa126d0319e87751305f2559be62bfdef85704cfb52e049d2a9065d2154"}
May 25 12:28:40.603877 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:40.603844    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm" event={"ID":"2f14ed01-2147-4647-b02a-17683660e3f2","Type":"ContainerStarted","Data":"3afce9aac8ea3babe73169c1b4572ca166149e6b62f5886a7611c6c0d2eb2f55"}
May 25 12:28:40.604237 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:40.603886    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm" event={"ID":"2f14ed01-2147-4647-b02a-17683660e3f2","Type":"ContainerStarted","Data":"282d7076d3e212876058185f1ba321a41364376e490b1a3c82041935bf901e44"}
May 25 12:28:40.604237 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:40.604075    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm"
May 25 12:28:40.622695 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:40.622652    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm" podStartSLOduration=6.622637525 podStartE2EDuration="6.622637525s" podCreationTimestamp="2026-05-25 12:28:34 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:28:40.62115285 +0000 UTC m=+986.588544502" watchObservedRunningTime="2026-05-25 12:28:40.622637525 +0000 UTC m=+986.590029163"
May 25 12:28:41.606282 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:41.606243    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm"
May 25 12:28:41.607499 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:41.607473    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm" podUID="2f14ed01-2147-4647-b02a-17683660e3f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
May 25 12:28:42.608997 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:42.608956    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm" podUID="2f14ed01-2147-4647-b02a-17683660e3f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
May 25 12:28:43.177929 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:43.177889    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w" podUID="2c2670ee-0cf0-4b98-873d-182daa4ce2c4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
May 25 12:28:43.792258 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:43.792234    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w"
May 25 12:28:43.851314 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:43.851273    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-scale-raw-4a8bd-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c2670ee-0cf0-4b98-873d-182daa4ce2c4-isvc-sklearn-scale-raw-4a8bd-kube-rbac-proxy-sar-config\") pod \"2c2670ee-0cf0-4b98-873d-182daa4ce2c4\" (UID: \"2c2670ee-0cf0-4b98-873d-182daa4ce2c4\") "
May 25 12:28:43.851314 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:43.851323    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgfb2\" (UniqueName: \"kubernetes.io/projected/2c2670ee-0cf0-4b98-873d-182daa4ce2c4-kube-api-access-fgfb2\") pod \"2c2670ee-0cf0-4b98-873d-182daa4ce2c4\" (UID: \"2c2670ee-0cf0-4b98-873d-182daa4ce2c4\") "
May 25 12:28:43.851652 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:43.851354    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c2670ee-0cf0-4b98-873d-182daa4ce2c4-kserve-provision-location\") pod \"2c2670ee-0cf0-4b98-873d-182daa4ce2c4\" (UID: \"2c2670ee-0cf0-4b98-873d-182daa4ce2c4\") "
May 25 12:28:43.851652 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:43.851409    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c2670ee-0cf0-4b98-873d-182daa4ce2c4-proxy-tls\") pod \"2c2670ee-0cf0-4b98-873d-182daa4ce2c4\" (UID: \"2c2670ee-0cf0-4b98-873d-182daa4ce2c4\") "
May 25 12:28:43.851773 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:43.851716    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c2670ee-0cf0-4b98-873d-182daa4ce2c4-isvc-sklearn-scale-raw-4a8bd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-scale-raw-4a8bd-kube-rbac-proxy-sar-config") pod "2c2670ee-0cf0-4b98-873d-182daa4ce2c4" (UID: "2c2670ee-0cf0-4b98-873d-182daa4ce2c4"). InnerVolumeSpecName "isvc-sklearn-scale-raw-4a8bd-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:28:43.851773 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:43.851716    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c2670ee-0cf0-4b98-873d-182daa4ce2c4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2c2670ee-0cf0-4b98-873d-182daa4ce2c4" (UID: "2c2670ee-0cf0-4b98-873d-182daa4ce2c4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:28:43.853373 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:43.853347    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c2670ee-0cf0-4b98-873d-182daa4ce2c4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2c2670ee-0cf0-4b98-873d-182daa4ce2c4" (UID: "2c2670ee-0cf0-4b98-873d-182daa4ce2c4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:28:43.853469 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:43.853446    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c2670ee-0cf0-4b98-873d-182daa4ce2c4-kube-api-access-fgfb2" (OuterVolumeSpecName: "kube-api-access-fgfb2") pod "2c2670ee-0cf0-4b98-873d-182daa4ce2c4" (UID: "2c2670ee-0cf0-4b98-873d-182daa4ce2c4"). InnerVolumeSpecName "kube-api-access-fgfb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:28:43.952260 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:43.952160    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c2670ee-0cf0-4b98-873d-182daa4ce2c4-proxy-tls\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:28:43.952260 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:43.952206    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-scale-raw-4a8bd-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c2670ee-0cf0-4b98-873d-182daa4ce2c4-isvc-sklearn-scale-raw-4a8bd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:28:43.952260 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:43.952220    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgfb2\" (UniqueName: \"kubernetes.io/projected/2c2670ee-0cf0-4b98-873d-182daa4ce2c4-kube-api-access-fgfb2\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:28:43.952260 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:43.952230    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c2670ee-0cf0-4b98-873d-182daa4ce2c4-kserve-provision-location\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:28:44.615865 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:44.615834    2563 generic.go:358] "Generic (PLEG): container finished" podID="2c2670ee-0cf0-4b98-873d-182daa4ce2c4" containerID="8301882e4b4b9c40562c4f6f8bc8ee2fa9a9505080b2ade9373a7d44e9f34fc4" exitCode=0
May 25 12:28:44.616071 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:44.615912    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w"
May 25 12:28:44.616071 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:44.615924    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w" event={"ID":"2c2670ee-0cf0-4b98-873d-182daa4ce2c4","Type":"ContainerDied","Data":"8301882e4b4b9c40562c4f6f8bc8ee2fa9a9505080b2ade9373a7d44e9f34fc4"}
May 25 12:28:44.616071 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:44.615966    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w" event={"ID":"2c2670ee-0cf0-4b98-873d-182daa4ce2c4","Type":"ContainerDied","Data":"1c7612b2bdc6f339c789d03f32767a4fa6dd467a4ec0f43ef9d85337fcac881f"}
May 25 12:28:44.616071 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:44.615988    2563 scope.go:117] "RemoveContainer" containerID="d3435d39fce5807ca88ff905eecb013a7798ced579b6743daa246a056ef14dd8"
May 25 12:28:44.623589 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:44.623571    2563 scope.go:117] "RemoveContainer" containerID="8301882e4b4b9c40562c4f6f8bc8ee2fa9a9505080b2ade9373a7d44e9f34fc4"
May 25 12:28:44.631013 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:44.630995    2563 scope.go:117] "RemoveContainer" containerID="8ed057c6ed33d0be76cc2ea5a88a19d72ef623aa4652d6deb930401f62ba797d"
May 25 12:28:44.637819 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:44.637802    2563 scope.go:117] "RemoveContainer" containerID="d3435d39fce5807ca88ff905eecb013a7798ced579b6743daa246a056ef14dd8"
May 25 12:28:44.638042 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:28:44.638022    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3435d39fce5807ca88ff905eecb013a7798ced579b6743daa246a056ef14dd8\": container with ID starting with d3435d39fce5807ca88ff905eecb013a7798ced579b6743daa246a056ef14dd8 not found: ID does not exist" containerID="d3435d39fce5807ca88ff905eecb013a7798ced579b6743daa246a056ef14dd8"
May 25 12:28:44.638120 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:44.638056    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3435d39fce5807ca88ff905eecb013a7798ced579b6743daa246a056ef14dd8"} err="failed to get container status \"d3435d39fce5807ca88ff905eecb013a7798ced579b6743daa246a056ef14dd8\": rpc error: code = NotFound desc = could not find container \"d3435d39fce5807ca88ff905eecb013a7798ced579b6743daa246a056ef14dd8\": container with ID starting with d3435d39fce5807ca88ff905eecb013a7798ced579b6743daa246a056ef14dd8 not found: ID does not exist"
May 25 12:28:44.638120 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:44.638082    2563 scope.go:117] "RemoveContainer" containerID="8301882e4b4b9c40562c4f6f8bc8ee2fa9a9505080b2ade9373a7d44e9f34fc4"
May 25 12:28:44.638306 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:28:44.638287    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8301882e4b4b9c40562c4f6f8bc8ee2fa9a9505080b2ade9373a7d44e9f34fc4\": container with ID starting with 8301882e4b4b9c40562c4f6f8bc8ee2fa9a9505080b2ade9373a7d44e9f34fc4 not found: ID does not exist" containerID="8301882e4b4b9c40562c4f6f8bc8ee2fa9a9505080b2ade9373a7d44e9f34fc4"
May 25 12:28:44.638349 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:44.638312    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8301882e4b4b9c40562c4f6f8bc8ee2fa9a9505080b2ade9373a7d44e9f34fc4"} err="failed to get container status \"8301882e4b4b9c40562c4f6f8bc8ee2fa9a9505080b2ade9373a7d44e9f34fc4\": rpc error: code = NotFound desc = could not find container \"8301882e4b4b9c40562c4f6f8bc8ee2fa9a9505080b2ade9373a7d44e9f34fc4\": container with ID starting with 8301882e4b4b9c40562c4f6f8bc8ee2fa9a9505080b2ade9373a7d44e9f34fc4 not found: ID does not exist"
May 25 12:28:44.638349 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:44.638328    2563 scope.go:117] "RemoveContainer" containerID="8ed057c6ed33d0be76cc2ea5a88a19d72ef623aa4652d6deb930401f62ba797d"
May 25 12:28:44.638570 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:28:44.638552    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ed057c6ed33d0be76cc2ea5a88a19d72ef623aa4652d6deb930401f62ba797d\": container with ID starting with 8ed057c6ed33d0be76cc2ea5a88a19d72ef623aa4652d6deb930401f62ba797d not found: ID does not exist" containerID="8ed057c6ed33d0be76cc2ea5a88a19d72ef623aa4652d6deb930401f62ba797d"
May 25 12:28:44.638624 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:44.638576    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ed057c6ed33d0be76cc2ea5a88a19d72ef623aa4652d6deb930401f62ba797d"} err="failed to get container status \"8ed057c6ed33d0be76cc2ea5a88a19d72ef623aa4652d6deb930401f62ba797d\": rpc error: code = NotFound desc = could not find container \"8ed057c6ed33d0be76cc2ea5a88a19d72ef623aa4652d6deb930401f62ba797d\": container with ID starting with 8ed057c6ed33d0be76cc2ea5a88a19d72ef623aa4652d6deb930401f62ba797d not found: ID does not exist"
May 25 12:28:44.641020 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:44.641000    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w"]
May 25 12:28:44.646041 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:44.646012    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-4a8bd-predictor-6c78c97cc5-dj79w"]
May 25 12:28:46.636848 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:46.636809    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c2670ee-0cf0-4b98-873d-182daa4ce2c4" path="/var/lib/kubelet/pods/2c2670ee-0cf0-4b98-873d-182daa4ce2c4/volumes"
May 25 12:28:47.613152 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:47.613121    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm"
May 25 12:28:47.613693 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:47.613658    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm" podUID="2f14ed01-2147-4647-b02a-17683660e3f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
May 25 12:28:57.614264 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:28:57.614227    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm" podUID="2f14ed01-2147-4647-b02a-17683660e3f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
May 25 12:29:07.614690 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:29:07.614646    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm" podUID="2f14ed01-2147-4647-b02a-17683660e3f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
May 25 12:29:17.613943 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:29:17.613898    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm" podUID="2f14ed01-2147-4647-b02a-17683660e3f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
May 25 12:29:27.613709 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:29:27.613668    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm" podUID="2f14ed01-2147-4647-b02a-17683660e3f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
May 25 12:29:37.614701 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:29:37.614659    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm" podUID="2f14ed01-2147-4647-b02a-17683660e3f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
May 25 12:29:47.614461 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:29:47.614431    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm"
May 25 12:29:54.702877 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:29:54.702842    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-10f269-predictor-56d976fd67-72btd"]
May 25 12:29:54.703291 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:29:54.703148    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c2670ee-0cf0-4b98-873d-182daa4ce2c4" containerName="storage-initializer"
May 25 12:29:54.703291 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:29:54.703174    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c2670ee-0cf0-4b98-873d-182daa4ce2c4" containerName="storage-initializer"
May 25 12:29:54.703291 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:29:54.703187    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c2670ee-0cf0-4b98-873d-182daa4ce2c4" containerName="kserve-container"
May 25 12:29:54.703291 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:29:54.703192    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c2670ee-0cf0-4b98-873d-182daa4ce2c4" containerName="kserve-container"
May 25 12:29:54.703291 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:29:54.703208    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c2670ee-0cf0-4b98-873d-182daa4ce2c4" containerName="kube-rbac-proxy"
May 25 12:29:54.703291 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:29:54.703214    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c2670ee-0cf0-4b98-873d-182daa4ce2c4" containerName="kube-rbac-proxy"
May 25 12:29:54.703291 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:29:54.703257    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="2c2670ee-0cf0-4b98-873d-182daa4ce2c4" containerName="kserve-container"
May 25 12:29:54.703291 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:29:54.703267    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="2c2670ee-0cf0-4b98-873d-182daa4ce2c4" containerName="kube-rbac-proxy"
May 25 12:29:54.706504 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:29:54.706486    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-10f269-predictor-56d976fd67-72btd"
May 25 12:29:54.710460 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:29:54.710434    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-10f269\""
May 25 12:29:54.710460 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:29:54.710434    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-10f269-predictor-serving-cert\""
May 25 12:29:54.710673 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:29:54.710533    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-10f269-kube-rbac-proxy-sar-config\""
May 25 12:29:54.710673 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:29:54.710535    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
May 25 12:29:54.710673 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:29:54.710651    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-10f269-dockercfg-pjtgl\""
May 25 12:29:54.717718 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:29:54.717696    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-10f269-predictor-56d976fd67-72btd"]
May 25 12:29:54.815670 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:29:54.815637    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-secondary-10f269-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a260d669-c5e7-4595-ab9b-fb5a2a180c8a-isvc-secondary-10f269-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-10f269-predictor-56d976fd67-72btd\" (UID: \"a260d669-c5e7-4595-ab9b-fb5a2a180c8a\") " pod="kserve-ci-e2e-test/isvc-secondary-10f269-predictor-56d976fd67-72btd"
May 25 12:29:54.815850 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:29:54.815700    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a260d669-c5e7-4595-ab9b-fb5a2a180c8a-proxy-tls\") pod \"isvc-secondary-10f269-predictor-56d976fd67-72btd\" (UID: \"a260d669-c5e7-4595-ab9b-fb5a2a180c8a\") " pod="kserve-ci-e2e-test/isvc-secondary-10f269-predictor-56d976fd67-72btd"
May 25 12:29:54.815850 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:29:54.815755    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a260d669-c5e7-4595-ab9b-fb5a2a180c8a-kserve-provision-location\") pod \"isvc-secondary-10f269-predictor-56d976fd67-72btd\" (UID: \"a260d669-c5e7-4595-ab9b-fb5a2a180c8a\") " pod="kserve-ci-e2e-test/isvc-secondary-10f269-predictor-56d976fd67-72btd"
May 25 12:29:54.815850 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:29:54.815800    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a260d669-c5e7-4595-ab9b-fb5a2a180c8a-cabundle-cert\") pod \"isvc-secondary-10f269-predictor-56d976fd67-72btd\" (UID: \"a260d669-c5e7-4595-ab9b-fb5a2a180c8a\") " pod="kserve-ci-e2e-test/isvc-secondary-10f269-predictor-56d976fd67-72btd"
May 25 12:29:54.816009 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:29:54.815870    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4ksl\" (UniqueName: \"kubernetes.io/projected/a260d669-c5e7-4595-ab9b-fb5a2a180c8a-kube-api-access-s4ksl\") pod \"isvc-secondary-10f269-predictor-56d976fd67-72btd\" (UID: \"a260d669-c5e7-4595-ab9b-fb5a2a180c8a\") " pod="kserve-ci-e2e-test/isvc-secondary-10f269-predictor-56d976fd67-72btd"
May 25 12:29:54.916899 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:29:54.916866    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a260d669-c5e7-4595-ab9b-fb5a2a180c8a-proxy-tls\") pod \"isvc-secondary-10f269-predictor-56d976fd67-72btd\" (UID: \"a260d669-c5e7-4595-ab9b-fb5a2a180c8a\") " pod="kserve-ci-e2e-test/isvc-secondary-10f269-predictor-56d976fd67-72btd"
May 25 12:29:54.917083 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:29:54.916932    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a260d669-c5e7-4595-ab9b-fb5a2a180c8a-kserve-provision-location\") pod \"isvc-secondary-10f269-predictor-56d976fd67-72btd\" (UID: \"a260d669-c5e7-4595-ab9b-fb5a2a180c8a\") " pod="kserve-ci-e2e-test/isvc-secondary-10f269-predictor-56d976fd67-72btd"
May 25 12:29:54.917083 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:29:54.916966    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a260d669-c5e7-4595-ab9b-fb5a2a180c8a-cabundle-cert\") pod \"isvc-secondary-10f269-predictor-56d976fd67-72btd\" (UID: \"a260d669-c5e7-4595-ab9b-fb5a2a180c8a\") " pod="kserve-ci-e2e-test/isvc-secondary-10f269-predictor-56d976fd67-72btd"
May 25 12:29:54.917083 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:29:54.917009    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-s4ksl\" (UniqueName: \"kubernetes.io/projected/a260d669-c5e7-4595-ab9b-fb5a2a180c8a-kube-api-access-s4ksl\") pod \"isvc-secondary-10f269-predictor-56d976fd67-72btd\" (UID: \"a260d669-c5e7-4595-ab9b-fb5a2a180c8a\") " pod="kserve-ci-e2e-test/isvc-secondary-10f269-predictor-56d976fd67-72btd"
May 25 12:29:54.917083 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:29:54.917045    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-secondary-10f269-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a260d669-c5e7-4595-ab9b-fb5a2a180c8a-isvc-secondary-10f269-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-10f269-predictor-56d976fd67-72btd\" (UID: \"a260d669-c5e7-4595-ab9b-fb5a2a180c8a\") " pod="kserve-ci-e2e-test/isvc-secondary-10f269-predictor-56d976fd67-72btd"
May 25 12:29:54.917388 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:29:54.917367    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a260d669-c5e7-4595-ab9b-fb5a2a180c8a-kserve-provision-location\") pod \"isvc-secondary-10f269-predictor-56d976fd67-72btd\" (UID: \"a260d669-c5e7-4595-ab9b-fb5a2a180c8a\") " pod="kserve-ci-e2e-test/isvc-secondary-10f269-predictor-56d976fd67-72btd"
May 25 12:29:54.917755 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:29:54.917737    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a260d669-c5e7-4595-ab9b-fb5a2a180c8a-cabundle-cert\") pod \"isvc-secondary-10f269-predictor-56d976fd67-72btd\" (UID: \"a260d669-c5e7-4595-ab9b-fb5a2a180c8a\") " pod="kserve-ci-e2e-test/isvc-secondary-10f269-predictor-56d976fd67-72btd"
May 25 12:29:54.918041 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:29:54.918021    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-secondary-10f269-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a260d669-c5e7-4595-ab9b-fb5a2a180c8a-isvc-secondary-10f269-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-10f269-predictor-56d976fd67-72btd\" (UID: \"a260d669-c5e7-4595-ab9b-fb5a2a180c8a\") " pod="kserve-ci-e2e-test/isvc-secondary-10f269-predictor-56d976fd67-72btd"
May 25 12:29:54.919448 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:29:54.919430    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a260d669-c5e7-4595-ab9b-fb5a2a180c8a-proxy-tls\") pod \"isvc-secondary-10f269-predictor-56d976fd67-72btd\" (UID: \"a260d669-c5e7-4595-ab9b-fb5a2a180c8a\") " pod="kserve-ci-e2e-test/isvc-secondary-10f269-predictor-56d976fd67-72btd"
May 25 12:29:54.926118 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:29:54.926085    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4ksl\" (UniqueName: \"kubernetes.io/projected/a260d669-c5e7-4595-ab9b-fb5a2a180c8a-kube-api-access-s4ksl\") pod \"isvc-secondary-10f269-predictor-56d976fd67-72btd\" (UID: \"a260d669-c5e7-4595-ab9b-fb5a2a180c8a\") " pod="kserve-ci-e2e-test/isvc-secondary-10f269-predictor-56d976fd67-72btd"
May 25 12:29:55.016970 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:29:55.016935    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-10f269-predictor-56d976fd67-72btd"
May 25 12:29:55.145095 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:29:55.145070    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-10f269-predictor-56d976fd67-72btd"]
May 25 12:29:55.147678 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:29:55.147651    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda260d669_c5e7_4595_ab9b_fb5a2a180c8a.slice/crio-635aeb10e1abf7a76bf5731dcd6188757b17132485cb80076e570eab3c142e9d WatchSource:0}: Error finding container 635aeb10e1abf7a76bf5731dcd6188757b17132485cb80076e570eab3c142e9d: Status 404 returned error can't find the container with id 635aeb10e1abf7a76bf5731dcd6188757b17132485cb80076e570eab3c142e9d
May 25 12:29:55.149416 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:29:55.149396    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 12:29:55.810139 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:29:55.810099    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-10f269-predictor-56d976fd67-72btd" event={"ID":"a260d669-c5e7-4595-ab9b-fb5a2a180c8a","Type":"ContainerStarted","Data":"394f2e6c99feafb0988e97fea12bfcbfb273e82a70ad80ce312a5bfe53880c5c"}
May 25 12:29:55.810558 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:29:55.810146    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-10f269-predictor-56d976fd67-72btd" event={"ID":"a260d669-c5e7-4595-ab9b-fb5a2a180c8a","Type":"ContainerStarted","Data":"635aeb10e1abf7a76bf5731dcd6188757b17132485cb80076e570eab3c142e9d"}
May 25 12:30:01.828349 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:01.828323    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-10f269-predictor-56d976fd67-72btd_a260d669-c5e7-4595-ab9b-fb5a2a180c8a/storage-initializer/0.log"
May 25 12:30:01.828745 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:01.828360    2563 generic.go:358] "Generic (PLEG): container finished" podID="a260d669-c5e7-4595-ab9b-fb5a2a180c8a" containerID="394f2e6c99feafb0988e97fea12bfcbfb273e82a70ad80ce312a5bfe53880c5c" exitCode=1
May 25 12:30:01.828745 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:01.828400    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-10f269-predictor-56d976fd67-72btd" event={"ID":"a260d669-c5e7-4595-ab9b-fb5a2a180c8a","Type":"ContainerDied","Data":"394f2e6c99feafb0988e97fea12bfcbfb273e82a70ad80ce312a5bfe53880c5c"}
May 25 12:30:02.833043 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:02.833012    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-10f269-predictor-56d976fd67-72btd_a260d669-c5e7-4595-ab9b-fb5a2a180c8a/storage-initializer/0.log"
May 25 12:30:02.833436 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:02.833075    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-10f269-predictor-56d976fd67-72btd" event={"ID":"a260d669-c5e7-4595-ab9b-fb5a2a180c8a","Type":"ContainerStarted","Data":"aeffe13bb9242c0a4ab2cd596aa1a1e1e2adcd5f6090a570cb5e7617767a4491"}
May 25 12:30:07.848173 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:07.848143    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-10f269-predictor-56d976fd67-72btd_a260d669-c5e7-4595-ab9b-fb5a2a180c8a/storage-initializer/1.log"
May 25 12:30:07.848623 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:07.848473    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-10f269-predictor-56d976fd67-72btd_a260d669-c5e7-4595-ab9b-fb5a2a180c8a/storage-initializer/0.log"
May 25 12:30:07.848623 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:07.848505    2563 generic.go:358] "Generic (PLEG): container finished" podID="a260d669-c5e7-4595-ab9b-fb5a2a180c8a" containerID="aeffe13bb9242c0a4ab2cd596aa1a1e1e2adcd5f6090a570cb5e7617767a4491" exitCode=1
May 25 12:30:07.848623 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:07.848547    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-10f269-predictor-56d976fd67-72btd" event={"ID":"a260d669-c5e7-4595-ab9b-fb5a2a180c8a","Type":"ContainerDied","Data":"aeffe13bb9242c0a4ab2cd596aa1a1e1e2adcd5f6090a570cb5e7617767a4491"}
May 25 12:30:07.848623 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:07.848583    2563 scope.go:117] "RemoveContainer" containerID="394f2e6c99feafb0988e97fea12bfcbfb273e82a70ad80ce312a5bfe53880c5c"
May 25 12:30:07.848978 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:07.848962    2563 scope.go:117] "RemoveContainer" containerID="394f2e6c99feafb0988e97fea12bfcbfb273e82a70ad80ce312a5bfe53880c5c"
May 25 12:30:07.859018 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:30:07.858976    2563 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-10f269-predictor-56d976fd67-72btd_kserve-ci-e2e-test_a260d669-c5e7-4595-ab9b-fb5a2a180c8a_0 in pod sandbox 635aeb10e1abf7a76bf5731dcd6188757b17132485cb80076e570eab3c142e9d from index: no such id: '394f2e6c99feafb0988e97fea12bfcbfb273e82a70ad80ce312a5bfe53880c5c'" containerID="394f2e6c99feafb0988e97fea12bfcbfb273e82a70ad80ce312a5bfe53880c5c"
May 25 12:30:07.859165 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:30:07.859035    2563 kuberuntime_container.go:951] "Unhandled Error" err="failed to remove pod init container \"storage-initializer\": rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-10f269-predictor-56d976fd67-72btd_kserve-ci-e2e-test_a260d669-c5e7-4595-ab9b-fb5a2a180c8a_0 in pod sandbox 635aeb10e1abf7a76bf5731dcd6188757b17132485cb80076e570eab3c142e9d from index: no such id: '394f2e6c99feafb0988e97fea12bfcbfb273e82a70ad80ce312a5bfe53880c5c'; Skipping pod \"isvc-secondary-10f269-predictor-56d976fd67-72btd_kserve-ci-e2e-test(a260d669-c5e7-4595-ab9b-fb5a2a180c8a)\"" logger="UnhandledError"
May 25 12:30:07.860363 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:30:07.860335    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-10f269-predictor-56d976fd67-72btd_kserve-ci-e2e-test(a260d669-c5e7-4595-ab9b-fb5a2a180c8a)\"" pod="kserve-ci-e2e-test/isvc-secondary-10f269-predictor-56d976fd67-72btd" podUID="a260d669-c5e7-4595-ab9b-fb5a2a180c8a"
May 25 12:30:08.852157 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:08.852129    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-10f269-predictor-56d976fd67-72btd_a260d669-c5e7-4595-ab9b-fb5a2a180c8a/storage-initializer/1.log"
May 25 12:30:12.732062 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:12.732026    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm"]
May 25 12:30:12.732465 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:12.732358    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm" podUID="2f14ed01-2147-4647-b02a-17683660e3f2" containerName="kserve-container" containerID="cri-o://282d7076d3e212876058185f1ba321a41364376e490b1a3c82041935bf901e44" gracePeriod=30
May 25 12:30:12.732465 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:12.732404    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm" podUID="2f14ed01-2147-4647-b02a-17683660e3f2" containerName="kube-rbac-proxy" containerID="cri-o://3afce9aac8ea3babe73169c1b4572ca166149e6b62f5886a7611c6c0d2eb2f55" gracePeriod=30
May 25 12:30:12.806642 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:12.806606    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-10f269-predictor-56d976fd67-72btd"]
May 25 12:30:12.868872 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:12.868844    2563 generic.go:358] "Generic (PLEG): container finished" podID="2f14ed01-2147-4647-b02a-17683660e3f2" containerID="3afce9aac8ea3babe73169c1b4572ca166149e6b62f5886a7611c6c0d2eb2f55" exitCode=2
May 25 12:30:12.868996 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:12.868924    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm" event={"ID":"2f14ed01-2147-4647-b02a-17683660e3f2","Type":"ContainerDied","Data":"3afce9aac8ea3babe73169c1b4572ca166149e6b62f5886a7611c6c0d2eb2f55"}
May 25 12:30:12.903330 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:12.903295    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6"]
May 25 12:30:12.906658 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:12.906634    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6"
May 25 12:30:12.908810 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:12.908725    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-7ee0ab-predictor-serving-cert\""
May 25 12:30:12.908810 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:12.908745    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-7ee0ab\""
May 25 12:30:12.908810 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:12.908788    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-7ee0ab-dockercfg-2qffx\""
May 25 12:30:12.909128 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:12.908788    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-7ee0ab-kube-rbac-proxy-sar-config\""
May 25 12:30:12.915709 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:12.915244    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6"]
May 25 12:30:12.950269 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:12.950247    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-10f269-predictor-56d976fd67-72btd_a260d669-c5e7-4595-ab9b-fb5a2a180c8a/storage-initializer/1.log"
May 25 12:30:12.950399 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:12.950309    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-10f269-predictor-56d976fd67-72btd"
May 25 12:30:13.067805 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:13.067767    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a260d669-c5e7-4595-ab9b-fb5a2a180c8a-cabundle-cert\") pod \"a260d669-c5e7-4595-ab9b-fb5a2a180c8a\" (UID: \"a260d669-c5e7-4595-ab9b-fb5a2a180c8a\") "
May 25 12:30:13.068014 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:13.067833    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4ksl\" (UniqueName: \"kubernetes.io/projected/a260d669-c5e7-4595-ab9b-fb5a2a180c8a-kube-api-access-s4ksl\") pod \"a260d669-c5e7-4595-ab9b-fb5a2a180c8a\" (UID: \"a260d669-c5e7-4595-ab9b-fb5a2a180c8a\") "
May 25 12:30:13.068014 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:13.067862    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-secondary-10f269-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a260d669-c5e7-4595-ab9b-fb5a2a180c8a-isvc-secondary-10f269-kube-rbac-proxy-sar-config\") pod \"a260d669-c5e7-4595-ab9b-fb5a2a180c8a\" (UID: \"a260d669-c5e7-4595-ab9b-fb5a2a180c8a\") "
May 25 12:30:13.068014 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:13.067905    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a260d669-c5e7-4595-ab9b-fb5a2a180c8a-proxy-tls\") pod \"a260d669-c5e7-4595-ab9b-fb5a2a180c8a\" (UID: \"a260d669-c5e7-4595-ab9b-fb5a2a180c8a\") "
May 25 12:30:13.068014 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:13.067951    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a260d669-c5e7-4595-ab9b-fb5a2a180c8a-kserve-provision-location\") pod \"a260d669-c5e7-4595-ab9b-fb5a2a180c8a\" (UID: \"a260d669-c5e7-4595-ab9b-fb5a2a180c8a\") "
May 25 12:30:13.068223 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:13.068106    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-init-fail-7ee0ab-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e11c8aa2-1352-4ec6-9142-ba20823eb22d-isvc-init-fail-7ee0ab-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6\" (UID: \"e11c8aa2-1352-4ec6-9142-ba20823eb22d\") " pod="kserve-ci-e2e-test/isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6"
May 25 12:30:13.068223 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:13.068169    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2db4\" (UniqueName: \"kubernetes.io/projected/e11c8aa2-1352-4ec6-9142-ba20823eb22d-kube-api-access-f2db4\") pod \"isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6\" (UID: \"e11c8aa2-1352-4ec6-9142-ba20823eb22d\") " pod="kserve-ci-e2e-test/isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6"
May 25 12:30:13.068505 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:13.068254    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a260d669-c5e7-4595-ab9b-fb5a2a180c8a-isvc-secondary-10f269-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-secondary-10f269-kube-rbac-proxy-sar-config") pod "a260d669-c5e7-4595-ab9b-fb5a2a180c8a" (UID: "a260d669-c5e7-4595-ab9b-fb5a2a180c8a"). InnerVolumeSpecName "isvc-secondary-10f269-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:30:13.068505 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:13.068291    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a260d669-c5e7-4595-ab9b-fb5a2a180c8a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a260d669-c5e7-4595-ab9b-fb5a2a180c8a" (UID: "a260d669-c5e7-4595-ab9b-fb5a2a180c8a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:30:13.068505 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:13.068292    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e11c8aa2-1352-4ec6-9142-ba20823eb22d-cabundle-cert\") pod \"isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6\" (UID: \"e11c8aa2-1352-4ec6-9142-ba20823eb22d\") " pod="kserve-ci-e2e-test/isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6"
May 25 12:30:13.068505 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:13.068352    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e11c8aa2-1352-4ec6-9142-ba20823eb22d-proxy-tls\") pod \"isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6\" (UID: \"e11c8aa2-1352-4ec6-9142-ba20823eb22d\") " pod="kserve-ci-e2e-test/isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6"
May 25 12:30:13.068505 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:13.068384    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e11c8aa2-1352-4ec6-9142-ba20823eb22d-kserve-provision-location\") pod \"isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6\" (UID: \"e11c8aa2-1352-4ec6-9142-ba20823eb22d\") " pod="kserve-ci-e2e-test/isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6"
May 25 12:30:13.068505 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:13.068413    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a260d669-c5e7-4595-ab9b-fb5a2a180c8a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a260d669-c5e7-4595-ab9b-fb5a2a180c8a" (UID: "a260d669-c5e7-4595-ab9b-fb5a2a180c8a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:30:13.068505 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:13.068467    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-secondary-10f269-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a260d669-c5e7-4595-ab9b-fb5a2a180c8a-isvc-secondary-10f269-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:30:13.068505 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:13.068480    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a260d669-c5e7-4595-ab9b-fb5a2a180c8a-kserve-provision-location\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:30:13.070020 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:13.069997    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a260d669-c5e7-4595-ab9b-fb5a2a180c8a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a260d669-c5e7-4595-ab9b-fb5a2a180c8a" (UID: "a260d669-c5e7-4595-ab9b-fb5a2a180c8a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:30:13.070112 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:13.070090    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a260d669-c5e7-4595-ab9b-fb5a2a180c8a-kube-api-access-s4ksl" (OuterVolumeSpecName: "kube-api-access-s4ksl") pod "a260d669-c5e7-4595-ab9b-fb5a2a180c8a" (UID: "a260d669-c5e7-4595-ab9b-fb5a2a180c8a"). InnerVolumeSpecName "kube-api-access-s4ksl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:30:13.169308 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:13.169267    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e11c8aa2-1352-4ec6-9142-ba20823eb22d-cabundle-cert\") pod \"isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6\" (UID: \"e11c8aa2-1352-4ec6-9142-ba20823eb22d\") " pod="kserve-ci-e2e-test/isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6"
May 25 12:30:13.169483 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:13.169317    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e11c8aa2-1352-4ec6-9142-ba20823eb22d-proxy-tls\") pod \"isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6\" (UID: \"e11c8aa2-1352-4ec6-9142-ba20823eb22d\") " pod="kserve-ci-e2e-test/isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6"
May 25 12:30:13.169483 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:13.169337    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e11c8aa2-1352-4ec6-9142-ba20823eb22d-kserve-provision-location\") pod \"isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6\" (UID: \"e11c8aa2-1352-4ec6-9142-ba20823eb22d\") " pod="kserve-ci-e2e-test/isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6"
May 25 12:30:13.169483 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:13.169388    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-init-fail-7ee0ab-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e11c8aa2-1352-4ec6-9142-ba20823eb22d-isvc-init-fail-7ee0ab-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6\" (UID: \"e11c8aa2-1352-4ec6-9142-ba20823eb22d\") " pod="kserve-ci-e2e-test/isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6"
May 25 12:30:13.169483 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:13.169411    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f2db4\" (UniqueName: \"kubernetes.io/projected/e11c8aa2-1352-4ec6-9142-ba20823eb22d-kube-api-access-f2db4\") pod \"isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6\" (UID: \"e11c8aa2-1352-4ec6-9142-ba20823eb22d\") " pod="kserve-ci-e2e-test/isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6"
May 25 12:30:13.169483 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:13.169448    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a260d669-c5e7-4595-ab9b-fb5a2a180c8a-proxy-tls\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:30:13.169483 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:13.169462    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a260d669-c5e7-4595-ab9b-fb5a2a180c8a-cabundle-cert\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:30:13.169483 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:13.169476    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4ksl\" (UniqueName: \"kubernetes.io/projected/a260d669-c5e7-4595-ab9b-fb5a2a180c8a-kube-api-access-s4ksl\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:30:13.169895 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:13.169861    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e11c8aa2-1352-4ec6-9142-ba20823eb22d-kserve-provision-location\") pod \"isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6\" (UID: \"e11c8aa2-1352-4ec6-9142-ba20823eb22d\") " pod="kserve-ci-e2e-test/isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6"
May 25 12:30:13.170097 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:13.170077    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e11c8aa2-1352-4ec6-9142-ba20823eb22d-cabundle-cert\") pod \"isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6\" (UID: \"e11c8aa2-1352-4ec6-9142-ba20823eb22d\") " pod="kserve-ci-e2e-test/isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6"
May 25 12:30:13.170169 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:13.170149    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-init-fail-7ee0ab-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e11c8aa2-1352-4ec6-9142-ba20823eb22d-isvc-init-fail-7ee0ab-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6\" (UID: \"e11c8aa2-1352-4ec6-9142-ba20823eb22d\") " pod="kserve-ci-e2e-test/isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6"
May 25 12:30:13.171778 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:13.171759    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e11c8aa2-1352-4ec6-9142-ba20823eb22d-proxy-tls\") pod \"isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6\" (UID: \"e11c8aa2-1352-4ec6-9142-ba20823eb22d\") " pod="kserve-ci-e2e-test/isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6"
May 25 12:30:13.178025 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:13.178007    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2db4\" (UniqueName: \"kubernetes.io/projected/e11c8aa2-1352-4ec6-9142-ba20823eb22d-kube-api-access-f2db4\") pod \"isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6\" (UID: \"e11c8aa2-1352-4ec6-9142-ba20823eb22d\") " pod="kserve-ci-e2e-test/isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6"
May 25 12:30:13.220182 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:13.220140    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6"
May 25 12:30:13.354257 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:13.354159    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6"]
May 25 12:30:13.358484 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:30:13.358457    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode11c8aa2_1352_4ec6_9142_ba20823eb22d.slice/crio-bb2c4460d7afbabed40d8de7227b4d71c4ddf057be4ce3f0845b37dbeabec9e7 WatchSource:0}: Error finding container bb2c4460d7afbabed40d8de7227b4d71c4ddf057be4ce3f0845b37dbeabec9e7: Status 404 returned error can't find the container with id bb2c4460d7afbabed40d8de7227b4d71c4ddf057be4ce3f0845b37dbeabec9e7
May 25 12:30:13.873320 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:13.873269    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6" event={"ID":"e11c8aa2-1352-4ec6-9142-ba20823eb22d","Type":"ContainerStarted","Data":"5a71bb8b45dfc1c7a1ab05519ec88d3f98653369408b0be7d2a683c8dc37a651"}
May 25 12:30:13.873320 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:13.873320    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6" event={"ID":"e11c8aa2-1352-4ec6-9142-ba20823eb22d","Type":"ContainerStarted","Data":"bb2c4460d7afbabed40d8de7227b4d71c4ddf057be4ce3f0845b37dbeabec9e7"}
May 25 12:30:13.874483 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:13.874464    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-10f269-predictor-56d976fd67-72btd_a260d669-c5e7-4595-ab9b-fb5a2a180c8a/storage-initializer/1.log"
May 25 12:30:13.874594 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:13.874574    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-10f269-predictor-56d976fd67-72btd" event={"ID":"a260d669-c5e7-4595-ab9b-fb5a2a180c8a","Type":"ContainerDied","Data":"635aeb10e1abf7a76bf5731dcd6188757b17132485cb80076e570eab3c142e9d"}
May 25 12:30:13.874644 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:13.874593    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-10f269-predictor-56d976fd67-72btd"
May 25 12:30:13.874644 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:13.874617    2563 scope.go:117] "RemoveContainer" containerID="aeffe13bb9242c0a4ab2cd596aa1a1e1e2adcd5f6090a570cb5e7617767a4491"
May 25 12:30:13.918805 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:13.918768    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-10f269-predictor-56d976fd67-72btd"]
May 25 12:30:13.922931 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:13.922904    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-10f269-predictor-56d976fd67-72btd"]
May 25 12:30:14.637044 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:14.637008    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a260d669-c5e7-4595-ab9b-fb5a2a180c8a" path="/var/lib/kubelet/pods/a260d669-c5e7-4595-ab9b-fb5a2a180c8a/volumes"
May 25 12:30:17.383762 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:17.383736    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm"
May 25 12:30:17.509093 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:17.509057    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9nh5\" (UniqueName: \"kubernetes.io/projected/2f14ed01-2147-4647-b02a-17683660e3f2-kube-api-access-v9nh5\") pod \"2f14ed01-2147-4647-b02a-17683660e3f2\" (UID: \"2f14ed01-2147-4647-b02a-17683660e3f2\") "
May 25 12:30:17.509298 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:17.509126    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f14ed01-2147-4647-b02a-17683660e3f2-kserve-provision-location\") pod \"2f14ed01-2147-4647-b02a-17683660e3f2\" (UID: \"2f14ed01-2147-4647-b02a-17683660e3f2\") "
May 25 12:30:17.509298 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:17.509148    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f14ed01-2147-4647-b02a-17683660e3f2-proxy-tls\") pod \"2f14ed01-2147-4647-b02a-17683660e3f2\" (UID: \"2f14ed01-2147-4647-b02a-17683660e3f2\") "
May 25 12:30:17.509298 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:17.509188    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-primary-10f269-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f14ed01-2147-4647-b02a-17683660e3f2-isvc-primary-10f269-kube-rbac-proxy-sar-config\") pod \"2f14ed01-2147-4647-b02a-17683660e3f2\" (UID: \"2f14ed01-2147-4647-b02a-17683660e3f2\") "
May 25 12:30:17.509603 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:17.509571    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f14ed01-2147-4647-b02a-17683660e3f2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2f14ed01-2147-4647-b02a-17683660e3f2" (UID: "2f14ed01-2147-4647-b02a-17683660e3f2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:30:17.509678 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:17.509613    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f14ed01-2147-4647-b02a-17683660e3f2-isvc-primary-10f269-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-primary-10f269-kube-rbac-proxy-sar-config") pod "2f14ed01-2147-4647-b02a-17683660e3f2" (UID: "2f14ed01-2147-4647-b02a-17683660e3f2"). InnerVolumeSpecName "isvc-primary-10f269-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:30:17.511194 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:17.511169    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f14ed01-2147-4647-b02a-17683660e3f2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2f14ed01-2147-4647-b02a-17683660e3f2" (UID: "2f14ed01-2147-4647-b02a-17683660e3f2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:30:17.511291 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:17.511263    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f14ed01-2147-4647-b02a-17683660e3f2-kube-api-access-v9nh5" (OuterVolumeSpecName: "kube-api-access-v9nh5") pod "2f14ed01-2147-4647-b02a-17683660e3f2" (UID: "2f14ed01-2147-4647-b02a-17683660e3f2"). InnerVolumeSpecName "kube-api-access-v9nh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:30:17.610493 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:17.610451    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-primary-10f269-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f14ed01-2147-4647-b02a-17683660e3f2-isvc-primary-10f269-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:30:17.610493 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:17.610486    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9nh5\" (UniqueName: \"kubernetes.io/projected/2f14ed01-2147-4647-b02a-17683660e3f2-kube-api-access-v9nh5\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:30:17.610493 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:17.610496    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f14ed01-2147-4647-b02a-17683660e3f2-kserve-provision-location\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:30:17.610493 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:17.610505    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f14ed01-2147-4647-b02a-17683660e3f2-proxy-tls\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:30:17.888290 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:17.888204    2563 generic.go:358] "Generic (PLEG): container finished" podID="2f14ed01-2147-4647-b02a-17683660e3f2" containerID="282d7076d3e212876058185f1ba321a41364376e490b1a3c82041935bf901e44" exitCode=0
May 25 12:30:17.888290 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:17.888275    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm" event={"ID":"2f14ed01-2147-4647-b02a-17683660e3f2","Type":"ContainerDied","Data":"282d7076d3e212876058185f1ba321a41364376e490b1a3c82041935bf901e44"}
May 25 12:30:17.888555 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:17.888298    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm"
May 25 12:30:17.888555 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:17.888309    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm" event={"ID":"2f14ed01-2147-4647-b02a-17683660e3f2","Type":"ContainerDied","Data":"1f02273d2166cd8dbc54fbc5143b5995a3aef97e9947376905260864403f8d97"}
May 25 12:30:17.888555 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:17.888329    2563 scope.go:117] "RemoveContainer" containerID="3afce9aac8ea3babe73169c1b4572ca166149e6b62f5886a7611c6c0d2eb2f55"
May 25 12:30:17.896042 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:17.895932    2563 scope.go:117] "RemoveContainer" containerID="282d7076d3e212876058185f1ba321a41364376e490b1a3c82041935bf901e44"
May 25 12:30:17.903163 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:17.903145    2563 scope.go:117] "RemoveContainer" containerID="1c454aa126d0319e87751305f2559be62bfdef85704cfb52e049d2a9065d2154"
May 25 12:30:17.910046 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:17.910017    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm"]
May 25 12:30:17.910806 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:17.910686    2563 scope.go:117] "RemoveContainer" containerID="3afce9aac8ea3babe73169c1b4572ca166149e6b62f5886a7611c6c0d2eb2f55"
May 25 12:30:17.910969 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:30:17.910949    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3afce9aac8ea3babe73169c1b4572ca166149e6b62f5886a7611c6c0d2eb2f55\": container with ID starting with 3afce9aac8ea3babe73169c1b4572ca166149e6b62f5886a7611c6c0d2eb2f55 not found: ID does not exist" containerID="3afce9aac8ea3babe73169c1b4572ca166149e6b62f5886a7611c6c0d2eb2f55"
May 25 12:30:17.911033 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:17.910983    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3afce9aac8ea3babe73169c1b4572ca166149e6b62f5886a7611c6c0d2eb2f55"} err="failed to get container status \"3afce9aac8ea3babe73169c1b4572ca166149e6b62f5886a7611c6c0d2eb2f55\": rpc error: code = NotFound desc = could not find container \"3afce9aac8ea3babe73169c1b4572ca166149e6b62f5886a7611c6c0d2eb2f55\": container with ID starting with 3afce9aac8ea3babe73169c1b4572ca166149e6b62f5886a7611c6c0d2eb2f55 not found: ID does not exist"
May 25 12:30:17.911033 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:17.911009    2563 scope.go:117] "RemoveContainer" containerID="282d7076d3e212876058185f1ba321a41364376e490b1a3c82041935bf901e44"
May 25 12:30:17.911302 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:30:17.911279    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"282d7076d3e212876058185f1ba321a41364376e490b1a3c82041935bf901e44\": container with ID starting with 282d7076d3e212876058185f1ba321a41364376e490b1a3c82041935bf901e44 not found: ID does not exist" containerID="282d7076d3e212876058185f1ba321a41364376e490b1a3c82041935bf901e44"
May 25 12:30:17.911355 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:17.911307    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"282d7076d3e212876058185f1ba321a41364376e490b1a3c82041935bf901e44"} err="failed to get container status \"282d7076d3e212876058185f1ba321a41364376e490b1a3c82041935bf901e44\": rpc error: code = NotFound desc = could not find container \"282d7076d3e212876058185f1ba321a41364376e490b1a3c82041935bf901e44\": container with ID starting with 282d7076d3e212876058185f1ba321a41364376e490b1a3c82041935bf901e44 not found: ID does not exist"
May 25 12:30:17.911355 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:17.911323    2563 scope.go:117] "RemoveContainer" containerID="1c454aa126d0319e87751305f2559be62bfdef85704cfb52e049d2a9065d2154"
May 25 12:30:17.911594 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:30:17.911574    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c454aa126d0319e87751305f2559be62bfdef85704cfb52e049d2a9065d2154\": container with ID starting with 1c454aa126d0319e87751305f2559be62bfdef85704cfb52e049d2a9065d2154 not found: ID does not exist" containerID="1c454aa126d0319e87751305f2559be62bfdef85704cfb52e049d2a9065d2154"
May 25 12:30:17.911643 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:17.911602    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c454aa126d0319e87751305f2559be62bfdef85704cfb52e049d2a9065d2154"} err="failed to get container status \"1c454aa126d0319e87751305f2559be62bfdef85704cfb52e049d2a9065d2154\": rpc error: code = NotFound desc = could not find container \"1c454aa126d0319e87751305f2559be62bfdef85704cfb52e049d2a9065d2154\": container with ID starting with 1c454aa126d0319e87751305f2559be62bfdef85704cfb52e049d2a9065d2154 not found: ID does not exist"
May 25 12:30:17.914334 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:17.914309    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-10f269-predictor-5bbff9f956-2jzjm"]
May 25 12:30:18.636826 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:18.636798    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f14ed01-2147-4647-b02a-17683660e3f2" path="/var/lib/kubelet/pods/2f14ed01-2147-4647-b02a-17683660e3f2/volumes"
May 25 12:30:19.896284 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:19.896252    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6_e11c8aa2-1352-4ec6-9142-ba20823eb22d/storage-initializer/0.log"
May 25 12:30:19.896810 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:19.896299    2563 generic.go:358] "Generic (PLEG): container finished" podID="e11c8aa2-1352-4ec6-9142-ba20823eb22d" containerID="5a71bb8b45dfc1c7a1ab05519ec88d3f98653369408b0be7d2a683c8dc37a651" exitCode=1
May 25 12:30:19.896810 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:19.896362    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6" event={"ID":"e11c8aa2-1352-4ec6-9142-ba20823eb22d","Type":"ContainerDied","Data":"5a71bb8b45dfc1c7a1ab05519ec88d3f98653369408b0be7d2a683c8dc37a651"}
May 25 12:30:20.900188 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:20.900157    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6_e11c8aa2-1352-4ec6-9142-ba20823eb22d/storage-initializer/0.log"
May 25 12:30:20.900597 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:20.900239    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6" event={"ID":"e11c8aa2-1352-4ec6-9142-ba20823eb22d","Type":"ContainerStarted","Data":"b33f180f2728ceb048eedc3426d7f53be65d4ad42d055a772d0997a1d2cce502"}
May 25 12:30:22.907367 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:22.907342    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6_e11c8aa2-1352-4ec6-9142-ba20823eb22d/storage-initializer/1.log"
May 25 12:30:22.907830 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:22.907794    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6_e11c8aa2-1352-4ec6-9142-ba20823eb22d/storage-initializer/0.log"
May 25 12:30:22.907919 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:22.907836    2563 generic.go:358] "Generic (PLEG): container finished" podID="e11c8aa2-1352-4ec6-9142-ba20823eb22d" containerID="b33f180f2728ceb048eedc3426d7f53be65d4ad42d055a772d0997a1d2cce502" exitCode=1
May 25 12:30:22.907919 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:22.907876    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6" event={"ID":"e11c8aa2-1352-4ec6-9142-ba20823eb22d","Type":"ContainerDied","Data":"b33f180f2728ceb048eedc3426d7f53be65d4ad42d055a772d0997a1d2cce502"}
May 25 12:30:22.908022 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:22.907918    2563 scope.go:117] "RemoveContainer" containerID="5a71bb8b45dfc1c7a1ab05519ec88d3f98653369408b0be7d2a683c8dc37a651"
May 25 12:30:22.908360 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:22.908330    2563 kubelet_pods.go:1019] "Unable to retrieve pull secret, the image pull may not succeed." pod="kserve-ci-e2e-test/isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6" secret="" err="secret \"fail-s3-sa-7ee0ab-dockercfg-2qffx\" not found"
May 25 12:30:22.908464 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:22.908382    2563 scope.go:117] "RemoveContainer" containerID="5a71bb8b45dfc1c7a1ab05519ec88d3f98653369408b0be7d2a683c8dc37a651"
May 25 12:30:22.920492 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:30:22.920462    2563 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6_kserve-ci-e2e-test_e11c8aa2-1352-4ec6-9142-ba20823eb22d_0 in pod sandbox bb2c4460d7afbabed40d8de7227b4d71c4ddf057be4ce3f0845b37dbeabec9e7 from index: no such id: '5a71bb8b45dfc1c7a1ab05519ec88d3f98653369408b0be7d2a683c8dc37a651'" containerID="5a71bb8b45dfc1c7a1ab05519ec88d3f98653369408b0be7d2a683c8dc37a651"
May 25 12:30:22.920606 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:30:22.920535    2563 kuberuntime_container.go:951] "Unhandled Error" err="failed to remove pod init container \"storage-initializer\": rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6_kserve-ci-e2e-test_e11c8aa2-1352-4ec6-9142-ba20823eb22d_0 in pod sandbox bb2c4460d7afbabed40d8de7227b4d71c4ddf057be4ce3f0845b37dbeabec9e7 from index: no such id: '5a71bb8b45dfc1c7a1ab05519ec88d3f98653369408b0be7d2a683c8dc37a651'; Skipping pod \"isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6_kserve-ci-e2e-test(e11c8aa2-1352-4ec6-9142-ba20823eb22d)\"" logger="UnhandledError"
May 25 12:30:22.922028 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:30:22.921993    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6_kserve-ci-e2e-test(e11c8aa2-1352-4ec6-9142-ba20823eb22d)\"" pod="kserve-ci-e2e-test/isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6" podUID="e11c8aa2-1352-4ec6-9142-ba20823eb22d"
May 25 12:30:22.944742 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:22.944705    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6"]
May 25 12:30:23.057102 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:30:23.057070    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-7ee0ab-predictor-serving-cert: secret "isvc-init-fail-7ee0ab-predictor-serving-cert" not found
May 25 12:30:23.057274 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:30:23.057167    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e11c8aa2-1352-4ec6-9142-ba20823eb22d-proxy-tls podName:e11c8aa2-1352-4ec6-9142-ba20823eb22d nodeName:}" failed. No retries permitted until 2026-05-25 12:30:23.557142864 +0000 UTC m=+1089.524534484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e11c8aa2-1352-4ec6-9142-ba20823eb22d-proxy-tls") pod "isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6" (UID: "e11c8aa2-1352-4ec6-9142-ba20823eb22d") : secret "isvc-init-fail-7ee0ab-predictor-serving-cert" not found
May 25 12:30:23.067970 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:23.067917    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh"]
May 25 12:30:23.068223 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:23.068211    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a260d669-c5e7-4595-ab9b-fb5a2a180c8a" containerName="storage-initializer"
May 25 12:30:23.068266 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:23.068225    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a260d669-c5e7-4595-ab9b-fb5a2a180c8a" containerName="storage-initializer"
May 25 12:30:23.068266 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:23.068235    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f14ed01-2147-4647-b02a-17683660e3f2" containerName="storage-initializer"
May 25 12:30:23.068266 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:23.068241    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f14ed01-2147-4647-b02a-17683660e3f2" containerName="storage-initializer"
May 25 12:30:23.068266 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:23.068249    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a260d669-c5e7-4595-ab9b-fb5a2a180c8a" containerName="storage-initializer"
May 25 12:30:23.068266 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:23.068254    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a260d669-c5e7-4595-ab9b-fb5a2a180c8a" containerName="storage-initializer"
May 25 12:30:23.068266 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:23.068263    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f14ed01-2147-4647-b02a-17683660e3f2" containerName="kube-rbac-proxy"
May 25 12:30:23.068266 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:23.068268    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f14ed01-2147-4647-b02a-17683660e3f2" containerName="kube-rbac-proxy"
May 25 12:30:23.068466 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:23.068276    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f14ed01-2147-4647-b02a-17683660e3f2" containerName="kserve-container"
May 25 12:30:23.068466 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:23.068281    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f14ed01-2147-4647-b02a-17683660e3f2" containerName="kserve-container"
May 25 12:30:23.068466 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:23.068327    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="a260d669-c5e7-4595-ab9b-fb5a2a180c8a" containerName="storage-initializer"
May 25 12:30:23.068466 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:23.068334    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="2f14ed01-2147-4647-b02a-17683660e3f2" containerName="kube-rbac-proxy"
May 25 12:30:23.068466 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:23.068341    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="2f14ed01-2147-4647-b02a-17683660e3f2" containerName="kserve-container"
May 25 12:30:23.068466 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:23.068347    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="a260d669-c5e7-4595-ab9b-fb5a2a180c8a" containerName="storage-initializer"
May 25 12:30:23.072562 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:23.072546    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh"
May 25 12:30:23.074608 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:23.074588    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-79bgq\""
May 25 12:30:23.074608 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:23.074597    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-17a9a-predictor-serving-cert\""
May 25 12:30:23.074768 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:23.074703    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-17a9a-kube-rbac-proxy-sar-config\""
May 25 12:30:23.078979 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:23.078956    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh"]
May 25 12:30:23.258503 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:23.258467    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"raw-sklearn-17a9a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7778128c-2343-434f-8f8d-3e7bf42976fe-raw-sklearn-17a9a-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh\" (UID: \"7778128c-2343-434f-8f8d-3e7bf42976fe\") " pod="kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh"
May 25 12:30:23.258688 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:23.258508    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gj4m\" (UniqueName: \"kubernetes.io/projected/7778128c-2343-434f-8f8d-3e7bf42976fe-kube-api-access-6gj4m\") pod \"raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh\" (UID: \"7778128c-2343-434f-8f8d-3e7bf42976fe\") " pod="kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh"
May 25 12:30:23.258688 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:23.258655    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7778128c-2343-434f-8f8d-3e7bf42976fe-proxy-tls\") pod \"raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh\" (UID: \"7778128c-2343-434f-8f8d-3e7bf42976fe\") " pod="kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh"
May 25 12:30:23.258793 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:23.258699    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7778128c-2343-434f-8f8d-3e7bf42976fe-kserve-provision-location\") pod \"raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh\" (UID: \"7778128c-2343-434f-8f8d-3e7bf42976fe\") " pod="kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh"
May 25 12:30:23.359649 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:23.359604    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"raw-sklearn-17a9a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7778128c-2343-434f-8f8d-3e7bf42976fe-raw-sklearn-17a9a-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh\" (UID: \"7778128c-2343-434f-8f8d-3e7bf42976fe\") " pod="kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh"
May 25 12:30:23.359649 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:23.359646    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6gj4m\" (UniqueName: \"kubernetes.io/projected/7778128c-2343-434f-8f8d-3e7bf42976fe-kube-api-access-6gj4m\") pod \"raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh\" (UID: \"7778128c-2343-434f-8f8d-3e7bf42976fe\") " pod="kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh"
May 25 12:30:23.359899 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:23.359700    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7778128c-2343-434f-8f8d-3e7bf42976fe-proxy-tls\") pod \"raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh\" (UID: \"7778128c-2343-434f-8f8d-3e7bf42976fe\") " pod="kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh"
May 25 12:30:23.359899 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:23.359729    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7778128c-2343-434f-8f8d-3e7bf42976fe-kserve-provision-location\") pod \"raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh\" (UID: \"7778128c-2343-434f-8f8d-3e7bf42976fe\") " pod="kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh"
May 25 12:30:23.360094 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:23.360073    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7778128c-2343-434f-8f8d-3e7bf42976fe-kserve-provision-location\") pod \"raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh\" (UID: \"7778128c-2343-434f-8f8d-3e7bf42976fe\") " pod="kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh"
May 25 12:30:23.360317 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:23.360295    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"raw-sklearn-17a9a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7778128c-2343-434f-8f8d-3e7bf42976fe-raw-sklearn-17a9a-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh\" (UID: \"7778128c-2343-434f-8f8d-3e7bf42976fe\") " pod="kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh"
May 25 12:30:23.362149 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:23.362133    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7778128c-2343-434f-8f8d-3e7bf42976fe-proxy-tls\") pod \"raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh\" (UID: \"7778128c-2343-434f-8f8d-3e7bf42976fe\") " pod="kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh"
May 25 12:30:23.369182 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:23.369160    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gj4m\" (UniqueName: \"kubernetes.io/projected/7778128c-2343-434f-8f8d-3e7bf42976fe-kube-api-access-6gj4m\") pod \"raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh\" (UID: \"7778128c-2343-434f-8f8d-3e7bf42976fe\") " pod="kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh"
May 25 12:30:23.384981 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:23.384949    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh"
May 25 12:30:23.515755 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:23.515666    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh"]
May 25 12:30:23.518379 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:30:23.518344    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7778128c_2343_434f_8f8d_3e7bf42976fe.slice/crio-c904a5f25a65d46835b9b99dcdf382f2786bc7ef45aa177fad2abff4d79ff3ee WatchSource:0}: Error finding container c904a5f25a65d46835b9b99dcdf382f2786bc7ef45aa177fad2abff4d79ff3ee: Status 404 returned error can't find the container with id c904a5f25a65d46835b9b99dcdf382f2786bc7ef45aa177fad2abff4d79ff3ee
May 25 12:30:23.561307 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:30:23.561283    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-7ee0ab-predictor-serving-cert: secret "isvc-init-fail-7ee0ab-predictor-serving-cert" not found
May 25 12:30:23.561440 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:30:23.561347    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e11c8aa2-1352-4ec6-9142-ba20823eb22d-proxy-tls podName:e11c8aa2-1352-4ec6-9142-ba20823eb22d nodeName:}" failed. No retries permitted until 2026-05-25 12:30:24.561331904 +0000 UTC m=+1090.528723520 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e11c8aa2-1352-4ec6-9142-ba20823eb22d-proxy-tls") pod "isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6" (UID: "e11c8aa2-1352-4ec6-9142-ba20823eb22d") : secret "isvc-init-fail-7ee0ab-predictor-serving-cert" not found
May 25 12:30:23.912295 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:23.912199    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh" event={"ID":"7778128c-2343-434f-8f8d-3e7bf42976fe","Type":"ContainerStarted","Data":"5819030b37a58bac71dc9c7b7d3a969581e021ff232aac31fec1e2df57324417"}
May 25 12:30:23.912295 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:23.912251    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh" event={"ID":"7778128c-2343-434f-8f8d-3e7bf42976fe","Type":"ContainerStarted","Data":"c904a5f25a65d46835b9b99dcdf382f2786bc7ef45aa177fad2abff4d79ff3ee"}
May 25 12:30:23.913577 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:23.913558    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6_e11c8aa2-1352-4ec6-9142-ba20823eb22d/storage-initializer/1.log"
May 25 12:30:24.051025 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:24.051002    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6_e11c8aa2-1352-4ec6-9142-ba20823eb22d/storage-initializer/1.log"
May 25 12:30:24.051155 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:24.051064    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6"
May 25 12:30:24.166845 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:24.166754    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-init-fail-7ee0ab-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e11c8aa2-1352-4ec6-9142-ba20823eb22d-isvc-init-fail-7ee0ab-kube-rbac-proxy-sar-config\") pod \"e11c8aa2-1352-4ec6-9142-ba20823eb22d\" (UID: \"e11c8aa2-1352-4ec6-9142-ba20823eb22d\") "
May 25 12:30:24.166845 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:24.166804    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2db4\" (UniqueName: \"kubernetes.io/projected/e11c8aa2-1352-4ec6-9142-ba20823eb22d-kube-api-access-f2db4\") pod \"e11c8aa2-1352-4ec6-9142-ba20823eb22d\" (UID: \"e11c8aa2-1352-4ec6-9142-ba20823eb22d\") "
May 25 12:30:24.166845 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:24.166839    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e11c8aa2-1352-4ec6-9142-ba20823eb22d-proxy-tls\") pod \"e11c8aa2-1352-4ec6-9142-ba20823eb22d\" (UID: \"e11c8aa2-1352-4ec6-9142-ba20823eb22d\") "
May 25 12:30:24.167108 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:24.166863    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e11c8aa2-1352-4ec6-9142-ba20823eb22d-cabundle-cert\") pod \"e11c8aa2-1352-4ec6-9142-ba20823eb22d\" (UID: \"e11c8aa2-1352-4ec6-9142-ba20823eb22d\") "
May 25 12:30:24.167108 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:24.166898    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e11c8aa2-1352-4ec6-9142-ba20823eb22d-kserve-provision-location\") pod \"e11c8aa2-1352-4ec6-9142-ba20823eb22d\" (UID: \"e11c8aa2-1352-4ec6-9142-ba20823eb22d\") "
May 25 12:30:24.167255 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:24.167222    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e11c8aa2-1352-4ec6-9142-ba20823eb22d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e11c8aa2-1352-4ec6-9142-ba20823eb22d" (UID: "e11c8aa2-1352-4ec6-9142-ba20823eb22d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:30:24.167387 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:24.167273    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e11c8aa2-1352-4ec6-9142-ba20823eb22d-isvc-init-fail-7ee0ab-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-init-fail-7ee0ab-kube-rbac-proxy-sar-config") pod "e11c8aa2-1352-4ec6-9142-ba20823eb22d" (UID: "e11c8aa2-1352-4ec6-9142-ba20823eb22d"). InnerVolumeSpecName "isvc-init-fail-7ee0ab-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:30:24.167387 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:24.167313    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e11c8aa2-1352-4ec6-9142-ba20823eb22d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e11c8aa2-1352-4ec6-9142-ba20823eb22d" (UID: "e11c8aa2-1352-4ec6-9142-ba20823eb22d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:30:24.169081 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:24.169056    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e11c8aa2-1352-4ec6-9142-ba20823eb22d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e11c8aa2-1352-4ec6-9142-ba20823eb22d" (UID: "e11c8aa2-1352-4ec6-9142-ba20823eb22d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:30:24.169189 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:24.169155    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e11c8aa2-1352-4ec6-9142-ba20823eb22d-kube-api-access-f2db4" (OuterVolumeSpecName: "kube-api-access-f2db4") pod "e11c8aa2-1352-4ec6-9142-ba20823eb22d" (UID: "e11c8aa2-1352-4ec6-9142-ba20823eb22d"). InnerVolumeSpecName "kube-api-access-f2db4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:30:24.268436 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:24.268393    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-init-fail-7ee0ab-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e11c8aa2-1352-4ec6-9142-ba20823eb22d-isvc-init-fail-7ee0ab-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:30:24.268436 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:24.268428    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2db4\" (UniqueName: \"kubernetes.io/projected/e11c8aa2-1352-4ec6-9142-ba20823eb22d-kube-api-access-f2db4\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:30:24.268436 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:24.268439    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e11c8aa2-1352-4ec6-9142-ba20823eb22d-proxy-tls\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:30:24.268436 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:24.268449    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e11c8aa2-1352-4ec6-9142-ba20823eb22d-cabundle-cert\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:30:24.268717 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:24.268457    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e11c8aa2-1352-4ec6-9142-ba20823eb22d-kserve-provision-location\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:30:24.918248 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:24.918163    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6_e11c8aa2-1352-4ec6-9142-ba20823eb22d/storage-initializer/1.log"
May 25 12:30:24.918661 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:24.918238    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6" event={"ID":"e11c8aa2-1352-4ec6-9142-ba20823eb22d","Type":"ContainerDied","Data":"bb2c4460d7afbabed40d8de7227b4d71c4ddf057be4ce3f0845b37dbeabec9e7"}
May 25 12:30:24.918661 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:24.918286    2563 scope.go:117] "RemoveContainer" containerID="b33f180f2728ceb048eedc3426d7f53be65d4ad42d055a772d0997a1d2cce502"
May 25 12:30:24.918661 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:24.918477    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6"
May 25 12:30:24.951127 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:24.951091    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6"]
May 25 12:30:24.957773 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:24.957745    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-7ee0ab-predictor-5b88f9f499-b8js6"]
May 25 12:30:26.637368 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:26.637331    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e11c8aa2-1352-4ec6-9142-ba20823eb22d" path="/var/lib/kubelet/pods/e11c8aa2-1352-4ec6-9142-ba20823eb22d/volumes"
May 25 12:30:27.931203 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:27.931172    2563 generic.go:358] "Generic (PLEG): container finished" podID="7778128c-2343-434f-8f8d-3e7bf42976fe" containerID="5819030b37a58bac71dc9c7b7d3a969581e021ff232aac31fec1e2df57324417" exitCode=0
May 25 12:30:27.931609 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:27.931220    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh" event={"ID":"7778128c-2343-434f-8f8d-3e7bf42976fe","Type":"ContainerDied","Data":"5819030b37a58bac71dc9c7b7d3a969581e021ff232aac31fec1e2df57324417"}
May 25 12:30:28.935804 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:28.935765    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh" event={"ID":"7778128c-2343-434f-8f8d-3e7bf42976fe","Type":"ContainerStarted","Data":"ca1555c3dfe0701442dd8d791849a11a45aee52442b24ef4c39d8de5b1073c1d"}
May 25 12:30:28.935804 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:28.935800    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh" event={"ID":"7778128c-2343-434f-8f8d-3e7bf42976fe","Type":"ContainerStarted","Data":"b6eb5d7880115766e568d705090b520d8161acde4a207648b787ba981b95ffed"}
May 25 12:30:28.936225 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:28.936075    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh"
May 25 12:30:28.956084 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:28.956032    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh" podStartSLOduration=5.956017783 podStartE2EDuration="5.956017783s" podCreationTimestamp="2026-05-25 12:30:23 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:30:28.954763095 +0000 UTC m=+1094.922154734" watchObservedRunningTime="2026-05-25 12:30:28.956017783 +0000 UTC m=+1094.923409422"
May 25 12:30:29.938445 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:29.938407    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh"
May 25 12:30:29.939672 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:29.939639    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh" podUID="7778128c-2343-434f-8f8d-3e7bf42976fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
May 25 12:30:30.940667 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:30.940624    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh" podUID="7778128c-2343-434f-8f8d-3e7bf42976fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
May 25 12:30:35.944999 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:35.944963    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh"
May 25 12:30:35.945434 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:35.945409    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh" podUID="7778128c-2343-434f-8f8d-3e7bf42976fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
May 25 12:30:45.946310 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:45.946270    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh" podUID="7778128c-2343-434f-8f8d-3e7bf42976fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
May 25 12:30:55.945934 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:30:55.945892    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh" podUID="7778128c-2343-434f-8f8d-3e7bf42976fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
May 25 12:31:05.945380 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:05.945340    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh" podUID="7778128c-2343-434f-8f8d-3e7bf42976fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
May 25 12:31:15.946260 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:15.946212    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh" podUID="7778128c-2343-434f-8f8d-3e7bf42976fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
May 25 12:31:25.946154 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:25.946108    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh" podUID="7778128c-2343-434f-8f8d-3e7bf42976fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
May 25 12:31:35.946259 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:35.946173    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh"
May 25 12:31:43.165359 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:43.165326    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh"]
May 25 12:31:43.165786 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:43.165639    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh" podUID="7778128c-2343-434f-8f8d-3e7bf42976fe" containerName="kserve-container" containerID="cri-o://b6eb5d7880115766e568d705090b520d8161acde4a207648b787ba981b95ffed" gracePeriod=30
May 25 12:31:43.165786 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:43.165684    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh" podUID="7778128c-2343-434f-8f8d-3e7bf42976fe" containerName="kube-rbac-proxy" containerID="cri-o://ca1555c3dfe0701442dd8d791849a11a45aee52442b24ef4c39d8de5b1073c1d" gracePeriod=30
May 25 12:31:43.246653 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:43.246619    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8"]
May 25 12:31:43.246909 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:43.246898    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e11c8aa2-1352-4ec6-9142-ba20823eb22d" containerName="storage-initializer"
May 25 12:31:43.246953 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:43.246911    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e11c8aa2-1352-4ec6-9142-ba20823eb22d" containerName="storage-initializer"
May 25 12:31:43.246987 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:43.246969    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="e11c8aa2-1352-4ec6-9142-ba20823eb22d" containerName="storage-initializer"
May 25 12:31:43.247028 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:43.247019    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e11c8aa2-1352-4ec6-9142-ba20823eb22d" containerName="storage-initializer"
May 25 12:31:43.247062 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:43.247027    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e11c8aa2-1352-4ec6-9142-ba20823eb22d" containerName="storage-initializer"
May 25 12:31:43.247094 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:43.247065    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="e11c8aa2-1352-4ec6-9142-ba20823eb22d" containerName="storage-initializer"
May 25 12:31:43.250159 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:43.250137    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8"
May 25 12:31:43.251937 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:43.251915    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-dfccf-kube-rbac-proxy-sar-config\""
May 25 12:31:43.252059 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:43.251977    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-dfccf-predictor-serving-cert\""
May 25 12:31:43.260837 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:43.260815    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8"]
May 25 12:31:43.280795 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:43.280765    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e607074-6986-474f-bda8-082c65fff8a2-proxy-tls\") pod \"raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8\" (UID: \"4e607074-6986-474f-bda8-082c65fff8a2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8"
May 25 12:31:43.280960 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:43.280812    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e607074-6986-474f-bda8-082c65fff8a2-kserve-provision-location\") pod \"raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8\" (UID: \"4e607074-6986-474f-bda8-082c65fff8a2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8"
May 25 12:31:43.280960 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:43.280890    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqx7j\" (UniqueName: \"kubernetes.io/projected/4e607074-6986-474f-bda8-082c65fff8a2-kube-api-access-kqx7j\") pod \"raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8\" (UID: \"4e607074-6986-474f-bda8-082c65fff8a2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8"
May 25 12:31:43.281047 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:43.280961    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"raw-sklearn-runtime-dfccf-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e607074-6986-474f-bda8-082c65fff8a2-raw-sklearn-runtime-dfccf-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8\" (UID: \"4e607074-6986-474f-bda8-082c65fff8a2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8"
May 25 12:31:43.381455 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:43.381406    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"raw-sklearn-runtime-dfccf-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e607074-6986-474f-bda8-082c65fff8a2-raw-sklearn-runtime-dfccf-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8\" (UID: \"4e607074-6986-474f-bda8-082c65fff8a2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8"
May 25 12:31:43.381679 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:43.381475    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e607074-6986-474f-bda8-082c65fff8a2-proxy-tls\") pod \"raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8\" (UID: \"4e607074-6986-474f-bda8-082c65fff8a2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8"
May 25 12:31:43.381679 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:31:43.381615    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-serving-cert: secret "raw-sklearn-runtime-dfccf-predictor-serving-cert" not found
May 25 12:31:43.381679 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:43.381636    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e607074-6986-474f-bda8-082c65fff8a2-kserve-provision-location\") pod \"raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8\" (UID: \"4e607074-6986-474f-bda8-082c65fff8a2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8"
May 25 12:31:43.381840 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:31:43.381682    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e607074-6986-474f-bda8-082c65fff8a2-proxy-tls podName:4e607074-6986-474f-bda8-082c65fff8a2 nodeName:}" failed. No retries permitted until 2026-05-25 12:31:43.881659298 +0000 UTC m=+1169.849050916 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4e607074-6986-474f-bda8-082c65fff8a2-proxy-tls") pod "raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8" (UID: "4e607074-6986-474f-bda8-082c65fff8a2") : secret "raw-sklearn-runtime-dfccf-predictor-serving-cert" not found
May 25 12:31:43.381840 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:43.381740    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kqx7j\" (UniqueName: \"kubernetes.io/projected/4e607074-6986-474f-bda8-082c65fff8a2-kube-api-access-kqx7j\") pod \"raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8\" (UID: \"4e607074-6986-474f-bda8-082c65fff8a2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8"
May 25 12:31:43.382040 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:43.382017    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e607074-6986-474f-bda8-082c65fff8a2-kserve-provision-location\") pod \"raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8\" (UID: \"4e607074-6986-474f-bda8-082c65fff8a2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8"
May 25 12:31:43.382210 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:43.382193    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"raw-sklearn-runtime-dfccf-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e607074-6986-474f-bda8-082c65fff8a2-raw-sklearn-runtime-dfccf-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8\" (UID: \"4e607074-6986-474f-bda8-082c65fff8a2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8"
May 25 12:31:43.389824 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:43.389793    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqx7j\" (UniqueName: \"kubernetes.io/projected/4e607074-6986-474f-bda8-082c65fff8a2-kube-api-access-kqx7j\") pod \"raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8\" (UID: \"4e607074-6986-474f-bda8-082c65fff8a2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8"
May 25 12:31:43.886100 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:43.886054    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e607074-6986-474f-bda8-082c65fff8a2-proxy-tls\") pod \"raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8\" (UID: \"4e607074-6986-474f-bda8-082c65fff8a2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8"
May 25 12:31:43.886286 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:31:43.886188    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-serving-cert: secret "raw-sklearn-runtime-dfccf-predictor-serving-cert" not found
May 25 12:31:43.886286 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:31:43.886255    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e607074-6986-474f-bda8-082c65fff8a2-proxy-tls podName:4e607074-6986-474f-bda8-082c65fff8a2 nodeName:}" failed. No retries permitted until 2026-05-25 12:31:44.886239358 +0000 UTC m=+1170.853630978 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4e607074-6986-474f-bda8-082c65fff8a2-proxy-tls") pod "raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8" (UID: "4e607074-6986-474f-bda8-082c65fff8a2") : secret "raw-sklearn-runtime-dfccf-predictor-serving-cert" not found
May 25 12:31:44.149284 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:44.149187    2563 generic.go:358] "Generic (PLEG): container finished" podID="7778128c-2343-434f-8f8d-3e7bf42976fe" containerID="ca1555c3dfe0701442dd8d791849a11a45aee52442b24ef4c39d8de5b1073c1d" exitCode=2
May 25 12:31:44.149284 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:44.149249    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh" event={"ID":"7778128c-2343-434f-8f8d-3e7bf42976fe","Type":"ContainerDied","Data":"ca1555c3dfe0701442dd8d791849a11a45aee52442b24ef4c39d8de5b1073c1d"}
May 25 12:31:44.892759 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:44.892724    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e607074-6986-474f-bda8-082c65fff8a2-proxy-tls\") pod \"raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8\" (UID: \"4e607074-6986-474f-bda8-082c65fff8a2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8"
May 25 12:31:44.895149 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:44.895128    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e607074-6986-474f-bda8-082c65fff8a2-proxy-tls\") pod \"raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8\" (UID: \"4e607074-6986-474f-bda8-082c65fff8a2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8"
May 25 12:31:45.062566 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:45.062486    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8"
May 25 12:31:45.192062 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:45.192034    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8"]
May 25 12:31:45.194202 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:31:45.194175    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e607074_6986_474f_bda8_082c65fff8a2.slice/crio-72f6ff5913603768bb7f48902c354c6fb80da908217a9c6b828aaa3598dca6a5 WatchSource:0}: Error finding container 72f6ff5913603768bb7f48902c354c6fb80da908217a9c6b828aaa3598dca6a5: Status 404 returned error can't find the container with id 72f6ff5913603768bb7f48902c354c6fb80da908217a9c6b828aaa3598dca6a5
May 25 12:31:45.940948 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:45.940905    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh" podUID="7778128c-2343-434f-8f8d-3e7bf42976fe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
May 25 12:31:45.945591 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:45.945560    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh" podUID="7778128c-2343-434f-8f8d-3e7bf42976fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
May 25 12:31:46.158803 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:46.158765    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8" event={"ID":"4e607074-6986-474f-bda8-082c65fff8a2","Type":"ContainerStarted","Data":"1f336be4d39b2314a3ee6cbadf20d09c3b8fa97ec92aa83800bd553b59c176fe"}
May 25 12:31:46.158803 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:46.158803    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8" event={"ID":"4e607074-6986-474f-bda8-082c65fff8a2","Type":"ContainerStarted","Data":"72f6ff5913603768bb7f48902c354c6fb80da908217a9c6b828aaa3598dca6a5"}
May 25 12:31:47.710176 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:47.710152    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh"
May 25 12:31:47.811884 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:47.811792    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"raw-sklearn-17a9a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7778128c-2343-434f-8f8d-3e7bf42976fe-raw-sklearn-17a9a-kube-rbac-proxy-sar-config\") pod \"7778128c-2343-434f-8f8d-3e7bf42976fe\" (UID: \"7778128c-2343-434f-8f8d-3e7bf42976fe\") "
May 25 12:31:47.811884 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:47.811846    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gj4m\" (UniqueName: \"kubernetes.io/projected/7778128c-2343-434f-8f8d-3e7bf42976fe-kube-api-access-6gj4m\") pod \"7778128c-2343-434f-8f8d-3e7bf42976fe\" (UID: \"7778128c-2343-434f-8f8d-3e7bf42976fe\") "
May 25 12:31:47.811884 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:47.811885    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7778128c-2343-434f-8f8d-3e7bf42976fe-kserve-provision-location\") pod \"7778128c-2343-434f-8f8d-3e7bf42976fe\" (UID: \"7778128c-2343-434f-8f8d-3e7bf42976fe\") "
May 25 12:31:47.812157 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:47.811914    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7778128c-2343-434f-8f8d-3e7bf42976fe-proxy-tls\") pod \"7778128c-2343-434f-8f8d-3e7bf42976fe\" (UID: \"7778128c-2343-434f-8f8d-3e7bf42976fe\") "
May 25 12:31:47.812218 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:47.812174    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7778128c-2343-434f-8f8d-3e7bf42976fe-raw-sklearn-17a9a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "raw-sklearn-17a9a-kube-rbac-proxy-sar-config") pod "7778128c-2343-434f-8f8d-3e7bf42976fe" (UID: "7778128c-2343-434f-8f8d-3e7bf42976fe"). InnerVolumeSpecName "raw-sklearn-17a9a-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:31:47.812280 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:47.812207    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7778128c-2343-434f-8f8d-3e7bf42976fe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7778128c-2343-434f-8f8d-3e7bf42976fe" (UID: "7778128c-2343-434f-8f8d-3e7bf42976fe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:31:47.813949 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:47.813925    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7778128c-2343-434f-8f8d-3e7bf42976fe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7778128c-2343-434f-8f8d-3e7bf42976fe" (UID: "7778128c-2343-434f-8f8d-3e7bf42976fe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:31:47.814029 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:47.814009    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7778128c-2343-434f-8f8d-3e7bf42976fe-kube-api-access-6gj4m" (OuterVolumeSpecName: "kube-api-access-6gj4m") pod "7778128c-2343-434f-8f8d-3e7bf42976fe" (UID: "7778128c-2343-434f-8f8d-3e7bf42976fe"). InnerVolumeSpecName "kube-api-access-6gj4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:31:47.912453 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:47.912413    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7778128c-2343-434f-8f8d-3e7bf42976fe-kserve-provision-location\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:31:47.912453 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:47.912445    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7778128c-2343-434f-8f8d-3e7bf42976fe-proxy-tls\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:31:47.912453 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:47.912459    2563 reconciler_common.go:299] "Volume detached for volume \"raw-sklearn-17a9a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7778128c-2343-434f-8f8d-3e7bf42976fe-raw-sklearn-17a9a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:31:47.912727 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:47.912469    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gj4m\" (UniqueName: \"kubernetes.io/projected/7778128c-2343-434f-8f8d-3e7bf42976fe-kube-api-access-6gj4m\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:31:48.165462 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:48.165367    2563 generic.go:358] "Generic (PLEG): container finished" podID="7778128c-2343-434f-8f8d-3e7bf42976fe" containerID="b6eb5d7880115766e568d705090b520d8161acde4a207648b787ba981b95ffed" exitCode=0
May 25 12:31:48.165462 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:48.165450    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh"
May 25 12:31:48.165681 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:48.165458    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh" event={"ID":"7778128c-2343-434f-8f8d-3e7bf42976fe","Type":"ContainerDied","Data":"b6eb5d7880115766e568d705090b520d8161acde4a207648b787ba981b95ffed"}
May 25 12:31:48.165681 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:48.165501    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh" event={"ID":"7778128c-2343-434f-8f8d-3e7bf42976fe","Type":"ContainerDied","Data":"c904a5f25a65d46835b9b99dcdf382f2786bc7ef45aa177fad2abff4d79ff3ee"}
May 25 12:31:48.165681 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:48.165535    2563 scope.go:117] "RemoveContainer" containerID="ca1555c3dfe0701442dd8d791849a11a45aee52442b24ef4c39d8de5b1073c1d"
May 25 12:31:48.173488 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:48.173470    2563 scope.go:117] "RemoveContainer" containerID="b6eb5d7880115766e568d705090b520d8161acde4a207648b787ba981b95ffed"
May 25 12:31:48.180412 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:48.180392    2563 scope.go:117] "RemoveContainer" containerID="5819030b37a58bac71dc9c7b7d3a969581e021ff232aac31fec1e2df57324417"
May 25 12:31:48.187875 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:48.187847    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh"]
May 25 12:31:48.188279 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:48.188261    2563 scope.go:117] "RemoveContainer" containerID="ca1555c3dfe0701442dd8d791849a11a45aee52442b24ef4c39d8de5b1073c1d"
May 25 12:31:48.188548 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:31:48.188512    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca1555c3dfe0701442dd8d791849a11a45aee52442b24ef4c39d8de5b1073c1d\": container with ID starting with ca1555c3dfe0701442dd8d791849a11a45aee52442b24ef4c39d8de5b1073c1d not found: ID does not exist" containerID="ca1555c3dfe0701442dd8d791849a11a45aee52442b24ef4c39d8de5b1073c1d"
May 25 12:31:48.188639 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:48.188555    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca1555c3dfe0701442dd8d791849a11a45aee52442b24ef4c39d8de5b1073c1d"} err="failed to get container status \"ca1555c3dfe0701442dd8d791849a11a45aee52442b24ef4c39d8de5b1073c1d\": rpc error: code = NotFound desc = could not find container \"ca1555c3dfe0701442dd8d791849a11a45aee52442b24ef4c39d8de5b1073c1d\": container with ID starting with ca1555c3dfe0701442dd8d791849a11a45aee52442b24ef4c39d8de5b1073c1d not found: ID does not exist"
May 25 12:31:48.188639 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:48.188574    2563 scope.go:117] "RemoveContainer" containerID="b6eb5d7880115766e568d705090b520d8161acde4a207648b787ba981b95ffed"
May 25 12:31:48.188816 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:31:48.188798    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6eb5d7880115766e568d705090b520d8161acde4a207648b787ba981b95ffed\": container with ID starting with b6eb5d7880115766e568d705090b520d8161acde4a207648b787ba981b95ffed not found: ID does not exist" containerID="b6eb5d7880115766e568d705090b520d8161acde4a207648b787ba981b95ffed"
May 25 12:31:48.188885 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:48.188825    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6eb5d7880115766e568d705090b520d8161acde4a207648b787ba981b95ffed"} err="failed to get container status \"b6eb5d7880115766e568d705090b520d8161acde4a207648b787ba981b95ffed\": rpc error: code = NotFound desc = could not find container \"b6eb5d7880115766e568d705090b520d8161acde4a207648b787ba981b95ffed\": container with ID starting with b6eb5d7880115766e568d705090b520d8161acde4a207648b787ba981b95ffed not found: ID does not exist"
May 25 12:31:48.188885 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:48.188849    2563 scope.go:117] "RemoveContainer" containerID="5819030b37a58bac71dc9c7b7d3a969581e021ff232aac31fec1e2df57324417"
May 25 12:31:48.189065 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:31:48.189049    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5819030b37a58bac71dc9c7b7d3a969581e021ff232aac31fec1e2df57324417\": container with ID starting with 5819030b37a58bac71dc9c7b7d3a969581e021ff232aac31fec1e2df57324417 not found: ID does not exist" containerID="5819030b37a58bac71dc9c7b7d3a969581e021ff232aac31fec1e2df57324417"
May 25 12:31:48.189110 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:48.189070    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5819030b37a58bac71dc9c7b7d3a969581e021ff232aac31fec1e2df57324417"} err="failed to get container status \"5819030b37a58bac71dc9c7b7d3a969581e021ff232aac31fec1e2df57324417\": rpc error: code = NotFound desc = could not find container \"5819030b37a58bac71dc9c7b7d3a969581e021ff232aac31fec1e2df57324417\": container with ID starting with 5819030b37a58bac71dc9c7b7d3a969581e021ff232aac31fec1e2df57324417 not found: ID does not exist"
May 25 12:31:48.192507 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:48.192486    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-17a9a-predictor-66b6f69ff7-gtwnh"]
May 25 12:31:48.636479 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:48.636443    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7778128c-2343-434f-8f8d-3e7bf42976fe" path="/var/lib/kubelet/pods/7778128c-2343-434f-8f8d-3e7bf42976fe/volumes"
May 25 12:31:49.170468 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:49.170431    2563 generic.go:358] "Generic (PLEG): container finished" podID="4e607074-6986-474f-bda8-082c65fff8a2" containerID="1f336be4d39b2314a3ee6cbadf20d09c3b8fa97ec92aa83800bd553b59c176fe" exitCode=0
May 25 12:31:49.170977 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:49.170511    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8" event={"ID":"4e607074-6986-474f-bda8-082c65fff8a2","Type":"ContainerDied","Data":"1f336be4d39b2314a3ee6cbadf20d09c3b8fa97ec92aa83800bd553b59c176fe"}
May 25 12:31:50.175372 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:50.175341    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8" event={"ID":"4e607074-6986-474f-bda8-082c65fff8a2","Type":"ContainerStarted","Data":"944879ca5236717c3e9cd49b40d10527ccd7f9c81d48da10c08dabebd8350955"}
May 25 12:31:50.175372 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:50.175374    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8" event={"ID":"4e607074-6986-474f-bda8-082c65fff8a2","Type":"ContainerStarted","Data":"69c7e566e625c91bfaf3fd497e2f7389ad39fd42997a36e7faa24916fa03711e"}
May 25 12:31:50.175822 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:50.175573    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8"
May 25 12:31:50.193080 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:50.193032    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8" podStartSLOduration=7.193015544 podStartE2EDuration="7.193015544s" podCreationTimestamp="2026-05-25 12:31:43 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:31:50.191762025 +0000 UTC m=+1176.159153664" watchObservedRunningTime="2026-05-25 12:31:50.193015544 +0000 UTC m=+1176.160407183"
May 25 12:31:51.178874 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:51.178838    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8"
May 25 12:31:51.180236 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:51.180211    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8" podUID="4e607074-6986-474f-bda8-082c65fff8a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
May 25 12:31:52.181080 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:52.181035    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8" podUID="4e607074-6986-474f-bda8-082c65fff8a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
May 25 12:31:57.185265 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:57.185237    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8"
May 25 12:31:57.185775 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:31:57.185714    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8" podUID="4e607074-6986-474f-bda8-082c65fff8a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
May 25 12:32:07.186613 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:32:07.186569    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8" podUID="4e607074-6986-474f-bda8-082c65fff8a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
May 25 12:32:14.581915 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:32:14.581884    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw9bb_012c465e-2683-488a-abc1-18ad86529eb3/ovn-acl-logging/0.log"
May 25 12:32:14.586910 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:32:14.586889    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw9bb_012c465e-2683-488a-abc1-18ad86529eb3/ovn-acl-logging/0.log"
May 25 12:32:17.186339 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:32:17.186301    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8" podUID="4e607074-6986-474f-bda8-082c65fff8a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
May 25 12:32:27.186043 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:32:27.186003    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8" podUID="4e607074-6986-474f-bda8-082c65fff8a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
May 25 12:32:37.186492 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:32:37.186450    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8" podUID="4e607074-6986-474f-bda8-082c65fff8a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
May 25 12:32:47.186394 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:32:47.186352    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8" podUID="4e607074-6986-474f-bda8-082c65fff8a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
May 25 12:32:57.186591 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:32:57.186557    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8"
May 25 12:33:03.356601 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:03.356567    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8"]
May 25 12:33:03.357009 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:03.356866    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8" podUID="4e607074-6986-474f-bda8-082c65fff8a2" containerName="kserve-container" containerID="cri-o://69c7e566e625c91bfaf3fd497e2f7389ad39fd42997a36e7faa24916fa03711e" gracePeriod=30
May 25 12:33:03.357009 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:03.356904    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8" podUID="4e607074-6986-474f-bda8-082c65fff8a2" containerName="kube-rbac-proxy" containerID="cri-o://944879ca5236717c3e9cd49b40d10527ccd7f9c81d48da10c08dabebd8350955" gracePeriod=30
May 25 12:33:04.384875 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:04.384797    2563 generic.go:358] "Generic (PLEG): container finished" podID="4e607074-6986-474f-bda8-082c65fff8a2" containerID="944879ca5236717c3e9cd49b40d10527ccd7f9c81d48da10c08dabebd8350955" exitCode=2
May 25 12:33:04.385231 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:04.384870    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8" event={"ID":"4e607074-6986-474f-bda8-082c65fff8a2","Type":"ContainerDied","Data":"944879ca5236717c3e9cd49b40d10527ccd7f9c81d48da10c08dabebd8350955"}
May 25 12:33:07.181416 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:07.181370    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8" podUID="4e607074-6986-474f-bda8-082c65fff8a2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
May 25 12:33:07.185701 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:07.185672    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8" podUID="4e607074-6986-474f-bda8-082c65fff8a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
May 25 12:33:07.899442 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:07.899415    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8"
May 25 12:33:07.950189 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:07.950149    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqx7j\" (UniqueName: \"kubernetes.io/projected/4e607074-6986-474f-bda8-082c65fff8a2-kube-api-access-kqx7j\") pod \"4e607074-6986-474f-bda8-082c65fff8a2\" (UID: \"4e607074-6986-474f-bda8-082c65fff8a2\") "
May 25 12:33:07.950189 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:07.950192    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e607074-6986-474f-bda8-082c65fff8a2-proxy-tls\") pod \"4e607074-6986-474f-bda8-082c65fff8a2\" (UID: \"4e607074-6986-474f-bda8-082c65fff8a2\") "
May 25 12:33:07.950428 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:07.950227    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"raw-sklearn-runtime-dfccf-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e607074-6986-474f-bda8-082c65fff8a2-raw-sklearn-runtime-dfccf-kube-rbac-proxy-sar-config\") pod \"4e607074-6986-474f-bda8-082c65fff8a2\" (UID: \"4e607074-6986-474f-bda8-082c65fff8a2\") "
May 25 12:33:07.950428 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:07.950279    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e607074-6986-474f-bda8-082c65fff8a2-kserve-provision-location\") pod \"4e607074-6986-474f-bda8-082c65fff8a2\" (UID: \"4e607074-6986-474f-bda8-082c65fff8a2\") "
May 25 12:33:07.950653 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:07.950624    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e607074-6986-474f-bda8-082c65fff8a2-raw-sklearn-runtime-dfccf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "raw-sklearn-runtime-dfccf-kube-rbac-proxy-sar-config") pod "4e607074-6986-474f-bda8-082c65fff8a2" (UID: "4e607074-6986-474f-bda8-082c65fff8a2"). InnerVolumeSpecName "raw-sklearn-runtime-dfccf-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:33:07.950653 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:07.950637    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e607074-6986-474f-bda8-082c65fff8a2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4e607074-6986-474f-bda8-082c65fff8a2" (UID: "4e607074-6986-474f-bda8-082c65fff8a2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:33:07.952335 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:07.952302    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e607074-6986-474f-bda8-082c65fff8a2-kube-api-access-kqx7j" (OuterVolumeSpecName: "kube-api-access-kqx7j") pod "4e607074-6986-474f-bda8-082c65fff8a2" (UID: "4e607074-6986-474f-bda8-082c65fff8a2"). InnerVolumeSpecName "kube-api-access-kqx7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:33:07.952395 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:07.952380    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e607074-6986-474f-bda8-082c65fff8a2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4e607074-6986-474f-bda8-082c65fff8a2" (UID: "4e607074-6986-474f-bda8-082c65fff8a2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:33:08.051847 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:08.051807    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqx7j\" (UniqueName: \"kubernetes.io/projected/4e607074-6986-474f-bda8-082c65fff8a2-kube-api-access-kqx7j\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:33:08.051847 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:08.051839    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e607074-6986-474f-bda8-082c65fff8a2-proxy-tls\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:33:08.051847 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:08.051850    2563 reconciler_common.go:299] "Volume detached for volume \"raw-sklearn-runtime-dfccf-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e607074-6986-474f-bda8-082c65fff8a2-raw-sklearn-runtime-dfccf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:33:08.051847 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:08.051860    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e607074-6986-474f-bda8-082c65fff8a2-kserve-provision-location\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
May 25 12:33:08.397338 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:08.397247    2563 generic.go:358] "Generic (PLEG): container finished" podID="4e607074-6986-474f-bda8-082c65fff8a2" containerID="69c7e566e625c91bfaf3fd497e2f7389ad39fd42997a36e7faa24916fa03711e" exitCode=0
May 25 12:33:08.397338 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:08.397320    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8" event={"ID":"4e607074-6986-474f-bda8-082c65fff8a2","Type":"ContainerDied","Data":"69c7e566e625c91bfaf3fd497e2f7389ad39fd42997a36e7faa24916fa03711e"}
May 25 12:33:08.397338 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:08.397330    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8"
May 25 12:33:08.397835 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:08.397348    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8" event={"ID":"4e607074-6986-474f-bda8-082c65fff8a2","Type":"ContainerDied","Data":"72f6ff5913603768bb7f48902c354c6fb80da908217a9c6b828aaa3598dca6a5"}
May 25 12:33:08.397835 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:08.397363    2563 scope.go:117] "RemoveContainer" containerID="944879ca5236717c3e9cd49b40d10527ccd7f9c81d48da10c08dabebd8350955"
May 25 12:33:08.405311 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:08.405276    2563 scope.go:117] "RemoveContainer" containerID="69c7e566e625c91bfaf3fd497e2f7389ad39fd42997a36e7faa24916fa03711e"
May 25 12:33:08.412234 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:08.412214    2563 scope.go:117] "RemoveContainer" containerID="1f336be4d39b2314a3ee6cbadf20d09c3b8fa97ec92aa83800bd553b59c176fe"
May 25 12:33:08.419248 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:08.419229    2563 scope.go:117] "RemoveContainer" containerID="944879ca5236717c3e9cd49b40d10527ccd7f9c81d48da10c08dabebd8350955"
May 25 12:33:08.419508 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:33:08.419489    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"944879ca5236717c3e9cd49b40d10527ccd7f9c81d48da10c08dabebd8350955\": container with ID starting with 944879ca5236717c3e9cd49b40d10527ccd7f9c81d48da10c08dabebd8350955 not found: ID does not exist" containerID="944879ca5236717c3e9cd49b40d10527ccd7f9c81d48da10c08dabebd8350955"
May 25 12:33:08.419594 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:08.419544    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"944879ca5236717c3e9cd49b40d10527ccd7f9c81d48da10c08dabebd8350955"} err="failed to get container status \"944879ca5236717c3e9cd49b40d10527ccd7f9c81d48da10c08dabebd8350955\": rpc error: code = NotFound desc = could not find container \"944879ca5236717c3e9cd49b40d10527ccd7f9c81d48da10c08dabebd8350955\": container with ID starting with 944879ca5236717c3e9cd49b40d10527ccd7f9c81d48da10c08dabebd8350955 not found: ID does not exist"
May 25 12:33:08.419594 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:08.419565    2563 scope.go:117] "RemoveContainer" containerID="69c7e566e625c91bfaf3fd497e2f7389ad39fd42997a36e7faa24916fa03711e"
May 25 12:33:08.419817 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:33:08.419802    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69c7e566e625c91bfaf3fd497e2f7389ad39fd42997a36e7faa24916fa03711e\": container with ID starting with 69c7e566e625c91bfaf3fd497e2f7389ad39fd42997a36e7faa24916fa03711e not found: ID does not exist" containerID="69c7e566e625c91bfaf3fd497e2f7389ad39fd42997a36e7faa24916fa03711e"
May 25 12:33:08.419864 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:08.419822    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69c7e566e625c91bfaf3fd497e2f7389ad39fd42997a36e7faa24916fa03711e"} err="failed to get container status \"69c7e566e625c91bfaf3fd497e2f7389ad39fd42997a36e7faa24916fa03711e\": rpc error: code = NotFound desc = could not find container \"69c7e566e625c91bfaf3fd497e2f7389ad39fd42997a36e7faa24916fa03711e\": container with ID starting with 69c7e566e625c91bfaf3fd497e2f7389ad39fd42997a36e7faa24916fa03711e not found: ID does not exist"
May 25 12:33:08.419864 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:08.419840    2563 scope.go:117] "RemoveContainer" containerID="1f336be4d39b2314a3ee6cbadf20d09c3b8fa97ec92aa83800bd553b59c176fe"
May 25 12:33:08.419951 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:08.419937    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8"]
May 25 12:33:08.420063 ip-10-0-134-183 kubenswrapper[2563]: E0525 12:33:08.420040    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f336be4d39b2314a3ee6cbadf20d09c3b8fa97ec92aa83800bd553b59c176fe\": container with ID starting with 1f336be4d39b2314a3ee6cbadf20d09c3b8fa97ec92aa83800bd553b59c176fe not found: ID does not exist" containerID="1f336be4d39b2314a3ee6cbadf20d09c3b8fa97ec92aa83800bd553b59c176fe"
May 25 12:33:08.420104 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:08.420071    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f336be4d39b2314a3ee6cbadf20d09c3b8fa97ec92aa83800bd553b59c176fe"} err="failed to get container status \"1f336be4d39b2314a3ee6cbadf20d09c3b8fa97ec92aa83800bd553b59c176fe\": rpc error: code = NotFound desc = could not find container \"1f336be4d39b2314a3ee6cbadf20d09c3b8fa97ec92aa83800bd553b59c176fe\": container with ID starting with 1f336be4d39b2314a3ee6cbadf20d09c3b8fa97ec92aa83800bd553b59c176fe not found: ID does not exist"
May 25 12:33:08.426825 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:08.426804    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-dfccf-predictor-759f4d9cb5-p5rv8"]
May 25 12:33:08.636989 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:08.636953    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e607074-6986-474f-bda8-082c65fff8a2" path="/var/lib/kubelet/pods/4e607074-6986-474f-bda8-082c65fff8a2/volumes"
May 25 12:33:33.291334 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:33.291294    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zhhvd/must-gather-88zsq"]
May 25 12:33:33.292030 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:33.291595    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7778128c-2343-434f-8f8d-3e7bf42976fe" containerName="storage-initializer"
May 25 12:33:33.292030 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:33.291608    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7778128c-2343-434f-8f8d-3e7bf42976fe" containerName="storage-initializer"
May 25 12:33:33.292030 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:33.291615    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e607074-6986-474f-bda8-082c65fff8a2" containerName="storage-initializer"
May 25 12:33:33.292030 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:33.291621    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e607074-6986-474f-bda8-082c65fff8a2" containerName="storage-initializer"
May 25 12:33:33.292030 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:33.291629    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e607074-6986-474f-bda8-082c65fff8a2" containerName="kserve-container"
May 25 12:33:33.292030 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:33.291635    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e607074-6986-474f-bda8-082c65fff8a2" containerName="kserve-container"
May 25 12:33:33.292030 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:33.291648    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7778128c-2343-434f-8f8d-3e7bf42976fe" containerName="kube-rbac-proxy"
May 25 12:33:33.292030 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:33.291653    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7778128c-2343-434f-8f8d-3e7bf42976fe" containerName="kube-rbac-proxy"
May 25 12:33:33.292030 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:33.291660    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e607074-6986-474f-bda8-082c65fff8a2" containerName="kube-rbac-proxy"
May 25 12:33:33.292030 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:33.291665    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e607074-6986-474f-bda8-082c65fff8a2" containerName="kube-rbac-proxy"
May 25 12:33:33.292030 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:33.291673    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7778128c-2343-434f-8f8d-3e7bf42976fe" containerName="kserve-container"
May 25 12:33:33.292030 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:33.291678    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7778128c-2343-434f-8f8d-3e7bf42976fe" containerName="kserve-container"
May 25 12:33:33.292030 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:33.291722    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="4e607074-6986-474f-bda8-082c65fff8a2" containerName="kube-rbac-proxy"
May 25 12:33:33.292030 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:33.291731    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="7778128c-2343-434f-8f8d-3e7bf42976fe" containerName="kube-rbac-proxy"
May 25 12:33:33.292030 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:33.291740    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="7778128c-2343-434f-8f8d-3e7bf42976fe" containerName="kserve-container"
May 25 12:33:33.292030 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:33.291746    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="4e607074-6986-474f-bda8-082c65fff8a2" containerName="kserve-container"
May 25 12:33:33.295835 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:33.295816    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zhhvd/must-gather-88zsq"
May 25 12:33:33.297730 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:33.297711    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zhhvd\"/\"kube-root-ca.crt\""
May 25 12:33:33.298307 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:33.298291    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zhhvd\"/\"openshift-service-ca.crt\""
May 25 12:33:33.298369 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:33.298311    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zhhvd\"/\"default-dockercfg-dzb8w\""
May 25 12:33:33.301801 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:33.301776    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zhhvd/must-gather-88zsq"]
May 25 12:33:33.351510 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:33.351479    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/04a59bd7-ddef-4621-95fb-b7a5deb3b88b-must-gather-output\") pod \"must-gather-88zsq\" (UID: \"04a59bd7-ddef-4621-95fb-b7a5deb3b88b\") " pod="openshift-must-gather-zhhvd/must-gather-88zsq"
May 25 12:33:33.351705 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:33.351513    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssxvx\" (UniqueName: \"kubernetes.io/projected/04a59bd7-ddef-4621-95fb-b7a5deb3b88b-kube-api-access-ssxvx\") pod \"must-gather-88zsq\" (UID: \"04a59bd7-ddef-4621-95fb-b7a5deb3b88b\") " pod="openshift-must-gather-zhhvd/must-gather-88zsq"
May 25 12:33:33.452509 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:33.452471    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/04a59bd7-ddef-4621-95fb-b7a5deb3b88b-must-gather-output\") pod \"must-gather-88zsq\" (UID: \"04a59bd7-ddef-4621-95fb-b7a5deb3b88b\") " pod="openshift-must-gather-zhhvd/must-gather-88zsq"
May 25 12:33:33.452700 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:33.452553    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ssxvx\" (UniqueName: \"kubernetes.io/projected/04a59bd7-ddef-4621-95fb-b7a5deb3b88b-kube-api-access-ssxvx\") pod \"must-gather-88zsq\" (UID: \"04a59bd7-ddef-4621-95fb-b7a5deb3b88b\") " pod="openshift-must-gather-zhhvd/must-gather-88zsq"
May 25 12:33:33.452879 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:33.452857    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/04a59bd7-ddef-4621-95fb-b7a5deb3b88b-must-gather-output\") pod \"must-gather-88zsq\" (UID: \"04a59bd7-ddef-4621-95fb-b7a5deb3b88b\") " pod="openshift-must-gather-zhhvd/must-gather-88zsq"
May 25 12:33:33.460819 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:33.460797    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssxvx\" (UniqueName: \"kubernetes.io/projected/04a59bd7-ddef-4621-95fb-b7a5deb3b88b-kube-api-access-ssxvx\") pod \"must-gather-88zsq\" (UID: \"04a59bd7-ddef-4621-95fb-b7a5deb3b88b\") " pod="openshift-must-gather-zhhvd/must-gather-88zsq"
May 25 12:33:33.614878 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:33.614770    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zhhvd/must-gather-88zsq"
May 25 12:33:33.739349 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:33.739314    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zhhvd/must-gather-88zsq"]
May 25 12:33:33.742761 ip-10-0-134-183 kubenswrapper[2563]: W0525 12:33:33.742721    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04a59bd7_ddef_4621_95fb_b7a5deb3b88b.slice/crio-6d833357c2146c236ad36d4aa0f82dc5586b8096eaade80d87d57f9f3c7eb07f WatchSource:0}: Error finding container 6d833357c2146c236ad36d4aa0f82dc5586b8096eaade80d87d57f9f3c7eb07f: Status 404 returned error can't find the container with id 6d833357c2146c236ad36d4aa0f82dc5586b8096eaade80d87d57f9f3c7eb07f
May 25 12:33:34.470075 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:34.470032    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zhhvd/must-gather-88zsq" event={"ID":"04a59bd7-ddef-4621-95fb-b7a5deb3b88b","Type":"ContainerStarted","Data":"6d833357c2146c236ad36d4aa0f82dc5586b8096eaade80d87d57f9f3c7eb07f"}
May 25 12:33:35.475811 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:35.475764    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zhhvd/must-gather-88zsq" event={"ID":"04a59bd7-ddef-4621-95fb-b7a5deb3b88b","Type":"ContainerStarted","Data":"0922c0e98f39bb5a6c4a6d456443a57c7b070a6ba76c07240332f27413fb48bc"}
May 25 12:33:35.476286 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:35.475819    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zhhvd/must-gather-88zsq" event={"ID":"04a59bd7-ddef-4621-95fb-b7a5deb3b88b","Type":"ContainerStarted","Data":"6b47279038735c0bf8b75434a7f3f92085350cc87623df44915afa85576a6fbd"}
May 25 12:33:35.491385 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:35.491314    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-zhhvd/must-gather-88zsq" podStartSLOduration=1.693909418 podStartE2EDuration="2.491292782s" podCreationTimestamp="2026-05-25 12:33:33 +0000 UTC" firstStartedPulling="2026-05-25 12:33:33.744493708 +0000 UTC m=+1279.711885325" lastFinishedPulling="2026-05-25 12:33:34.541877059 +0000 UTC m=+1280.509268689" observedRunningTime="2026-05-25 12:33:35.490444288 +0000 UTC m=+1281.457835927" watchObservedRunningTime="2026-05-25 12:33:35.491292782 +0000 UTC m=+1281.458684422"
May 25 12:33:36.001926 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:36.001894    2563 ???:1] "http: TLS handshake error from 10.0.134.183:34706: EOF"
May 25 12:33:36.007078 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:36.007039    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-475zv_be7c8adf-8b89-4ab0-9f79-b5d4fd568ded/global-pull-secret-syncer/0.log"
May 25 12:33:36.170574 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:36.170540    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vq5g9_45b823f7-17fe-46bc-9c01-c126b9d66b9e/konnectivity-agent/0.log"
May 25 12:33:36.225536 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:36.225488    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-183.ec2.internal_44c84599553bc2d6ec69b56732f924ff/haproxy/0.log"
May 25 12:33:39.394020 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:39.393910    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-w7xgj_87277b1c-0afb-4fb9-8304-d9cf97d95458/kube-state-metrics/0.log"
May 25 12:33:39.424367 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:39.424328    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-w7xgj_87277b1c-0afb-4fb9-8304-d9cf97d95458/kube-rbac-proxy-main/0.log"
May 25 12:33:39.470406 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:39.470355    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-w7xgj_87277b1c-0afb-4fb9-8304-d9cf97d95458/kube-rbac-proxy-self/0.log"
May 25 12:33:39.550722 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:39.550674    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-665c9c8584-c6x4s_723fcb86-a890-4564-a706-997b76626043/monitoring-plugin/0.log"
May 25 12:33:39.757262 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:39.757227    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n6wtl_92294667-9ef8-4c97-8bc2-4cb222142abe/node-exporter/0.log"
May 25 12:33:39.782834 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:39.782803    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n6wtl_92294667-9ef8-4c97-8bc2-4cb222142abe/kube-rbac-proxy/0.log"
May 25 12:33:39.810415 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:39.810378    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n6wtl_92294667-9ef8-4c97-8bc2-4cb222142abe/init-textfile/0.log"
May 25 12:33:39.947775 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:39.947738    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5cef7c45-3024-47d3-88d9-bab6710a1e14/prometheus/0.log"
May 25 12:33:39.973729 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:39.973701    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5cef7c45-3024-47d3-88d9-bab6710a1e14/config-reloader/0.log"
May 25 12:33:40.001770 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:40.001734    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5cef7c45-3024-47d3-88d9-bab6710a1e14/thanos-sidecar/0.log"
May 25 12:33:40.029807 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:40.029728    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5cef7c45-3024-47d3-88d9-bab6710a1e14/kube-rbac-proxy-web/0.log"
May 25 12:33:40.058479 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:40.058451    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5cef7c45-3024-47d3-88d9-bab6710a1e14/kube-rbac-proxy/0.log"
May 25 12:33:40.087455 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:40.087429    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5cef7c45-3024-47d3-88d9-bab6710a1e14/kube-rbac-proxy-thanos/0.log"
May 25 12:33:40.116025 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:40.115973    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5cef7c45-3024-47d3-88d9-bab6710a1e14/init-config-reloader/0.log"
May 25 12:33:43.263957 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:43.263918    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zhhvd/perf-node-gather-daemonset-6525x"]
May 25 12:33:43.268421 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:43.268394    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zhhvd/perf-node-gather-daemonset-6525x"
May 25 12:33:43.276540 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:43.275576    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zhhvd/perf-node-gather-daemonset-6525x"]
May 25 12:33:43.344946 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:43.344912    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3aa3b96a-b452-4e44-8ea1-6cd4c2971918-sys\") pod \"perf-node-gather-daemonset-6525x\" (UID: \"3aa3b96a-b452-4e44-8ea1-6cd4c2971918\") " pod="openshift-must-gather-zhhvd/perf-node-gather-daemonset-6525x"
May 25 12:33:43.344946 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:43.344949    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3aa3b96a-b452-4e44-8ea1-6cd4c2971918-lib-modules\") pod \"perf-node-gather-daemonset-6525x\" (UID: \"3aa3b96a-b452-4e44-8ea1-6cd4c2971918\") " pod="openshift-must-gather-zhhvd/perf-node-gather-daemonset-6525x"
May 25 12:33:43.345160 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:43.344973    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3aa3b96a-b452-4e44-8ea1-6cd4c2971918-podres\") pod \"perf-node-gather-daemonset-6525x\" (UID: \"3aa3b96a-b452-4e44-8ea1-6cd4c2971918\") " pod="openshift-must-gather-zhhvd/perf-node-gather-daemonset-6525x"
May 25 12:33:43.345160 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:43.345058    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3aa3b96a-b452-4e44-8ea1-6cd4c2971918-proc\") pod \"perf-node-gather-daemonset-6525x\" (UID: \"3aa3b96a-b452-4e44-8ea1-6cd4c2971918\") " pod="openshift-must-gather-zhhvd/perf-node-gather-daemonset-6525x"
May 25 12:33:43.345160 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:43.345102    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x4x8\" (UniqueName: \"kubernetes.io/projected/3aa3b96a-b452-4e44-8ea1-6cd4c2971918-kube-api-access-7x4x8\") pod \"perf-node-gather-daemonset-6525x\" (UID: \"3aa3b96a-b452-4e44-8ea1-6cd4c2971918\") " pod="openshift-must-gather-zhhvd/perf-node-gather-daemonset-6525x"
May 25 12:33:43.446507 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:43.446460    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3aa3b96a-b452-4e44-8ea1-6cd4c2971918-proc\") pod \"perf-node-gather-daemonset-6525x\" (UID: \"3aa3b96a-b452-4e44-8ea1-6cd4c2971918\") " pod="openshift-must-gather-zhhvd/perf-node-gather-daemonset-6525x"
May 25 12:33:43.446735 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:43.446567    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7x4x8\" (UniqueName: \"kubernetes.io/projected/3aa3b96a-b452-4e44-8ea1-6cd4c2971918-kube-api-access-7x4x8\") pod \"perf-node-gather-daemonset-6525x\" (UID: \"3aa3b96a-b452-4e44-8ea1-6cd4c2971918\") " pod="openshift-must-gather-zhhvd/perf-node-gather-daemonset-6525x"
May 25 12:33:43.446735 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:43.446617    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3aa3b96a-b452-4e44-8ea1-6cd4c2971918-proc\") pod \"perf-node-gather-daemonset-6525x\" (UID: \"3aa3b96a-b452-4e44-8ea1-6cd4c2971918\") " pod="openshift-must-gather-zhhvd/perf-node-gather-daemonset-6525x"
May 25 12:33:43.446735 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:43.446660    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3aa3b96a-b452-4e44-8ea1-6cd4c2971918-sys\") pod \"perf-node-gather-daemonset-6525x\" (UID: \"3aa3b96a-b452-4e44-8ea1-6cd4c2971918\") " pod="openshift-must-gather-zhhvd/perf-node-gather-daemonset-6525x"
May 25 12:33:43.446735 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:43.446688    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3aa3b96a-b452-4e44-8ea1-6cd4c2971918-lib-modules\") pod \"perf-node-gather-daemonset-6525x\" (UID: \"3aa3b96a-b452-4e44-8ea1-6cd4c2971918\") " pod="openshift-must-gather-zhhvd/perf-node-gather-daemonset-6525x"
May 25 12:33:43.446735 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:43.446723    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3aa3b96a-b452-4e44-8ea1-6cd4c2971918-podres\") pod \"perf-node-gather-daemonset-6525x\" (UID: \"3aa3b96a-b452-4e44-8ea1-6cd4c2971918\") " pod="openshift-must-gather-zhhvd/perf-node-gather-daemonset-6525x"
May 25 12:33:43.447096 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:43.446771    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3aa3b96a-b452-4e44-8ea1-6cd4c2971918-sys\") pod \"perf-node-gather-daemonset-6525x\" (UID: \"3aa3b96a-b452-4e44-8ea1-6cd4c2971918\") " pod="openshift-must-gather-zhhvd/perf-node-gather-daemonset-6525x"
May 25 12:33:43.447096 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:43.446884    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3aa3b96a-b452-4e44-8ea1-6cd4c2971918-podres\") pod \"perf-node-gather-daemonset-6525x\" (UID: \"3aa3b96a-b452-4e44-8ea1-6cd4c2971918\") " pod="openshift-must-gather-zhhvd/perf-node-gather-daemonset-6525x"
May 25 12:33:43.447096 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:43.446885    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3aa3b96a-b452-4e44-8ea1-6cd4c2971918-lib-modules\") pod \"perf-node-gather-daemonset-6525x\" (UID: \"3aa3b96a-b452-4e44-8ea1-6cd4c2971918\") " pod="openshift-must-gather-zhhvd/perf-node-gather-daemonset-6525x"
May 25 12:33:43.457178 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:43.457153    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x4x8\" (UniqueName: \"kubernetes.io/projected/3aa3b96a-b452-4e44-8ea1-6cd4c2971918-kube-api-access-7x4x8\") pod \"perf-node-gather-daemonset-6525x\" (UID: \"3aa3b96a-b452-4e44-8ea1-6cd4c2971918\") " pod="openshift-must-gather-zhhvd/perf-node-gather-daemonset-6525x"
May 25 12:33:43.580424 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:43.580326    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zhhvd/perf-node-gather-daemonset-6525x"
May 25 12:33:43.731611 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:43.731569    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zhhvd/perf-node-gather-daemonset-6525x"]
May 25 12:33:43.818793 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:43.818769    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mwj87_59e9e457-5adf-41c0-ab76-f1a65d2fd966/dns/0.log"
May 25 12:33:43.846835 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:43.846767    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mwj87_59e9e457-5adf-41c0-ab76-f1a65d2fd966/kube-rbac-proxy/0.log"
May 25 12:33:43.926987 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:43.926954    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5mdjk_27b08c8c-8256-4b27-86d6-6eb05ad8bfff/dns-node-resolver/0.log"
May 25 12:33:44.399299 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:44.399269    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5d8f856f7c-fk5cm_e2f19b37-4109-4d51-b6a9-4463b728ad1d/registry/0.log"
May 25 12:33:44.474110 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:44.474080    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sbtvl_4d8f666d-fd9f-4ff8-90c8-b6bfeb3be623/node-ca/0.log"
May 25 12:33:44.515234 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:44.515196    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zhhvd/perf-node-gather-daemonset-6525x" event={"ID":"3aa3b96a-b452-4e44-8ea1-6cd4c2971918","Type":"ContainerStarted","Data":"66c8c480ff06c6464d366b4e30787800185441c7b934f6eaa178fad2ad25f957"}
May 25 12:33:44.515234 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:44.515236    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zhhvd/perf-node-gather-daemonset-6525x" event={"ID":"3aa3b96a-b452-4e44-8ea1-6cd4c2971918","Type":"ContainerStarted","Data":"5b035d56c019cb90d1671ff3ee91a9beb1e9ea8d2d3804271dd2bc81df0971a7"}
May 25 12:33:44.515451 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:44.515335    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zhhvd/perf-node-gather-daemonset-6525x"
May 25 12:33:44.531104 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:44.531049    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-zhhvd/perf-node-gather-daemonset-6525x" podStartSLOduration=1.531031532 podStartE2EDuration="1.531031532s" podCreationTimestamp="2026-05-25 12:33:43 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:33:44.529634888 +0000 UTC m=+1290.497026527" watchObservedRunningTime="2026-05-25 12:33:44.531031532 +0000 UTC m=+1290.498423172"
May 25 12:33:45.593466 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:45.593430    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jv4s8_79e13d71-6227-450f-b6fb-2f361de9fc36/serve-healthcheck-canary/0.log"
May 25 12:33:46.027954 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:46.027917    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nd42c_ca12cb94-76a6-451c-965d-f9ef23d61c90/kube-rbac-proxy/0.log"
May 25 12:33:46.051406 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:46.051378    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nd42c_ca12cb94-76a6-451c-965d-f9ef23d61c90/exporter/0.log"
May 25 12:33:46.075462 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:46.075431    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nd42c_ca12cb94-76a6-451c-965d-f9ef23d61c90/extractor/0.log"
May 25 12:33:48.154097 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:48.154059    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-6768985688-ws6fx_50a56e05-8a6f-4fc0-9fc1-928cd0912875/manager/0.log"
May 25 12:33:48.319252 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:48.319221    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-bj9zz_6614a31e-ecf8-4bc8-b8c6-d7ec0a070375/s3-init/0.log"
May 25 12:33:50.529795 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:50.529767    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zhhvd/perf-node-gather-daemonset-6525x"
May 25 12:33:52.349689 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:52.349595    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-5sbp7_9251ed75-3aac-4f81-88ab-e0af8a0ee19a/migrator/0.log"
May 25 12:33:52.376213 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:52.376169    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-5sbp7_9251ed75-3aac-4f81-88ab-e0af8a0ee19a/graceful-termination/0.log"
May 25 12:33:53.739692 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:53.739661    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-96cz9_9f26ab5b-c341-495e-bac3-01d4475a795a/kube-multus-additional-cni-plugins/0.log"
May 25 12:33:53.765581 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:53.765545    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-96cz9_9f26ab5b-c341-495e-bac3-01d4475a795a/egress-router-binary-copy/0.log"
May 25 12:33:53.790215 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:53.790184    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-96cz9_9f26ab5b-c341-495e-bac3-01d4475a795a/cni-plugins/0.log"
May 25 12:33:53.815121 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:53.815094    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-96cz9_9f26ab5b-c341-495e-bac3-01d4475a795a/bond-cni-plugin/0.log"
May 25 12:33:53.838588 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:53.838563    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-96cz9_9f26ab5b-c341-495e-bac3-01d4475a795a/routeoverride-cni/0.log"
May 25 12:33:53.861656 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:53.861623    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-96cz9_9f26ab5b-c341-495e-bac3-01d4475a795a/whereabouts-cni-bincopy/0.log"
May 25 12:33:53.884757 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:53.884732    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-96cz9_9f26ab5b-c341-495e-bac3-01d4475a795a/whereabouts-cni/0.log"
May 25 12:33:54.283441 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:54.283407    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-qnq6q_7cd43613-c252-4e65-9c14-f9bfa930be35/kube-multus/0.log"
May 25 12:33:54.367300 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:54.367217    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6jsxz_80098ceb-2e1b-4378-9371-6c6150ed88b6/network-metrics-daemon/0.log"
May 25 12:33:54.390825 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:54.390797    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6jsxz_80098ceb-2e1b-4378-9371-6c6150ed88b6/kube-rbac-proxy/0.log"
May 25 12:33:55.301583 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:55.301551    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw9bb_012c465e-2683-488a-abc1-18ad86529eb3/ovn-controller/0.log"
May 25 12:33:55.321477 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:55.321435    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw9bb_012c465e-2683-488a-abc1-18ad86529eb3/ovn-acl-logging/0.log"
May 25 12:33:55.334066 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:55.334029    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw9bb_012c465e-2683-488a-abc1-18ad86529eb3/ovn-acl-logging/1.log"
May 25 12:33:55.358452 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:55.358423    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw9bb_012c465e-2683-488a-abc1-18ad86529eb3/kube-rbac-proxy-node/0.log"
May 25 12:33:55.382390 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:55.382359    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw9bb_012c465e-2683-488a-abc1-18ad86529eb3/kube-rbac-proxy-ovn-metrics/0.log"
May 25 12:33:55.404450 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:55.404422    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw9bb_012c465e-2683-488a-abc1-18ad86529eb3/northd/0.log"
May 25 12:33:55.429136 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:55.429110    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw9bb_012c465e-2683-488a-abc1-18ad86529eb3/nbdb/0.log"
May 25 12:33:55.452664 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:55.452632    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw9bb_012c465e-2683-488a-abc1-18ad86529eb3/sbdb/0.log"
May 25 12:33:55.646901 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:55.646831    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw9bb_012c465e-2683-488a-abc1-18ad86529eb3/ovnkube-controller/0.log"
May 25 12:33:57.420857 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:57.420828    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-slzvp_e54faaa8-8c11-4a93-b337-c914cc73a410/network-check-target-container/0.log"
May 25 12:33:58.387487 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:58.387459    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-k65b6_873ca31f-e11b-4104-a3ae-243fdc6b54c0/iptables-alerter/0.log"
May 25 12:33:59.069081 ip-10-0-134-183 kubenswrapper[2563]: I0525 12:33:59.069053    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-5zd7t_edeeb54c-d929-487b-9fd3-d09c06073cdd/tuned/0.log"