Jun 02 15:45:51.175290 ip-10-0-140-246 systemd[1]: Starting Kubernetes Kubelet...
Jun 02 15:45:51.644321 ip-10-0-140-246 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 02 15:45:51.644321 ip-10-0-140-246 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jun 02 15:45:51.644321 ip-10-0-140-246 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 02 15:45:51.644321 ip-10-0-140-246 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jun 02 15:45:51.644321 ip-10-0-140-246 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 02 15:45:51.646251 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.646121    2578 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jun 02 15:45:51.652255 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652230    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 02 15:45:51.652255 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652251    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 02 15:45:51.652255 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652256    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 02 15:45:51.652255 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652261    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 02 15:45:51.652255 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652265    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 02 15:45:51.652562 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652270    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 02 15:45:51.652562 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652274    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 02 15:45:51.652562 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652277    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 02 15:45:51.652562 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652281    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 02 15:45:51.652562 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652285    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 02 15:45:51.652562 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652289    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 02 15:45:51.652562 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652292    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 02 15:45:51.652562 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652295    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 02 15:45:51.652562 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652301    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 02 15:45:51.652562 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652307    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 02 15:45:51.652562 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652311    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 02 15:45:51.652562 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652315    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 02 15:45:51.652562 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652319    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 02 15:45:51.652562 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652323    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 02 15:45:51.652562 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652327    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 02 15:45:51.652562 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652331    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 02 15:45:51.652562 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652335    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 02 15:45:51.652562 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652338    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 02 15:45:51.652562 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652343    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 02 15:45:51.653321 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652347    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jun 02 15:45:51.653321 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652351    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 02 15:45:51.653321 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652355    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 02 15:45:51.653321 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652359    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 02 15:45:51.653321 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652372    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 02 15:45:51.653321 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652376    2578 feature_gate.go:328] unrecognized feature gate: Example
Jun 02 15:45:51.653321 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652380    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 02 15:45:51.653321 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652384    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 02 15:45:51.653321 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652388    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 02 15:45:51.653321 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652392    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 02 15:45:51.653321 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652396    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 02 15:45:51.653321 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652400    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 02 15:45:51.653321 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652404    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 02 15:45:51.653321 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652409    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 02 15:45:51.653321 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652415    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 02 15:45:51.653321 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652419    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 02 15:45:51.653321 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652423    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 02 15:45:51.653321 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652427    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 02 15:45:51.653321 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652431    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 02 15:45:51.653321 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652435    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 02 15:45:51.653321 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652439    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 02 15:45:51.654200 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652443    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 02 15:45:51.654200 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652447    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 02 15:45:51.654200 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652451    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 02 15:45:51.654200 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652455    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 02 15:45:51.654200 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652459    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 02 15:45:51.654200 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652463    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 02 15:45:51.654200 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652467    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 02 15:45:51.654200 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652471    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 02 15:45:51.654200 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652475    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 02 15:45:51.654200 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652479    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 02 15:45:51.654200 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652483    2578 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Jun 02 15:45:51.654200 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652487    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 02 15:45:51.654200 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652492    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Jun 02 15:45:51.654200 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652496    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 02 15:45:51.654200 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652500    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 02 15:45:51.654200 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652504    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 02 15:45:51.654200 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652509    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 02 15:45:51.654200 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652512    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 02 15:45:51.654200 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652516    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 02 15:45:51.654963 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652520    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 02 15:45:51.654963 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652525    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 02 15:45:51.654963 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652528    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 02 15:45:51.654963 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652532    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 02 15:45:51.654963 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652539    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 02 15:45:51.654963 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652547    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 02 15:45:51.654963 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652552    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 02 15:45:51.654963 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652556    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 02 15:45:51.654963 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652560    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 02 15:45:51.654963 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652564    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 02 15:45:51.654963 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652570    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Jun 02 15:45:51.654963 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652575    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 02 15:45:51.654963 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652580    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 02 15:45:51.654963 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652584    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 02 15:45:51.654963 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652588    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 02 15:45:51.654963 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652592    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 02 15:45:51.654963 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652596    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 02 15:45:51.654963 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652601    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 02 15:45:51.654963 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652605    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Jun 02 15:45:51.654963 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652610    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 02 15:45:51.655621 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652613    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 02 15:45:51.655621 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.652617    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 02 15:45:51.655621 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653289    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 02 15:45:51.655621 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653299    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 02 15:45:51.655621 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653304    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 02 15:45:51.655621 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653308    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Jun 02 15:45:51.655621 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653312    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 02 15:45:51.655621 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653316    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 02 15:45:51.655621 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653320    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 02 15:45:51.655621 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653324    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Jun 02 15:45:51.655621 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653328    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 02 15:45:51.655621 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653332    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 02 15:45:51.655621 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653336    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 02 15:45:51.655621 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653339    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 02 15:45:51.655621 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653343    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 02 15:45:51.655621 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653347    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 02 15:45:51.655621 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653350    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 02 15:45:51.655621 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653355    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 02 15:45:51.655621 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653360    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 02 15:45:51.656145 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653364    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 02 15:45:51.656145 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653368    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 02 15:45:51.656145 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653373    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 02 15:45:51.656145 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653377    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 02 15:45:51.656145 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653381    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 02 15:45:51.656145 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653385    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 02 15:45:51.656145 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653389    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 02 15:45:51.656145 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653393    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 02 15:45:51.656145 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653397    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 02 15:45:51.656145 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653402    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 02 15:45:51.656145 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653406    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 02 15:45:51.656145 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653410    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 02 15:45:51.656145 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653414    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 02 15:45:51.656145 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653418    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 02 15:45:51.656145 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653422    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 02 15:45:51.656145 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653426    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 02 15:45:51.656145 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653430    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 02 15:45:51.656145 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653434    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 02 15:45:51.656145 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653438    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 02 15:45:51.656145 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653442    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 02 15:45:51.656692 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653446    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 02 15:45:51.656692 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653449    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 02 15:45:51.656692 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653454    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 02 15:45:51.656692 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653458    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 02 15:45:51.656692 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653462    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 02 15:45:51.656692 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653468    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 02 15:45:51.656692 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653474    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 02 15:45:51.656692 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653479    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 02 15:45:51.656692 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653484    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 02 15:45:51.656692 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653488    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 02 15:45:51.656692 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653493    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 02 15:45:51.656692 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653498    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 02 15:45:51.656692 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653503    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 02 15:45:51.656692 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653508    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 02 15:45:51.656692 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653513    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 02 15:45:51.656692 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653517    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 02 15:45:51.656692 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653521    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 02 15:45:51.656692 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653526    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 02 15:45:51.656692 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653530    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 02 15:45:51.656692 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653534    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 02 15:45:51.657411 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653538    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 02 15:45:51.657411 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653542    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 02 15:45:51.657411 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653546    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 02 15:45:51.657411 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653550    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 02 15:45:51.657411 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653553    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 02 15:45:51.657411 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653557    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 02 15:45:51.657411 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653561    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 02 15:45:51.657411 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653565    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 02 15:45:51.657411 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653569    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Jun 02 15:45:51.657411 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653573    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 02 15:45:51.657411 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653577    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jun 02 15:45:51.657411 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653581    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 02 15:45:51.657411 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653585    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 02 15:45:51.657411 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653589    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 02 15:45:51.657411 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653593    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 02 15:45:51.657411 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653597    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 02 15:45:51.657411 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653601    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 02 15:45:51.657411 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653604    2578 feature_gate.go:328] unrecognized feature gate: Example
Jun 02 15:45:51.657411 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653608    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 02 15:45:51.657411 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653612    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 02 15:45:51.657970 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653618    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 02 15:45:51.657970 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653622    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 02 15:45:51.657970 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653627    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 02 15:45:51.657970 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653631    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 02 15:45:51.657970 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653637    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 02 15:45:51.657970 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653641    2578 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Jun 02 15:45:51.657970 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653645    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 02 15:45:51.657970 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653650    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 02 15:45:51.657970 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.653653    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 02 15:45:51.657970 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.653778    2578 flags.go:64] FLAG: --address="0.0.0.0"
Jun 02 15:45:51.657970 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.653807    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jun 02 15:45:51.657970 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.653830    2578 flags.go:64] FLAG: --anonymous-auth="true"
Jun 02 15:45:51.657970 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.653837    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jun 02 15:45:51.657970 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.653844    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Jun 02 15:45:51.657970 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.653849    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jun 02 15:45:51.657970 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.653856    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jun 02 15:45:51.657970 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.653863    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jun 02 15:45:51.657970 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.653868    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jun 02 15:45:51.657970 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.653873    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jun 02 15:45:51.657970 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.653879    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jun 02 15:45:51.657970 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.653884    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jun 02 15:45:51.658495 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.653889    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jun 02 15:45:51.658495 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.653894    2578 flags.go:64] FLAG: --cgroup-root=""
Jun 02 15:45:51.658495 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.653899    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Jun 02 15:45:51.658495 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.653903    2578 flags.go:64] FLAG: --client-ca-file=""
Jun 02 15:45:51.658495 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.653908    2578 flags.go:64] FLAG: --cloud-config=""
Jun 02 15:45:51.658495 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.653912    2578 flags.go:64] FLAG: --cloud-provider="external"
Jun 02 15:45:51.658495 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.653917    2578 flags.go:64] FLAG: --cluster-dns="[]"
Jun 02 15:45:51.658495 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.653923    2578 flags.go:64] FLAG: --cluster-domain=""
Jun 02 15:45:51.658495 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.653929    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jun 02 15:45:51.658495 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.653934    2578 flags.go:64] FLAG: --config-dir=""
Jun 02 15:45:51.658495 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.653938    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jun 02 15:45:51.658495 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.653944    2578 flags.go:64] FLAG: --container-log-max-files="5"
Jun 02 15:45:51.658495 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.653950    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jun 02 15:45:51.658495 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.653958    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jun 02 15:45:51.658495 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.653966    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jun 02 15:45:51.658495 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.653971    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jun 02 15:45:51.658495 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.653976    2578 flags.go:64] FLAG: --contention-profiling="false"
Jun 02 15:45:51.658495 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.653981    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jun 02 15:45:51.658495 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.653986    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jun 02 15:45:51.658495 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654013    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Jun 02 15:45:51.658495 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654019    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jun 02 15:45:51.658495 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654025    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jun 02 15:45:51.658495 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654031    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jun 02 15:45:51.658495 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654035    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jun 02 15:45:51.658495 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654040    2578 flags.go:64] FLAG: --enable-load-reader="false"
Jun 02 15:45:51.659210 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654044    2578 flags.go:64] FLAG: --enable-server="true"
Jun 02 15:45:51.659210 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654049    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jun 02 15:45:51.659210 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654056    2578 flags.go:64] FLAG: --event-burst="100"
Jun 02 15:45:51.659210 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654061    2578 flags.go:64] FLAG: --event-qps="50"
Jun 02 15:45:51.659210 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654066    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jun 02 15:45:51.659210 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654071    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jun 02 15:45:51.659210 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654076    2578 flags.go:64] FLAG: --eviction-hard=""
Jun 02 15:45:51.659210 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654081    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jun 02 15:45:51.659210 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654086    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jun 02 15:45:51.659210 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654091    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jun 02 15:45:51.659210 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654096    2578 flags.go:64] FLAG: --eviction-soft=""
Jun 02 15:45:51.659210 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654100    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jun 02 15:45:51.659210 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654105    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jun 02 15:45:51.659210 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654110    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jun 02 15:45:51.659210 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654114    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Jun 02 15:45:51.659210 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654119    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Jun 02 15:45:51.659210 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654123    2578 flags.go:64] FLAG: --fail-swap-on="true"
Jun 02 15:45:51.659210 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654127    2578 flags.go:64] FLAG: --feature-gates=""
Jun 02 15:45:51.659210 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654134    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Jun 02 15:45:51.659210 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654138    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jun 02 15:45:51.659210 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654145    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jun 02 15:45:51.659210 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654152    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jun 02 15:45:51.659210 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654157    2578 flags.go:64] FLAG: --healthz-port="10248"
Jun 02 15:45:51.659210 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654162    2578 flags.go:64] FLAG: --help="false"
Jun 02 15:45:51.659210 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654167    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-140-246.ec2.internal"
Jun 02 15:45:51.659823 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654172    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Jun 02 15:45:51.659823 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654177    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Jun 02 15:45:51.659823 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654182    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jun 02 15:45:51.659823 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654188    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jun 02 15:45:51.659823 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654194    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jun 02 15:45:51.659823 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654199    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jun 02 15:45:51.659823 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654203    2578 flags.go:64] FLAG: --image-service-endpoint=""
Jun 02 15:45:51.659823 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654207    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jun 02 15:45:51.659823 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654213    2578 flags.go:64] FLAG: --kube-api-burst="100"
Jun 02 15:45:51.659823 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654217    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jun 02 15:45:51.659823 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654223    2578 flags.go:64] FLAG: --kube-api-qps="50"
Jun 02 15:45:51.659823 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654227    2578 flags.go:64] FLAG: --kube-reserved=""
Jun 02 15:45:51.659823 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654232    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jun 02 15:45:51.659823 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654237    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jun 02 15:45:51.659823 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654243    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Jun 02 15:45:51.659823 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654248    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jun 02 15:45:51.659823 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654252    2578 flags.go:64] FLAG: --lock-file=""
Jun 02 15:45:51.659823 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654257    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jun 02 15:45:51.659823 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654262    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Jun 02 15:45:51.659823 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654267    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jun 02 15:45:51.659823 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654276    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Jun 02 15:45:51.659823 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654280    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jun 02 15:45:51.659823 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654285    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Jun 02 15:45:51.659823 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654290    2578 flags.go:64] FLAG: --logging-format="text"
Jun 02 15:45:51.660407 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654294    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jun 02 15:45:51.660407 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654300    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jun 02 15:45:51.660407 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654304    2578 flags.go:64] FLAG: --manifest-url=""
Jun 02 15:45:51.660407 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654310    2578 flags.go:64] FLAG: --manifest-url-header=""
Jun 02 15:45:51.660407 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654319    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jun 02 15:45:51.660407 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654324    2578 flags.go:64] FLAG: --max-open-files="1000000"
Jun 02 15:45:51.660407 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654330    2578 flags.go:64] FLAG: --max-pods="110"
Jun 02 15:45:51.660407 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654335    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jun 02 15:45:51.660407 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654340    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jun 02 15:45:51.660407 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654345    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Jun 02 15:45:51.660407 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654350    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jun 02 15:45:51.660407 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654355    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jun 02 15:45:51.660407 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654360    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jun 02 15:45:51.660407 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654364    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jun 02 15:45:51.660407 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654377    2578 flags.go:64] FLAG: --node-status-max-images="50"
Jun 02 15:45:51.660407 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654383    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jun 02 15:45:51.660407 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654387    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Jun 02 15:45:51.660407 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654393    2578 flags.go:64] FLAG: --pod-cidr=""
Jun 02 15:45:51.660407 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654397    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f1083dc6abd9805f79797fa11c9accacc4a38caa073c94cb99097faffce05983"
Jun 02 15:45:51.660407 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654406    2578 flags.go:64] FLAG: --pod-manifest-path=""
Jun 02 15:45:51.660407 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654411    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Jun 02 15:45:51.660407 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654416    2578 flags.go:64] FLAG: --pods-per-core="0"
Jun 02 15:45:51.660407 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654420    2578 flags.go:64] FLAG: --port="10250"
Jun 02 15:45:51.660407 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654425    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jun 02 15:45:51.661016 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654430    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05a1e22f3516f4f25"
Jun 02 15:45:51.661016 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654435    2578 flags.go:64] FLAG: --qos-reserved=""
Jun 02 15:45:51.661016 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654440    2578 flags.go:64] FLAG: --read-only-port="10255"
Jun 02 15:45:51.661016 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654445    2578 flags.go:64] FLAG: --register-node="true"
Jun 02 15:45:51.661016 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654450    2578 flags.go:64] FLAG: --register-schedulable="true"
Jun 02 15:45:51.661016 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654454    2578 flags.go:64] FLAG: --register-with-taints=""
Jun 02 15:45:51.661016 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654460    2578 flags.go:64] FLAG: --registry-burst="10"
Jun 02 15:45:51.661016 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654465    2578 flags.go:64] FLAG: --registry-qps="5"
Jun 02 15:45:51.661016 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654469    2578 flags.go:64] FLAG: --reserved-cpus=""
Jun 02 15:45:51.661016 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654474    2578 flags.go:64] FLAG: --reserved-memory=""
Jun 02 15:45:51.661016 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654480    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jun 02 15:45:51.661016 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654485    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jun 02 15:45:51.661016 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654494    2578 flags.go:64] FLAG: --rotate-certificates="false"
Jun 02 15:45:51.661016 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654499    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Jun 02 15:45:51.661016 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654504    2578 flags.go:64] FLAG: --runonce="false"
Jun 02 15:45:51.661016 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654509    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jun 02 15:45:51.661016 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654514    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jun 02 15:45:51.661016 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654519    2578 flags.go:64] FLAG: --seccomp-default="false"
Jun 02 15:45:51.661016 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654523    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Jun 02 15:45:51.661016 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654528    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jun 02 15:45:51.661016 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654533    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jun 02 15:45:51.661016 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654538    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jun 02 15:45:51.661016 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654543    2578 flags.go:64] FLAG: --storage-driver-password="root"
Jun 02 15:45:51.661016 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654548    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Jun 02 15:45:51.661016 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654553    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Jun 02 15:45:51.661016 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654557    2578 flags.go:64] FLAG: --storage-driver-user="root"
Jun 02 15:45:51.661627 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654563    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jun 02 15:45:51.661627 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654567    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Jun 02 15:45:51.661627 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654572    2578 flags.go:64] FLAG: --system-cgroups=""
Jun 02 15:45:51.661627 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654576    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jun 02 15:45:51.661627 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654586    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Jun 02 15:45:51.661627 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654590    2578 flags.go:64] FLAG: --tls-cert-file=""
Jun 02 15:45:51.661627 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654595    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jun 02 15:45:51.661627 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654601    2578 flags.go:64] FLAG: --tls-min-version=""
Jun 02 15:45:51.661627 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654606    2578 flags.go:64] FLAG: --tls-private-key-file=""
Jun 02 15:45:51.661627 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654610    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Jun 02 15:45:51.661627 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654615    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Jun 02 15:45:51.661627 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654619    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Jun 02 15:45:51.661627 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654625    2578 flags.go:64] FLAG: --v="2"
Jun 02 15:45:51.661627 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654631    2578 flags.go:64] FLAG: --version="false"
Jun 02 15:45:51.661627 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654638    2578 flags.go:64] FLAG: --vmodule=""
Jun 02 15:45:51.661627 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654644    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jun 02 15:45:51.661627 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.654649    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jun 02 15:45:51.661627 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.654844    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 02 15:45:51.661627 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.654855    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 02 15:45:51.661627 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.654863    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 02 15:45:51.661627 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.654868    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 02 15:45:51.661627 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.654873    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 02 15:45:51.661627 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.654880    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 02 15:45:51.662240 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.654884    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 02 15:45:51.662240 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.654888    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 02 15:45:51.662240 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.654891    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 02 15:45:51.662240 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.654896    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 02 15:45:51.662240 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.654900    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 02 15:45:51.662240 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.654904    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 02 15:45:51.662240 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.654908    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 02 15:45:51.662240 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.654913    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 02 15:45:51.662240 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.654918    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 02 15:45:51.662240 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.654922    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 02 15:45:51.662240 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.654925    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 02 15:45:51.662240 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.654930    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 02 15:45:51.662240 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.654934    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 02 15:45:51.662240 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.654938    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 02 15:45:51.662240 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.654942    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 02 15:45:51.662240 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.654946    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 02 15:45:51.662240 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.654950    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jun 02 15:45:51.662240 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.654954    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 02 15:45:51.662240 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.654958    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 02 15:45:51.662240 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.654962    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Jun 02 15:45:51.662787 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.654966    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 02 15:45:51.662787 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.654969    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 02 15:45:51.662787 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.654973    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 02 15:45:51.662787 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.654977    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 02 15:45:51.662787 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.654982    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 02 15:45:51.662787 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.654988    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 02 15:45:51.662787 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655010    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 02 15:45:51.662787 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655017    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 02 15:45:51.662787 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655022    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 02 15:45:51.662787 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655027    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 02 15:45:51.662787 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655031    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 02 15:45:51.662787 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655035    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 02 15:45:51.662787 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655039    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 02 15:45:51.662787 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655043    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 02 15:45:51.662787 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655047    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 02 15:45:51.662787 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655051    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 02 15:45:51.662787 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655055    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 02 15:45:51.662787 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655059    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 02 15:45:51.662787 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655064    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 02 15:45:51.662787 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655069    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 02 15:45:51.663294 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655073    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 02 15:45:51.663294 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655078    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Jun 02 15:45:51.663294 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655082    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 02 15:45:51.663294 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655086    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 02 15:45:51.663294 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655090    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 02 15:45:51.663294 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655095    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 02 15:45:51.663294 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655098    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 02 15:45:51.663294 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655103    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 02 15:45:51.663294 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655109    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 02 15:45:51.663294 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655116    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 02 15:45:51.663294 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655120    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 02 15:45:51.663294 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655124    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 02 15:45:51.663294 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655128    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 02 15:45:51.663294 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655132    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 02 15:45:51.663294 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655136    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 02 15:45:51.663294 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655140    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 02 15:45:51.663294 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655144    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 02 15:45:51.663294 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655149    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 02 15:45:51.663294 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655153    2578 feature_gate.go:328] unrecognized feature gate: Example
Jun 02 15:45:51.663294 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655159    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 02 15:45:51.663771 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655164    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 02 15:45:51.663771 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655168    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 02 15:45:51.663771 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655172    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 02 15:45:51.663771 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655177    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 02 15:45:51.663771 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655181    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 02 15:45:51.663771 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655184    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 02 15:45:51.663771 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655188    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 02 15:45:51.663771 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655192    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 02 15:45:51.663771 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655197    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 02 15:45:51.663771 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655201    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 02 15:45:51.663771 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655204    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Jun 02 15:45:51.663771 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655209    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 02 15:45:51.663771 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655213    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 02 15:45:51.663771 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655219    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 02 15:45:51.663771 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655223    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 02 15:45:51.663771 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655227    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 02 15:45:51.663771 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655232    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 02 15:45:51.663771 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655236    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 02 15:45:51.663771 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655240    2578 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Jun 02 15:45:51.663771 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.655245    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 02 15:45:51.664280 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.656010    2578 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
Jun 02 15:45:51.664280 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.663068    2578 server.go:530] "Kubelet version" kubeletVersion="v1.33.12"
Jun 02 15:45:51.664280 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.663085    2578 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jun 02 15:45:51.664280 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663133    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 02 15:45:51.664280 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663138    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 02 15:45:51.664280 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663141    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 02 15:45:51.664280 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663145    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 02 15:45:51.664280 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663148    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Jun 02 15:45:51.664280 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663151    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 02 15:45:51.664280 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663155    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 02 15:45:51.664280 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663158    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 02 15:45:51.664280 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663161    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 02 15:45:51.664280 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663164    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 02 15:45:51.664280 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663167    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 02 15:45:51.664280 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663169    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 02 15:45:51.664668 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663172    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 02 15:45:51.664668 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663174    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 02 15:45:51.664668 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663177    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 02 15:45:51.664668 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663180    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 02 15:45:51.664668 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663182    2578 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Jun 02 15:45:51.664668 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663185    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 02 15:45:51.664668 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663188    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 02 15:45:51.664668 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663190    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 02 15:45:51.664668 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663193    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 02 15:45:51.664668 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663195    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 02 15:45:51.664668 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663199    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 02 15:45:51.664668 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663201    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 02 15:45:51.664668 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663204    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 02 15:45:51.664668 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663206    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 02 15:45:51.664668 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663209    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 02 15:45:51.664668 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663211    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 02 15:45:51.664668 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663215    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 02 15:45:51.664668 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663219    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 02 15:45:51.664668 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663222    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 02 15:45:51.665160 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663226    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 02 15:45:51.665160 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663229    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 02 15:45:51.665160 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663231    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 02 15:45:51.665160 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663234    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 02 15:45:51.665160 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663237    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 02 15:45:51.665160 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663240    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 02 15:45:51.665160 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663242    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 02 15:45:51.665160 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663244    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 02 15:45:51.665160 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663247    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 02 15:45:51.665160 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663249    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 02 15:45:51.665160 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663252    2578 feature_gate.go:328] unrecognized feature gate: Example
Jun 02 15:45:51.665160 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663255    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 02 15:45:51.665160 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663257    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 02 15:45:51.665160 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663260    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 02 15:45:51.665160 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663262    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 02 15:45:51.665160 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663265    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 02 15:45:51.665160 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663267    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 02 15:45:51.665160 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663270    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 02 15:45:51.665160 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663272    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 02 15:45:51.665616 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663275    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 02 15:45:51.665616 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663277    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 02 15:45:51.665616 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663280    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 02 15:45:51.665616 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663282    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 02 15:45:51.665616 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663285    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 02 15:45:51.665616 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663288    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 02 15:45:51.665616 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663291    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 02 15:45:51.665616 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663293    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 02 15:45:51.665616 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663296    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Jun 02 15:45:51.665616 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663298    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Jun 02 15:45:51.665616 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663301    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 02 15:45:51.665616 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663304    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 02 15:45:51.665616 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663306    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 02 15:45:51.665616 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663310    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 02 15:45:51.665616 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663312    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 02 15:45:51.665616 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663315    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 02 15:45:51.665616 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663318    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 02 15:45:51.665616 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663320    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 02 15:45:51.665616 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663323    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 02 15:45:51.665616 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663325    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 02 15:45:51.666158 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663327    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 02 15:45:51.666158 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663330    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 02 15:45:51.666158 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663332    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 02 15:45:51.666158 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663334    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 02 15:45:51.666158 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663337    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 02 15:45:51.666158 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663339    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 02 15:45:51.666158 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663342    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 02 15:45:51.666158 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663344    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 02 15:45:51.666158 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663346    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 02 15:45:51.666158 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663349    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 02 15:45:51.666158 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663351    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 02 15:45:51.666158 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663354    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 02 15:45:51.666158 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663356    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 02 15:45:51.666158 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663359    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jun 02 15:45:51.666158 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663361    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 02 15:45:51.666158 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663364    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 02 15:45:51.666555 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.663368    2578 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
Jun 02 15:45:51.666555 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663466    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 02 15:45:51.666555 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663471    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jun 02 15:45:51.666555 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663473    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 02 15:45:51.666555 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663476    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 02 15:45:51.666555 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663479    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 02 15:45:51.666555 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663482    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 02 15:45:51.666555 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663485    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 02 15:45:51.666555 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663487    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 02 15:45:51.666555 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663491    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 02 15:45:51.666555 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663494    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 02 15:45:51.666555 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663497    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 02 15:45:51.666555 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663501    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 02 15:45:51.666555 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663504    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 02 15:45:51.666555 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663507    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 02 15:45:51.666555 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663510    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 02 15:45:51.666950 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663512    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Jun 02 15:45:51.666950 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663515    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 02 15:45:51.666950 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663517    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 02 15:45:51.666950 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663520    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 02 15:45:51.666950 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663522    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 02 15:45:51.666950 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663525    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 02 15:45:51.666950 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663528    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 02 15:45:51.666950 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663530    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 02 15:45:51.666950 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663532    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 02 15:45:51.666950 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663535    2578 feature_gate.go:328] unrecognized feature gate: Example
Jun 02 15:45:51.666950 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663537    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 02 15:45:51.666950 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663539    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 02 15:45:51.666950 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663543    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 02 15:45:51.666950 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663546    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 02 15:45:51.666950 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663549    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 02 15:45:51.666950 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663552    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 02 15:45:51.666950 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663555    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 02 15:45:51.666950 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663557    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 02 15:45:51.666950 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663560    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 02 15:45:51.667466 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663563    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 02 15:45:51.667466 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663565    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 02 15:45:51.667466 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663568    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 02 15:45:51.667466 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663571    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 02 15:45:51.667466 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663573    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 02 15:45:51.667466 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663576    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 02 15:45:51.667466 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663578    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 02 15:45:51.667466 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663581    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 02 15:45:51.667466 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663583    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 02 15:45:51.667466 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663586    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 02 15:45:51.667466 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663588    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 02 15:45:51.667466 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663591    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 02 15:45:51.667466 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663593    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 02 15:45:51.667466 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663596    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 02 15:45:51.667466 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663598    2578 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Jun 02 15:45:51.667466 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663600    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Jun 02 15:45:51.667466 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663603    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 02 15:45:51.667466 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663605    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 02 15:45:51.667466 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663608    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 02 15:45:51.667911 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663610    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 02 15:45:51.667911 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663613    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 02 15:45:51.667911 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663615    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 02 15:45:51.667911 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663617    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 02 15:45:51.667911 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663620    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 02 15:45:51.667911 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663622    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 02 15:45:51.667911 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663625    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 02 15:45:51.667911 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663627    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 02 15:45:51.667911 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663629    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 02 15:45:51.667911 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663632    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 02 15:45:51.667911 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663634    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Jun 02 15:45:51.667911 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663636    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 02 15:45:51.667911 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663639    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 02 15:45:51.667911 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663641    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 02 15:45:51.667911 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663650    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 02 15:45:51.667911 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663653    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 02 15:45:51.667911 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663656    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 02 15:45:51.667911 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663658    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 02 15:45:51.667911 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663661    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 02 15:45:51.667911 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663664    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 02 15:45:51.668400 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663666    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 02 15:45:51.668400 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663669    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 02 15:45:51.668400 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663671    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 02 15:45:51.668400 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663674    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 02 15:45:51.668400 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663676    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 02 15:45:51.668400 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663679    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 02 15:45:51.668400 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663681    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 02 15:45:51.668400 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663684    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 02 15:45:51.668400 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663686    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 02 15:45:51.668400 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663688    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 02 15:45:51.668400 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663691    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 02 15:45:51.668400 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663693    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 02 15:45:51.668400 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:51.663695    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 02 15:45:51.668400 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.663700    2578 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
Jun 02 15:45:51.668400 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.664428    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Jun 02 15:45:51.670092 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.670076    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jun 02 15:45:51.671048 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.671036    2578 server.go:1019] "Starting client certificate rotation"
Jun 02 15:45:51.671151 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.671136    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jun 02 15:45:51.671181 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.671168    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jun 02 15:45:51.696068 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.696044    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jun 02 15:45:51.700597 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.700571    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jun 02 15:45:51.716765 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.716742    2578 log.go:25] "Validated CRI v1 runtime API"
Jun 02 15:45:51.724726 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.724705    2578 log.go:25] "Validated CRI v1 image API"
Jun 02 15:45:51.725943 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.725929    2578 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jun 02 15:45:51.729972 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.729948    2578 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 94b65b8f-0e6d-4e72-b345-1312599fb1ec:/dev/nvme0n1p4 db62e89e-6779-43f8-ade9-ef2636c2433c:/dev/nvme0n1p3]
Jun 02 15:45:51.730051 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.729969    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jun 02 15:45:51.732527 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.732511    2578 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jun 02 15:45:51.734547 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.734436    2578 manager.go:217] Machine: {Timestamp:2026-06-02 15:45:51.73392882 +0000 UTC m=+0.436392329 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3198496 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27a4c4dacb889682d485f7bd392202 SystemUUID:ec27a4c4-dacb-8896-82d4-85f7bd392202 BootID:fb30f204-22a9-43a8-b37f-e55a23ce4385 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:14:90:c0:ce:c1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:14:90:c0:ce:c1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:03:a1:1c:21:72 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jun 02 15:45:51.734547 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.734542    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jun 02 15:45:51.734698 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.734652    2578 manager.go:233] Version: {KernelVersion:5.14.0-570.117.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260521-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jun 02 15:45:51.735828 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.735802    2578 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jun 02 15:45:51.736032 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.735829    2578 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-246.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jun 02 15:45:51.736088 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.736044    2578 topology_manager.go:138] "Creating topology manager with none policy"
Jun 02 15:45:51.736088 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.736052    2578 container_manager_linux.go:306] "Creating device plugin manager"
Jun 02 15:45:51.736088 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.736064    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jun 02 15:45:51.736844 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.736834    2578 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jun 02 15:45:51.737703 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.737694    2578 state_mem.go:36] "Initialized new in-memory state store"
Jun 02 15:45:51.737807 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.737798    2578 server.go:1267] "Using root directory" path="/var/lib/kubelet"
Jun 02 15:45:51.740900 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.740889    2578 kubelet.go:491] "Attempting to sync node with API server"
Jun 02 15:45:51.740956 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.740910    2578 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
Jun 02 15:45:51.740956 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.740927    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jun 02 15:45:51.740956 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.740935    2578 kubelet.go:397] "Adding apiserver pod source"
Jun 02 15:45:51.740956 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.740945    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jun 02 15:45:51.742112 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.742099    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jun 02 15:45:51.742158 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.742124    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jun 02 15:45:51.746062 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.746039    2578 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.12-3.rhaos4.20.git64c6a00.el9" apiVersion="v1"
Jun 02 15:45:51.748420 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.748404    2578 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jun 02 15:45:51.749782 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.749766    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fq5j4"
Jun 02 15:45:51.750776 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.750760    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jun 02 15:45:51.750866 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.750783    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jun 02 15:45:51.750866 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.750793    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jun 02 15:45:51.750866 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.750801    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jun 02 15:45:51.750866 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.750813    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jun 02 15:45:51.750866 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.750821    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jun 02 15:45:51.750866 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.750830    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jun 02 15:45:51.750866 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.750837    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jun 02 15:45:51.750866 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.750847    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jun 02 15:45:51.750866 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.750858    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jun 02 15:45:51.750866 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.750871    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jun 02 15:45:51.751151 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.750879    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jun 02 15:45:51.752219 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.752207    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jun 02 15:45:51.752219 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.752219    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jun 02 15:45:51.752608 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:51.752587    2578 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jun 02 15:45:51.752650 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:51.752621    2578 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-246.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jun 02 15:45:51.755327 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.755313    2578 watchdog_linux.go:99] "Systemd watchdog is not enabled"
Jun 02 15:45:51.755403 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.755351    2578 server.go:1295] "Started kubelet"
Jun 02 15:45:51.755487 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.755440    2578 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jun 02 15:45:51.755568 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.755535    2578 server_v1.go:47] "podresources" method="list" useActivePods=true
Jun 02 15:45:51.756108 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.756083    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jun 02 15:45:51.756111 ip-10-0-140-246 systemd[1]: Started Kubernetes Kubelet.
Jun 02 15:45:51.756744 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.756730    2578 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jun 02 15:45:51.758193 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.758173    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fq5j4"
Jun 02 15:45:51.759016 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.758985    2578 server.go:317] "Adding debug handlers to kubelet server"
Jun 02 15:45:51.760942 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.760927    2578 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-246.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jun 02 15:45:51.761957 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:51.760975    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-246.ec2.internal.18b54e25c1197236  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-246.ec2.internal,UID:ip-10-0-140-246.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-246.ec2.internal,},FirstTimestamp:2026-06-02 15:45:51.755326006 +0000 UTC m=+0.457789515,LastTimestamp:2026-06-02 15:45:51.755326006 +0000 UTC m=+0.457789515,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-246.ec2.internal,}"
Jun 02 15:45:51.763032 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.763018    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jun 02 15:45:51.763596 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.763574    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jun 02 15:45:51.763820 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.763803    2578 volume_manager.go:295] "The desired_state_of_world populator starts"
Jun 02 15:45:51.763820 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.763820    2578 volume_manager.go:297] "Starting Kubelet Volume Manager"
Jun 02 15:45:51.763940 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.763923    2578 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
Jun 02 15:45:51.764011 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.763977    2578 reconstruct.go:97] "Volume reconstruction finished"
Jun 02 15:45:51.764011 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.763984    2578 reconciler.go:26] "Reconciler: start to sync state"
Jun 02 15:45:51.764442 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.764424    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jun 02 15:45:51.764442 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.764443    2578 factory.go:55] Registering systemd factory
Jun 02 15:45:51.764442 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.764455    2578 factory.go:223] Registration of the systemd container factory successfully
Jun 02 15:45:51.764717 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.764700    2578 factory.go:153] Registering CRI-O factory
Jun 02 15:45:51.764783 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.764721    2578 factory.go:223] Registration of the crio container factory successfully
Jun 02 15:45:51.764783 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:51.764742    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-246.ec2.internal\" not found"
Jun 02 15:45:51.764783 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.764752    2578 factory.go:103] Registering Raw factory
Jun 02 15:45:51.764915 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.764823    2578 manager.go:1196] Started watching for new ooms in manager
Jun 02 15:45:51.765176 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:51.765140    2578 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jun 02 15:45:51.765262 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.765250    2578 manager.go:319] Starting recovery of all containers
Jun 02 15:45:51.771496 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.771467    2578 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 02 15:45:51.775176 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:51.775152    2578 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-246.ec2.internal\" not found" node="ip-10-0-140-246.ec2.internal"
Jun 02 15:45:51.776838 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.776818    2578 manager.go:324] Recovery completed
Jun 02 15:45:51.781329 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.781316    2578 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Jun 02 15:45:51.783005 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.782980    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-246.ec2.internal" event="NodeHasSufficientMemory"
Jun 02 15:45:51.783074 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.783022    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-246.ec2.internal" event="NodeHasNoDiskPressure"
Jun 02 15:45:51.783074 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.783036    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-246.ec2.internal" event="NodeHasSufficientPID"
Jun 02 15:45:51.783567 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.783549    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jun 02 15:45:51.783567 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.783563    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jun 02 15:45:51.783652 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.783580    2578 state_mem.go:36] "Initialized new in-memory state store"
Jun 02 15:45:51.785950 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.785939    2578 policy_none.go:49] "None policy: Start"
Jun 02 15:45:51.786005 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.785954    2578 memory_manager.go:186] "Starting memorymanager" policy="None"
Jun 02 15:45:51.786005 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.785964    2578 state_mem.go:35] "Initializing new in-memory state store"
Jun 02 15:45:51.821865 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.821651    2578 manager.go:341] "Starting Device Plugin manager"
Jun 02 15:45:51.838806 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:51.821897    2578 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jun 02 15:45:51.838806 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.821914    2578 server.go:85] "Starting device plugin registration server"
Jun 02 15:45:51.838806 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.822159    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Jun 02 15:45:51.838806 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.822172    2578 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jun 02 15:45:51.838806 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.822268    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jun 02 15:45:51.838806 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.822347    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jun 02 15:45:51.838806 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.822356    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jun 02 15:45:51.838806 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:51.823913    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jun 02 15:45:51.838806 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:51.823954    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-246.ec2.internal\" not found"
Jun 02 15:45:51.901941 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.901905    2578 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
Jun 02 15:45:51.903108 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.903095    2578 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
Jun 02 15:45:51.903178 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.903121    2578 status_manager.go:230] "Starting to sync pod status with apiserver"
Jun 02 15:45:51.903178 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.903143    2578 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jun 02 15:45:51.903178 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.903151    2578 kubelet.go:2451] "Starting kubelet main sync loop"
Jun 02 15:45:51.903285 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:51.903189    2578 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jun 02 15:45:51.906100 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.906077    2578 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 02 15:45:51.922394 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.922378    2578 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Jun 02 15:45:51.923285 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.923263    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-246.ec2.internal" event="NodeHasSufficientMemory"
Jun 02 15:45:51.923380 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.923298    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-246.ec2.internal" event="NodeHasNoDiskPressure"
Jun 02 15:45:51.923380 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.923314    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-246.ec2.internal" event="NodeHasSufficientPID"
Jun 02 15:45:51.923380 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.923344    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-246.ec2.internal"
Jun 02 15:45:51.934413 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:51.934398    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-246.ec2.internal"
Jun 02 15:45:51.934482 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:51.934420    2578 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-246.ec2.internal\": node \"ip-10-0-140-246.ec2.internal\" not found"
Jun 02 15:45:51.954304 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:51.954280    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-246.ec2.internal\" not found"
Jun 02 15:45:52.004276 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.004229    2578 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-246.ec2.internal"]
Jun 02 15:45:52.004434 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.004362    2578 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Jun 02 15:45:52.006084 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.006069    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-246.ec2.internal" event="NodeHasSufficientMemory"
Jun 02 15:45:52.006172 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.006098    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-246.ec2.internal" event="NodeHasNoDiskPressure"
Jun 02 15:45:52.006172 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.006112    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-246.ec2.internal" event="NodeHasSufficientPID"
Jun 02 15:45:52.007271 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.007258    2578 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Jun 02 15:45:52.007449 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.007434    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.ec2.internal"
Jun 02 15:45:52.007513 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.007470    2578 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Jun 02 15:45:52.008086 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.008071    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-246.ec2.internal" event="NodeHasSufficientMemory"
Jun 02 15:45:52.008167 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.008105    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-246.ec2.internal" event="NodeHasNoDiskPressure"
Jun 02 15:45:52.008167 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.008106    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-246.ec2.internal" event="NodeHasSufficientMemory"
Jun 02 15:45:52.008167 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.008118    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-246.ec2.internal" event="NodeHasSufficientPID"
Jun 02 15:45:52.008167 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.008131    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-246.ec2.internal" event="NodeHasNoDiskPressure"
Jun 02 15:45:52.008167 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.008144    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-246.ec2.internal" event="NodeHasSufficientPID"
Jun 02 15:45:52.009251 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.009238    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-246.ec2.internal"
Jun 02 15:45:52.009305 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.009261    2578 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Jun 02 15:45:52.009955 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.009937    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-246.ec2.internal" event="NodeHasSufficientMemory"
Jun 02 15:45:52.010069 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.009963    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-246.ec2.internal" event="NodeHasNoDiskPressure"
Jun 02 15:45:52.010069 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.009973    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-246.ec2.internal" event="NodeHasSufficientPID"
Jun 02 15:45:52.036716 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:52.036696    2578 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-246.ec2.internal\" not found" node="ip-10-0-140-246.ec2.internal"
Jun 02 15:45:52.041052 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:52.041035    2578 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-246.ec2.internal\" not found" node="ip-10-0-140-246.ec2.internal"
Jun 02 15:45:52.054917 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:52.054888    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-246.ec2.internal\" not found"
Jun 02 15:45:52.066250 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.066230    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/36fed07468a4db314a589b059de2ede5-config\") pod \"kube-apiserver-proxy-ip-10-0-140-246.ec2.internal\" (UID: \"36fed07468a4db314a589b059de2ede5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-246.ec2.internal"
Jun 02 15:45:52.066352 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.066255    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f52a050f57e8707b036e889aacbf12b0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-246.ec2.internal\" (UID: \"f52a050f57e8707b036e889aacbf12b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.ec2.internal"
Jun 02 15:45:52.066352 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.066276    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f52a050f57e8707b036e889aacbf12b0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-246.ec2.internal\" (UID: \"f52a050f57e8707b036e889aacbf12b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.ec2.internal"
Jun 02 15:45:52.155073 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:52.154983    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-246.ec2.internal\" not found"
Jun 02 15:45:52.167393 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.167364    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f52a050f57e8707b036e889aacbf12b0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-246.ec2.internal\" (UID: \"f52a050f57e8707b036e889aacbf12b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.ec2.internal"
Jun 02 15:45:52.167462 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.167410    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f52a050f57e8707b036e889aacbf12b0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-246.ec2.internal\" (UID: \"f52a050f57e8707b036e889aacbf12b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.ec2.internal"
Jun 02 15:45:52.167462 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.167429    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/36fed07468a4db314a589b059de2ede5-config\") pod \"kube-apiserver-proxy-ip-10-0-140-246.ec2.internal\" (UID: \"36fed07468a4db314a589b059de2ede5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-246.ec2.internal"
Jun 02 15:45:52.167462 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.167379    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f52a050f57e8707b036e889aacbf12b0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-246.ec2.internal\" (UID: \"f52a050f57e8707b036e889aacbf12b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.ec2.internal"
Jun 02 15:45:52.167576 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.167463    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/36fed07468a4db314a589b059de2ede5-config\") pod \"kube-apiserver-proxy-ip-10-0-140-246.ec2.internal\" (UID: \"36fed07468a4db314a589b059de2ede5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-246.ec2.internal"
Jun 02 15:45:52.167576 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.167501    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f52a050f57e8707b036e889aacbf12b0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-246.ec2.internal\" (UID: \"f52a050f57e8707b036e889aacbf12b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.ec2.internal"
Jun 02 15:45:52.255854 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:52.255822    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-246.ec2.internal\" not found"
Jun 02 15:45:52.339316 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.339276    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.ec2.internal"
Jun 02 15:45:52.344874 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.344858    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-246.ec2.internal"
Jun 02 15:45:52.356602 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:52.356577    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-246.ec2.internal\" not found"
Jun 02 15:45:52.457150 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:52.457063    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-246.ec2.internal\" not found"
Jun 02 15:45:52.557601 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:52.557564    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-246.ec2.internal\" not found"
Jun 02 15:45:52.658206 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:52.658174    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-246.ec2.internal\" not found"
Jun 02 15:45:52.670494 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.670468    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jun 02 15:45:52.670629 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.670602    2578 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jun 02 15:45:52.670682 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.670639    2578 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jun 02 15:45:52.759081 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:52.759039    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-246.ec2.internal\" not found"
Jun 02 15:45:52.760140 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.760104    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-06-01 15:40:51 +0000 UTC" deadline="2028-03-16 02:28:56.075759868 +0000 UTC"
Jun 02 15:45:52.760140 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.760138    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15658h43m3.315626723s"
Jun 02 15:45:52.763678 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.763656    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jun 02 15:45:52.778713 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.778688    2578 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jun 02 15:45:52.799304 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.799268    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-46tgz"
Jun 02 15:45:52.807935 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.807910    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-46tgz"
Jun 02 15:45:52.834087 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:52.834049    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf52a050f57e8707b036e889aacbf12b0.slice/crio-ea144368e51a809f3caf21df62729963a2459604cd1bb7a91965c8dd726aae3b WatchSource:0}: Error finding container ea144368e51a809f3caf21df62729963a2459604cd1bb7a91965c8dd726aae3b: Status 404 returned error can't find the container with id ea144368e51a809f3caf21df62729963a2459604cd1bb7a91965c8dd726aae3b
Jun 02 15:45:52.834292 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:52.834268    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36fed07468a4db314a589b059de2ede5.slice/crio-3180854ea46b392f3bede144c49e0997e4b40b829cad6b34c68e166ab48a38a1 WatchSource:0}: Error finding container 3180854ea46b392f3bede144c49e0997e4b40b829cad6b34c68e166ab48a38a1: Status 404 returned error can't find the container with id 3180854ea46b392f3bede144c49e0997e4b40b829cad6b34c68e166ab48a38a1
Jun 02 15:45:52.837262 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.837248    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 15:45:52.859386 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:52.859353    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-246.ec2.internal\" not found"
Jun 02 15:45:52.859718 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.859706    2578 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 02 15:45:52.864577 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.864560    2578 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.ec2.internal"
Jun 02 15:45:52.878953 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.878931    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jun 02 15:45:52.880723 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.880710    2578 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-246.ec2.internal"
Jun 02 15:45:52.889201 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.889178    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jun 02 15:45:52.906288 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.906243    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-246.ec2.internal" event={"ID":"36fed07468a4db314a589b059de2ede5","Type":"ContainerStarted","Data":"3180854ea46b392f3bede144c49e0997e4b40b829cad6b34c68e166ab48a38a1"}
Jun 02 15:45:52.907169 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.907146    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.ec2.internal" event={"ID":"f52a050f57e8707b036e889aacbf12b0","Type":"ContainerStarted","Data":"ea144368e51a809f3caf21df62729963a2459604cd1bb7a91965c8dd726aae3b"}
Jun 02 15:45:52.999216 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:52.999142    2578 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 02 15:45:53.479387 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.479357    2578 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 02 15:45:53.742450 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.742364    2578 apiserver.go:52] "Watching apiserver"
Jun 02 15:45:53.748686 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.748661    2578 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jun 02 15:45:53.750659 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.750631    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-8bxt7","openshift-ovn-kubernetes/ovnkube-node-l5gxc","kube-system/konnectivity-agent-mrkbz","openshift-image-registry/node-ca-csg6c","openshift-multus/multus-additional-cni-plugins-wr9zh","openshift-multus/multus-nnqmn","openshift-network-diagnostics/network-check-target-smdm5","openshift-network-operator/iptables-alerter-2pmfl","kube-system/kube-apiserver-proxy-ip-10-0-140-246.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-572sg","openshift-cluster-node-tuning-operator/tuned-89hj9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.ec2.internal"]
Jun 02 15:45:53.752360 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.752339    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.753673 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.753657    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mrkbz"
Jun 02 15:45:53.754676 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.754650    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jun 02 15:45:53.754831 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.754806    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-smdm5"
Jun 02 15:45:53.754935 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:53.754870    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-smdm5" podUID="a664bf1a-e1bf-4107-9ebb-076ddd6b6234"
Jun 02 15:45:53.754935 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.754901    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jun 02 15:45:53.755074 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.755032    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jun 02 15:45:53.756379 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.756359    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-572sg"
Jun 02 15:45:53.757611 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.757466    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jun 02 15:45:53.757782 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.757764    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jun 02 15:45:53.757854 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.757809    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jun 02 15:45:53.757854 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.757835    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cwc8f\""
Jun 02 15:45:53.758082 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.758044    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bxt7"
Jun 02 15:45:53.758238 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:53.758161    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bxt7" podUID="c8a1a07e-d72f-407a-a361-9ac7ed55178f"
Jun 02 15:45:53.758238 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.758232    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jun 02 15:45:53.758624 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.758506    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jun 02 15:45:53.761521 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.761503    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wr9zh"
Jun 02 15:45:53.763050 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.763032    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-csg6c"
Jun 02 15:45:53.763141 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.763095    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.764677 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.764658    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2pmfl"
Jun 02 15:45:53.765889 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.765869    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:53.769547 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.769458    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-xntcn\""
Jun 02 15:45:53.769790 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.769772    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jun 02 15:45:53.769880 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.769778    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jun 02 15:45:53.769880 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.769858    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jun 02 15:45:53.769880 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.769861    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jun 02 15:45:53.770035 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.769904    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jun 02 15:45:53.770035 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.769923    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jun 02 15:45:53.770035 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.769973    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jun 02 15:45:53.770164 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.770068    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jun 02 15:45:53.770164 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.770071    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jun 02 15:45:53.770164 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.770109    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jun 02 15:45:53.770266 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.770165    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jun 02 15:45:53.770266 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.770241    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jun 02 15:45:53.770430 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.770414    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jun 02 15:45:53.771105 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.771089    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rdwgn\""
Jun 02 15:45:53.771386 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.771372    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-k68x5\""
Jun 02 15:45:53.771453 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.771411    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-f9t9j\""
Jun 02 15:45:53.771628 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.771589    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nqvfx\""
Jun 02 15:45:53.771710 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.771658    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jun 02 15:45:53.771710 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.771665    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jun 02 15:45:53.771710 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.771660    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-n2wt8\""
Jun 02 15:45:53.771862 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.771657    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hqfzf\""
Jun 02 15:45:53.775206 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.775192    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-node-log\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.775285 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.775215    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/22d16389-ab0a-4b82-97ae-e548268c773e-host-run-k8s-cni-cncf-io\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.775285 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.775235    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkgb2\" (UniqueName: \"kubernetes.io/projected/c8a1a07e-d72f-407a-a361-9ac7ed55178f-kube-api-access-mkgb2\") pod \"network-metrics-daemon-8bxt7\" (UID: \"c8a1a07e-d72f-407a-a361-9ac7ed55178f\") " pod="openshift-multus/network-metrics-daemon-8bxt7"
Jun 02 15:45:53.775285 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.775270    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d15a2da2-4507-43f8-913a-0414ef633f40-serviceca\") pod \"node-ca-csg6c\" (UID: \"d15a2da2-4507-43f8-913a-0414ef633f40\") " pod="openshift-image-registry/node-ca-csg6c"
Jun 02 15:45:53.775399 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.775288    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/05212ac5-a6ea-4c85-9a26-c518523c3fee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-572sg\" (UID: \"05212ac5-a6ea-4c85-9a26-c518523c3fee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-572sg"
Jun 02 15:45:53.775399 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.775302    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b0f9d3d-2024-423c-9afd-dc5094e82d2f-system-cni-dir\") pod \"multus-additional-cni-plugins-wr9zh\" (UID: \"4b0f9d3d-2024-423c-9afd-dc5094e82d2f\") " pod="openshift-multus/multus-additional-cni-plugins-wr9zh"
Jun 02 15:45:53.775399 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.775318    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/47c31aee-a2ce-4b2a-808d-eca33a9c31fc-iptables-alerter-script\") pod \"iptables-alerter-2pmfl\" (UID: \"47c31aee-a2ce-4b2a-808d-eca33a9c31fc\") " pod="openshift-network-operator/iptables-alerter-2pmfl"
Jun 02 15:45:53.775399 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.775349    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-etc-openvswitch\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.775399 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.775386    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22d16389-ab0a-4b82-97ae-e548268c773e-system-cni-dir\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.775585 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.775421    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22d16389-ab0a-4b82-97ae-e548268c773e-host-var-lib-kubelet\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.775585 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.775439    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22d16389-ab0a-4b82-97ae-e548268c773e-etc-kubernetes\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.775585 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.775455    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/05212ac5-a6ea-4c85-9a26-c518523c3fee-device-dir\") pod \"aws-ebs-csi-driver-node-572sg\" (UID: \"05212ac5-a6ea-4c85-9a26-c518523c3fee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-572sg"
Jun 02 15:45:53.775585 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.775470    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-host-kubelet\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.775585 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.775483    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22d16389-ab0a-4b82-97ae-e548268c773e-multus-cni-dir\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.775585 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.775499    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b0f9d3d-2024-423c-9afd-dc5094e82d2f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wr9zh\" (UID: \"4b0f9d3d-2024-423c-9afd-dc5094e82d2f\") " pod="openshift-multus/multus-additional-cni-plugins-wr9zh"
Jun 02 15:45:53.775585 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.775513    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-run-openvswitch\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.775585 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.775527    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-host-run-ovn-kubernetes\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.775585 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.775578    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-host-cni-netd\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.775865 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.775610    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d15a2da2-4507-43f8-913a-0414ef633f40-host\") pod \"node-ca-csg6c\" (UID: \"d15a2da2-4507-43f8-913a-0414ef633f40\") " pod="openshift-image-registry/node-ca-csg6c"
Jun 02 15:45:53.775865 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.775655    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/22d16389-ab0a-4b82-97ae-e548268c773e-cni-binary-copy\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.775865 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.775679    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/05212ac5-a6ea-4c85-9a26-c518523c3fee-sys-fs\") pod \"aws-ebs-csi-driver-node-572sg\" (UID: \"05212ac5-a6ea-4c85-9a26-c518523c3fee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-572sg"
Jun 02 15:45:53.775865 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.775702    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f56b6\" (UniqueName: \"kubernetes.io/projected/d15a2da2-4507-43f8-913a-0414ef633f40-kube-api-access-f56b6\") pod \"node-ca-csg6c\" (UID: \"d15a2da2-4507-43f8-913a-0414ef633f40\") " pod="openshift-image-registry/node-ca-csg6c"
Jun 02 15:45:53.775865 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.775726    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22d16389-ab0a-4b82-97ae-e548268c773e-host-run-netns\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.775865 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.775748    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/22d16389-ab0a-4b82-97ae-e548268c773e-multus-conf-dir\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.775865 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.775773    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-var-lib-openvswitch\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.775865 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.775796    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw64s\" (UniqueName: \"kubernetes.io/projected/becbbb00-675f-4263-a999-884ce0d688b2-kube-api-access-hw64s\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.775865 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.775821    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c81f2766-91c6-479a-bfcb-fc1144278194-agent-certs\") pod \"konnectivity-agent-mrkbz\" (UID: \"c81f2766-91c6-479a-bfcb-fc1144278194\") " pod="kube-system/konnectivity-agent-mrkbz"
Jun 02 15:45:53.776193 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.775868    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/22d16389-ab0a-4b82-97ae-e548268c773e-os-release\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.776193 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.775912    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wst8s\" (UniqueName: \"kubernetes.io/projected/4b0f9d3d-2024-423c-9afd-dc5094e82d2f-kube-api-access-wst8s\") pod \"multus-additional-cni-plugins-wr9zh\" (UID: \"4b0f9d3d-2024-423c-9afd-dc5094e82d2f\") " pod="openshift-multus/multus-additional-cni-plugins-wr9zh"
Jun 02 15:45:53.776193 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.775931    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/47c31aee-a2ce-4b2a-808d-eca33a9c31fc-host-slash\") pod \"iptables-alerter-2pmfl\" (UID: \"47c31aee-a2ce-4b2a-808d-eca33a9c31fc\") " pod="openshift-network-operator/iptables-alerter-2pmfl"
Jun 02 15:45:53.776193 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.775946    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-log-socket\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.776193 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.775967    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/becbbb00-675f-4263-a999-884ce0d688b2-ovnkube-script-lib\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.776193 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.775989    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22d16389-ab0a-4b82-97ae-e548268c773e-host-var-lib-cni-bin\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.776193 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.776029    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/05212ac5-a6ea-4c85-9a26-c518523c3fee-socket-dir\") pod \"aws-ebs-csi-driver-node-572sg\" (UID: \"05212ac5-a6ea-4c85-9a26-c518523c3fee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-572sg"
Jun 02 15:45:53.776193 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.776044    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c81f2766-91c6-479a-bfcb-fc1144278194-konnectivity-ca\") pod \"konnectivity-agent-mrkbz\" (UID: \"c81f2766-91c6-479a-bfcb-fc1144278194\") " pod="kube-system/konnectivity-agent-mrkbz"
Jun 02 15:45:53.776193 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.776060    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8a1a07e-d72f-407a-a361-9ac7ed55178f-metrics-certs\") pod \"network-metrics-daemon-8bxt7\" (UID: \"c8a1a07e-d72f-407a-a361-9ac7ed55178f\") " pod="openshift-multus/network-metrics-daemon-8bxt7"
Jun 02 15:45:53.776193 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.776089    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/22d16389-ab0a-4b82-97ae-e548268c773e-cnibin\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.776193 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.776113    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b0f9d3d-2024-423c-9afd-dc5094e82d2f-cnibin\") pod \"multus-additional-cni-plugins-wr9zh\" (UID: \"4b0f9d3d-2024-423c-9afd-dc5094e82d2f\") " pod="openshift-multus/multus-additional-cni-plugins-wr9zh"
Jun 02 15:45:53.776193 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.776136    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4b0f9d3d-2024-423c-9afd-dc5094e82d2f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wr9zh\" (UID: \"4b0f9d3d-2024-423c-9afd-dc5094e82d2f\") " pod="openshift-multus/multus-additional-cni-plugins-wr9zh"
Jun 02 15:45:53.776193 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.776160    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/becbbb00-675f-4263-a999-884ce0d688b2-env-overrides\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.776193 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.776188    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/22d16389-ab0a-4b82-97ae-e548268c773e-multus-socket-dir-parent\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.776719 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.776210    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/22d16389-ab0a-4b82-97ae-e548268c773e-host-var-lib-cni-multus\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.776719 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.776232    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmjq2\" (UniqueName: \"kubernetes.io/projected/a664bf1a-e1bf-4107-9ebb-076ddd6b6234-kube-api-access-mmjq2\") pod \"network-check-target-smdm5\" (UID: \"a664bf1a-e1bf-4107-9ebb-076ddd6b6234\") " pod="openshift-network-diagnostics/network-check-target-smdm5"
Jun 02 15:45:53.776719 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.776268    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz6vx\" (UniqueName: \"kubernetes.io/projected/05212ac5-a6ea-4c85-9a26-c518523c3fee-kube-api-access-kz6vx\") pod \"aws-ebs-csi-driver-node-572sg\" (UID: \"05212ac5-a6ea-4c85-9a26-c518523c3fee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-572sg"
Jun 02 15:45:53.776719 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.776293    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4b0f9d3d-2024-423c-9afd-dc5094e82d2f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wr9zh\" (UID: \"4b0f9d3d-2024-423c-9afd-dc5094e82d2f\") " pod="openshift-multus/multus-additional-cni-plugins-wr9zh"
Jun 02 15:45:53.776719 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.776329    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-run-systemd\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.776719 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.776368    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-run-ovn\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.776719 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.776391    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-host-cni-bin\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.776719 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.776415    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/22d16389-ab0a-4b82-97ae-e548268c773e-hostroot\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.776719 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.776438    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/05212ac5-a6ea-4c85-9a26-c518523c3fee-registration-dir\") pod \"aws-ebs-csi-driver-node-572sg\" (UID: \"05212ac5-a6ea-4c85-9a26-c518523c3fee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-572sg"
Jun 02 15:45:53.776719 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.776463    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/05212ac5-a6ea-4c85-9a26-c518523c3fee-etc-selinux\") pod \"aws-ebs-csi-driver-node-572sg\" (UID: \"05212ac5-a6ea-4c85-9a26-c518523c3fee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-572sg"
Jun 02 15:45:53.776719 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.776488    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-host-run-netns\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.776719 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.776525    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/becbbb00-675f-4263-a999-884ce0d688b2-ovnkube-config\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.776719 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.776554    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-host-slash\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.776719 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.776578    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/22d16389-ab0a-4b82-97ae-e548268c773e-host-run-multus-certs\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.776719 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.776622    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqqkq\" (UniqueName: \"kubernetes.io/projected/22d16389-ab0a-4b82-97ae-e548268c773e-kube-api-access-bqqkq\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.776719 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.776642    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4b0f9d3d-2024-423c-9afd-dc5094e82d2f-os-release\") pod \"multus-additional-cni-plugins-wr9zh\" (UID: \"4b0f9d3d-2024-423c-9afd-dc5094e82d2f\") " pod="openshift-multus/multus-additional-cni-plugins-wr9zh"
Jun 02 15:45:53.777167 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.776677    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7csqb\" (UniqueName: \"kubernetes.io/projected/47c31aee-a2ce-4b2a-808d-eca33a9c31fc-kube-api-access-7csqb\") pod \"iptables-alerter-2pmfl\" (UID: \"47c31aee-a2ce-4b2a-808d-eca33a9c31fc\") " pod="openshift-network-operator/iptables-alerter-2pmfl"
Jun 02 15:45:53.777167 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.776694    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/becbbb00-675f-4263-a999-884ce0d688b2-ovn-node-metrics-cert\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.777167 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.776709    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-systemd-units\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.777167 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.776741    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.777167 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.776763    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/22d16389-ab0a-4b82-97ae-e548268c773e-multus-daemon-config\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.777167 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.776777    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b0f9d3d-2024-423c-9afd-dc5094e82d2f-cni-binary-copy\") pod \"multus-additional-cni-plugins-wr9zh\" (UID: \"4b0f9d3d-2024-423c-9afd-dc5094e82d2f\") " pod="openshift-multus/multus-additional-cni-plugins-wr9zh"
Jun 02 15:45:53.777344 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.777276    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jun 02 15:45:53.781301 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.781278    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jun 02 15:45:53.809291 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.809256    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-01 15:40:52 +0000 UTC" deadline="2027-12-17 13:33:08.886397687 +0000 UTC"
Jun 02 15:45:53.809291 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.809284    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13509h47m15.077116441s"
Jun 02 15:45:53.864723 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.864685    2578 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
Jun 02 15:45:53.877410 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.877358    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/22d16389-ab0a-4b82-97ae-e548268c773e-cnibin\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.877571 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.877425    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6d6a55c1-8ee3-4dae-862e-1715d8d050d3-etc-sysconfig\") pod \"tuned-89hj9\" (UID: \"6d6a55c1-8ee3-4dae-862e-1715d8d050d3\") " pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:53.877571 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.877454    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6d6a55c1-8ee3-4dae-862e-1715d8d050d3-etc-sysctl-d\") pod \"tuned-89hj9\" (UID: \"6d6a55c1-8ee3-4dae-862e-1715d8d050d3\") " pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:53.877571 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.877457    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/22d16389-ab0a-4b82-97ae-e548268c773e-cnibin\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.877571 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.877479    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b0f9d3d-2024-423c-9afd-dc5094e82d2f-cnibin\") pod \"multus-additional-cni-plugins-wr9zh\" (UID: \"4b0f9d3d-2024-423c-9afd-dc5094e82d2f\") " pod="openshift-multus/multus-additional-cni-plugins-wr9zh"
Jun 02 15:45:53.877571 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.877503    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4b0f9d3d-2024-423c-9afd-dc5094e82d2f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wr9zh\" (UID: \"4b0f9d3d-2024-423c-9afd-dc5094e82d2f\") " pod="openshift-multus/multus-additional-cni-plugins-wr9zh"
Jun 02 15:45:53.877571 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.877528    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/becbbb00-675f-4263-a999-884ce0d688b2-env-overrides\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.877571 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.877550    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/22d16389-ab0a-4b82-97ae-e548268c773e-multus-socket-dir-parent\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.877571 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.877557    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b0f9d3d-2024-423c-9afd-dc5094e82d2f-cnibin\") pod \"multus-additional-cni-plugins-wr9zh\" (UID: \"4b0f9d3d-2024-423c-9afd-dc5094e82d2f\") " pod="openshift-multus/multus-additional-cni-plugins-wr9zh"
Jun 02 15:45:53.877918 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.877602    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/22d16389-ab0a-4b82-97ae-e548268c773e-host-var-lib-cni-multus\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.877918 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.877628    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6d6a55c1-8ee3-4dae-862e-1715d8d050d3-etc-systemd\") pod \"tuned-89hj9\" (UID: \"6d6a55c1-8ee3-4dae-862e-1715d8d050d3\") " pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:53.877918 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.877645    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mmjq2\" (UniqueName: \"kubernetes.io/projected/a664bf1a-e1bf-4107-9ebb-076ddd6b6234-kube-api-access-mmjq2\") pod \"network-check-target-smdm5\" (UID: \"a664bf1a-e1bf-4107-9ebb-076ddd6b6234\") " pod="openshift-network-diagnostics/network-check-target-smdm5"
Jun 02 15:45:53.877918 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.877663    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kz6vx\" (UniqueName: \"kubernetes.io/projected/05212ac5-a6ea-4c85-9a26-c518523c3fee-kube-api-access-kz6vx\") pod \"aws-ebs-csi-driver-node-572sg\" (UID: \"05212ac5-a6ea-4c85-9a26-c518523c3fee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-572sg"
Jun 02 15:45:53.877918 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.877652    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/22d16389-ab0a-4b82-97ae-e548268c773e-host-var-lib-cni-multus\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.877918 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.877699    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4b0f9d3d-2024-423c-9afd-dc5094e82d2f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wr9zh\" (UID: \"4b0f9d3d-2024-423c-9afd-dc5094e82d2f\") " pod="openshift-multus/multus-additional-cni-plugins-wr9zh"
Jun 02 15:45:53.877918 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.877701    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/22d16389-ab0a-4b82-97ae-e548268c773e-multus-socket-dir-parent\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.877918 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.877738    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-run-systemd\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.877918 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.877765    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6d6a55c1-8ee3-4dae-862e-1715d8d050d3-etc-sysctl-conf\") pod \"tuned-89hj9\" (UID: \"6d6a55c1-8ee3-4dae-862e-1715d8d050d3\") " pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:53.877918 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.877793    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-run-ovn\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.877918 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.877819    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-host-cni-bin\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.877918 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.877846    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/22d16389-ab0a-4b82-97ae-e548268c773e-hostroot\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.877918 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.877895    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6d6a55c1-8ee3-4dae-862e-1715d8d050d3-lib-modules\") pod \"tuned-89hj9\" (UID: \"6d6a55c1-8ee3-4dae-862e-1715d8d050d3\") " pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:53.879138 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.877935    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/05212ac5-a6ea-4c85-9a26-c518523c3fee-registration-dir\") pod \"aws-ebs-csi-driver-node-572sg\" (UID: \"05212ac5-a6ea-4c85-9a26-c518523c3fee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-572sg"
Jun 02 15:45:53.879138 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.877967    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/05212ac5-a6ea-4c85-9a26-c518523c3fee-etc-selinux\") pod \"aws-ebs-csi-driver-node-572sg\" (UID: \"05212ac5-a6ea-4c85-9a26-c518523c3fee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-572sg"
Jun 02 15:45:53.879138 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878007    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-host-run-netns\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.879138 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878033    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/becbbb00-675f-4263-a999-884ce0d688b2-ovnkube-config\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.879138 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878049    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/becbbb00-675f-4263-a999-884ce0d688b2-env-overrides\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.879138 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878057    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6d6a55c1-8ee3-4dae-862e-1715d8d050d3-etc-tuned\") pod \"tuned-89hj9\" (UID: \"6d6a55c1-8ee3-4dae-862e-1715d8d050d3\") " pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:53.879138 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878083    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-host-slash\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.879138 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878099    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4b0f9d3d-2024-423c-9afd-dc5094e82d2f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wr9zh\" (UID: \"4b0f9d3d-2024-423c-9afd-dc5094e82d2f\") " pod="openshift-multus/multus-additional-cni-plugins-wr9zh"
Jun 02 15:45:53.879138 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878107    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/22d16389-ab0a-4b82-97ae-e548268c773e-hostroot\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.879138 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878119    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/22d16389-ab0a-4b82-97ae-e548268c773e-host-run-multus-certs\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.879138 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878167    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/22d16389-ab0a-4b82-97ae-e548268c773e-host-run-multus-certs\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.879138 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878165    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bqqkq\" (UniqueName: \"kubernetes.io/projected/22d16389-ab0a-4b82-97ae-e548268c773e-kube-api-access-bqqkq\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.879138 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878175    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-run-systemd\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.879138 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878198    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4b0f9d3d-2024-423c-9afd-dc5094e82d2f-os-release\") pod \"multus-additional-cni-plugins-wr9zh\" (UID: \"4b0f9d3d-2024-423c-9afd-dc5094e82d2f\") " pod="openshift-multus/multus-additional-cni-plugins-wr9zh"
Jun 02 15:45:53.879138 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878229    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-host-run-netns\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.879138 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878247    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4b0f9d3d-2024-423c-9afd-dc5094e82d2f-os-release\") pod \"multus-additional-cni-plugins-wr9zh\" (UID: \"4b0f9d3d-2024-423c-9afd-dc5094e82d2f\") " pod="openshift-multus/multus-additional-cni-plugins-wr9zh"
Jun 02 15:45:53.879138 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878257    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7csqb\" (UniqueName: \"kubernetes.io/projected/47c31aee-a2ce-4b2a-808d-eca33a9c31fc-kube-api-access-7csqb\") pod \"iptables-alerter-2pmfl\" (UID: \"47c31aee-a2ce-4b2a-808d-eca33a9c31fc\") " pod="openshift-network-operator/iptables-alerter-2pmfl"
Jun 02 15:45:53.879887 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878283    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/becbbb00-675f-4263-a999-884ce0d688b2-ovn-node-metrics-cert\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.879887 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878313    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6d6a55c1-8ee3-4dae-862e-1715d8d050d3-run\") pod \"tuned-89hj9\" (UID: \"6d6a55c1-8ee3-4dae-862e-1715d8d050d3\") " pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:53.879887 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878335    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d6a55c1-8ee3-4dae-862e-1715d8d050d3-sys\") pod \"tuned-89hj9\" (UID: \"6d6a55c1-8ee3-4dae-862e-1715d8d050d3\") " pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:53.879887 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878376    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-systemd-units\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.879887 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878419    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-systemd-units\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.879887 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878423    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4b0f9d3d-2024-423c-9afd-dc5094e82d2f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wr9zh\" (UID: \"4b0f9d3d-2024-423c-9afd-dc5094e82d2f\") " pod="openshift-multus/multus-additional-cni-plugins-wr9zh"
Jun 02 15:45:53.879887 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878435    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-host-slash\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.879887 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878444    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/05212ac5-a6ea-4c85-9a26-c518523c3fee-registration-dir\") pod \"aws-ebs-csi-driver-node-572sg\" (UID: \"05212ac5-a6ea-4c85-9a26-c518523c3fee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-572sg"
Jun 02 15:45:53.879887 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878461    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-run-ovn\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.879887 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878480    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-host-cni-bin\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.879887 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878484    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.879887 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878514    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/22d16389-ab0a-4b82-97ae-e548268c773e-multus-daemon-config\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.879887 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878540    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjzjp\" (UniqueName: \"kubernetes.io/projected/6d6a55c1-8ee3-4dae-862e-1715d8d050d3-kube-api-access-sjzjp\") pod \"tuned-89hj9\" (UID: \"6d6a55c1-8ee3-4dae-862e-1715d8d050d3\") " pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:53.879887 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878546    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.879887 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878564    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b0f9d3d-2024-423c-9afd-dc5094e82d2f-cni-binary-copy\") pod \"multus-additional-cni-plugins-wr9zh\" (UID: \"4b0f9d3d-2024-423c-9afd-dc5094e82d2f\") " pod="openshift-multus/multus-additional-cni-plugins-wr9zh"
Jun 02 15:45:53.879887 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878589    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-node-log\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.879887 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878617    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/22d16389-ab0a-4b82-97ae-e548268c773e-host-run-k8s-cni-cncf-io\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.880698 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878644    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/05212ac5-a6ea-4c85-9a26-c518523c3fee-etc-selinux\") pod \"aws-ebs-csi-driver-node-572sg\" (UID: \"05212ac5-a6ea-4c85-9a26-c518523c3fee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-572sg"
Jun 02 15:45:53.880698 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878654    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/becbbb00-675f-4263-a999-884ce0d688b2-ovnkube-config\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.880698 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878663    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mkgb2\" (UniqueName: \"kubernetes.io/projected/c8a1a07e-d72f-407a-a361-9ac7ed55178f-kube-api-access-mkgb2\") pod \"network-metrics-daemon-8bxt7\" (UID: \"c8a1a07e-d72f-407a-a361-9ac7ed55178f\") " pod="openshift-multus/network-metrics-daemon-8bxt7"
Jun 02 15:45:53.880698 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878649    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jun 02 15:45:53.880698 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878689    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-node-log\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.880698 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878690    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d15a2da2-4507-43f8-913a-0414ef633f40-serviceca\") pod \"node-ca-csg6c\" (UID: \"d15a2da2-4507-43f8-913a-0414ef633f40\") " pod="openshift-image-registry/node-ca-csg6c"
Jun 02 15:45:53.880698 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878725    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/22d16389-ab0a-4b82-97ae-e548268c773e-host-run-k8s-cni-cncf-io\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.880698 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878744    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/05212ac5-a6ea-4c85-9a26-c518523c3fee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-572sg\" (UID: \"05212ac5-a6ea-4c85-9a26-c518523c3fee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-572sg"
Jun 02 15:45:53.880698 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878769    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b0f9d3d-2024-423c-9afd-dc5094e82d2f-system-cni-dir\") pod \"multus-additional-cni-plugins-wr9zh\" (UID: \"4b0f9d3d-2024-423c-9afd-dc5094e82d2f\") " pod="openshift-multus/multus-additional-cni-plugins-wr9zh"
Jun 02 15:45:53.880698 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878794    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/47c31aee-a2ce-4b2a-808d-eca33a9c31fc-iptables-alerter-script\") pod \"iptables-alerter-2pmfl\" (UID: \"47c31aee-a2ce-4b2a-808d-eca33a9c31fc\") " pod="openshift-network-operator/iptables-alerter-2pmfl"
Jun 02 15:45:53.880698 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878852    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-etc-openvswitch\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.880698 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878878    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22d16389-ab0a-4b82-97ae-e548268c773e-system-cni-dir\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.880698 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878903    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22d16389-ab0a-4b82-97ae-e548268c773e-host-var-lib-kubelet\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.880698 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878927    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22d16389-ab0a-4b82-97ae-e548268c773e-etc-kubernetes\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.880698 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878933    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/05212ac5-a6ea-4c85-9a26-c518523c3fee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-572sg\" (UID: \"05212ac5-a6ea-4c85-9a26-c518523c3fee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-572sg"
Jun 02 15:45:53.880698 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.878949    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/05212ac5-a6ea-4c85-9a26-c518523c3fee-device-dir\") pod \"aws-ebs-csi-driver-node-572sg\" (UID: \"05212ac5-a6ea-4c85-9a26-c518523c3fee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-572sg"
Jun 02 15:45:53.880698 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.879017    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-host-kubelet\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.880698 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.879047    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-etc-openvswitch\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.881400 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.879046    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22d16389-ab0a-4b82-97ae-e548268c773e-multus-cni-dir\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.881400 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.879052    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22d16389-ab0a-4b82-97ae-e548268c773e-etc-kubernetes\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.881400 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.879099    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22d16389-ab0a-4b82-97ae-e548268c773e-host-var-lib-kubelet\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.881400 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.879100    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b0f9d3d-2024-423c-9afd-dc5094e82d2f-cni-binary-copy\") pod \"multus-additional-cni-plugins-wr9zh\" (UID: \"4b0f9d3d-2024-423c-9afd-dc5094e82d2f\") " pod="openshift-multus/multus-additional-cni-plugins-wr9zh"
Jun 02 15:45:53.881400 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.879019    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22d16389-ab0a-4b82-97ae-e548268c773e-system-cni-dir\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.881400 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.879083    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d6a55c1-8ee3-4dae-862e-1715d8d050d3-var-lib-kubelet\") pod \"tuned-89hj9\" (UID: \"6d6a55c1-8ee3-4dae-862e-1715d8d050d3\") " pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:53.881400 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.879137    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-host-kubelet\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.881400 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.879146    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b0f9d3d-2024-423c-9afd-dc5094e82d2f-system-cni-dir\") pod \"multus-additional-cni-plugins-wr9zh\" (UID: \"4b0f9d3d-2024-423c-9afd-dc5094e82d2f\") " pod="openshift-multus/multus-additional-cni-plugins-wr9zh"
Jun 02 15:45:53.881400 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.879157    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6d6a55c1-8ee3-4dae-862e-1715d8d050d3-host\") pod \"tuned-89hj9\" (UID: \"6d6a55c1-8ee3-4dae-862e-1715d8d050d3\") " pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:53.881400 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.879172    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22d16389-ab0a-4b82-97ae-e548268c773e-multus-cni-dir\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.881400 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.879185    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b0f9d3d-2024-423c-9afd-dc5094e82d2f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wr9zh\" (UID: \"4b0f9d3d-2024-423c-9afd-dc5094e82d2f\") " pod="openshift-multus/multus-additional-cni-plugins-wr9zh"
Jun 02 15:45:53.881400 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.879211    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-run-openvswitch\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.881400 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.879186    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/05212ac5-a6ea-4c85-9a26-c518523c3fee-device-dir\") pod \"aws-ebs-csi-driver-node-572sg\" (UID: \"05212ac5-a6ea-4c85-9a26-c518523c3fee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-572sg"
Jun 02 15:45:53.881400 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.879234    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-host-run-ovn-kubernetes\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.881400 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.879259    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-run-openvswitch\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.881400 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.879261    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-host-cni-netd\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.881400 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.879297    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-host-run-ovn-kubernetes\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.882025 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.879312    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b0f9d3d-2024-423c-9afd-dc5094e82d2f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wr9zh\" (UID: \"4b0f9d3d-2024-423c-9afd-dc5094e82d2f\") " pod="openshift-multus/multus-additional-cni-plugins-wr9zh"
Jun 02 15:45:53.882025 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.879298    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-host-cni-netd\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.882025 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.879322    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d15a2da2-4507-43f8-913a-0414ef633f40-host\") pod \"node-ca-csg6c\" (UID: \"d15a2da2-4507-43f8-913a-0414ef633f40\") " pod="openshift-image-registry/node-ca-csg6c"
Jun 02 15:45:53.882025 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.879292    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d15a2da2-4507-43f8-913a-0414ef633f40-host\") pod \"node-ca-csg6c\" (UID: \"d15a2da2-4507-43f8-913a-0414ef633f40\") " pod="openshift-image-registry/node-ca-csg6c"
Jun 02 15:45:53.882025 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.879358    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/22d16389-ab0a-4b82-97ae-e548268c773e-cni-binary-copy\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.882025 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.879386    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6d6a55c1-8ee3-4dae-862e-1715d8d050d3-etc-modprobe-d\") pod \"tuned-89hj9\" (UID: \"6d6a55c1-8ee3-4dae-862e-1715d8d050d3\") " pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:53.882025 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.879413    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/05212ac5-a6ea-4c85-9a26-c518523c3fee-sys-fs\") pod \"aws-ebs-csi-driver-node-572sg\" (UID: \"05212ac5-a6ea-4c85-9a26-c518523c3fee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-572sg"
Jun 02 15:45:53.882025 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.879437    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f56b6\" (UniqueName: \"kubernetes.io/projected/d15a2da2-4507-43f8-913a-0414ef633f40-kube-api-access-f56b6\") pod \"node-ca-csg6c\" (UID: \"d15a2da2-4507-43f8-913a-0414ef633f40\") " pod="openshift-image-registry/node-ca-csg6c"
Jun 02 15:45:53.882025 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.879463    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22d16389-ab0a-4b82-97ae-e548268c773e-host-run-netns\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.882025 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.879488    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/22d16389-ab0a-4b82-97ae-e548268c773e-multus-conf-dir\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.882025 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.879503    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/47c31aee-a2ce-4b2a-808d-eca33a9c31fc-iptables-alerter-script\") pod \"iptables-alerter-2pmfl\" (UID: \"47c31aee-a2ce-4b2a-808d-eca33a9c31fc\") " pod="openshift-network-operator/iptables-alerter-2pmfl"
Jun 02 15:45:53.882025 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.879513    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d6a55c1-8ee3-4dae-862e-1715d8d050d3-etc-kubernetes\") pod \"tuned-89hj9\" (UID: \"6d6a55c1-8ee3-4dae-862e-1715d8d050d3\") " pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:53.882025 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.879542    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-var-lib-openvswitch\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.882025 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.879577    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/05212ac5-a6ea-4c85-9a26-c518523c3fee-sys-fs\") pod \"aws-ebs-csi-driver-node-572sg\" (UID: \"05212ac5-a6ea-4c85-9a26-c518523c3fee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-572sg"
Jun 02 15:45:53.882025 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.879631    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22d16389-ab0a-4b82-97ae-e548268c773e-host-run-netns\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.882025 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.879690    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d15a2da2-4507-43f8-913a-0414ef633f40-serviceca\") pod \"node-ca-csg6c\" (UID: \"d15a2da2-4507-43f8-913a-0414ef633f40\") " pod="openshift-image-registry/node-ca-csg6c"
Jun 02 15:45:53.882025 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.879741    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/22d16389-ab0a-4b82-97ae-e548268c773e-multus-conf-dir\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.882025 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.879865    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-var-lib-openvswitch\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.882784 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.879893    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hw64s\" (UniqueName: \"kubernetes.io/projected/becbbb00-675f-4263-a999-884ce0d688b2-kube-api-access-hw64s\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.882784 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.879926    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c81f2766-91c6-479a-bfcb-fc1144278194-agent-certs\") pod \"konnectivity-agent-mrkbz\" (UID: \"c81f2766-91c6-479a-bfcb-fc1144278194\") " pod="kube-system/konnectivity-agent-mrkbz"
Jun 02 15:45:53.882784 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.879952    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/22d16389-ab0a-4b82-97ae-e548268c773e-os-release\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.882784 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.879980    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wst8s\" (UniqueName: \"kubernetes.io/projected/4b0f9d3d-2024-423c-9afd-dc5094e82d2f-kube-api-access-wst8s\") pod \"multus-additional-cni-plugins-wr9zh\" (UID: \"4b0f9d3d-2024-423c-9afd-dc5094e82d2f\") " pod="openshift-multus/multus-additional-cni-plugins-wr9zh"
Jun 02 15:45:53.882784 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.880008    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/22d16389-ab0a-4b82-97ae-e548268c773e-cni-binary-copy\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.882784 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.880022    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/47c31aee-a2ce-4b2a-808d-eca33a9c31fc-host-slash\") pod \"iptables-alerter-2pmfl\" (UID: \"47c31aee-a2ce-4b2a-808d-eca33a9c31fc\") " pod="openshift-network-operator/iptables-alerter-2pmfl"
Jun 02 15:45:53.882784 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.880047    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-log-socket\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.882784 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.880073    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/becbbb00-675f-4263-a999-884ce0d688b2-ovnkube-script-lib\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.882784 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.880076    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/22d16389-ab0a-4b82-97ae-e548268c773e-os-release\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.882784 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.880115    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22d16389-ab0a-4b82-97ae-e548268c773e-host-var-lib-cni-bin\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.882784 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.880123    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/47c31aee-a2ce-4b2a-808d-eca33a9c31fc-host-slash\") pod \"iptables-alerter-2pmfl\" (UID: \"47c31aee-a2ce-4b2a-808d-eca33a9c31fc\") " pod="openshift-network-operator/iptables-alerter-2pmfl"
Jun 02 15:45:53.882784 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.880140    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d6a55c1-8ee3-4dae-862e-1715d8d050d3-tmp\") pod \"tuned-89hj9\" (UID: \"6d6a55c1-8ee3-4dae-862e-1715d8d050d3\") " pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:53.882784 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.880167    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/05212ac5-a6ea-4c85-9a26-c518523c3fee-socket-dir\") pod \"aws-ebs-csi-driver-node-572sg\" (UID: \"05212ac5-a6ea-4c85-9a26-c518523c3fee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-572sg"
Jun 02 15:45:53.882784 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.880197    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c81f2766-91c6-479a-bfcb-fc1144278194-konnectivity-ca\") pod \"konnectivity-agent-mrkbz\" (UID: \"c81f2766-91c6-479a-bfcb-fc1144278194\") " pod="kube-system/konnectivity-agent-mrkbz"
Jun 02 15:45:53.882784 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.880222    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8a1a07e-d72f-407a-a361-9ac7ed55178f-metrics-certs\") pod \"network-metrics-daemon-8bxt7\" (UID: \"c8a1a07e-d72f-407a-a361-9ac7ed55178f\") " pod="openshift-multus/network-metrics-daemon-8bxt7"
Jun 02 15:45:53.882784 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:53.880317    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:45:53.882784 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:53.880391    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8a1a07e-d72f-407a-a361-9ac7ed55178f-metrics-certs podName:c8a1a07e-d72f-407a-a361-9ac7ed55178f nodeName:}" failed. No retries permitted until 2026-06-02 15:45:54.380359945 +0000 UTC m=+3.082823447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c8a1a07e-d72f-407a-a361-9ac7ed55178f-metrics-certs") pod "network-metrics-daemon-8bxt7" (UID: "c8a1a07e-d72f-407a-a361-9ac7ed55178f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:45:53.883579 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.880450    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22d16389-ab0a-4b82-97ae-e548268c773e-host-var-lib-cni-bin\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.883579 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.880485    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/05212ac5-a6ea-4c85-9a26-c518523c3fee-socket-dir\") pod \"aws-ebs-csi-driver-node-572sg\" (UID: \"05212ac5-a6ea-4c85-9a26-c518523c3fee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-572sg"
Jun 02 15:45:53.883579 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.880496    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/becbbb00-675f-4263-a999-884ce0d688b2-log-socket\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.883579 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.880618    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/becbbb00-675f-4263-a999-884ce0d688b2-ovnkube-script-lib\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.883579 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.880755    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/22d16389-ab0a-4b82-97ae-e548268c773e-multus-daemon-config\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.883579 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.881389    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c81f2766-91c6-479a-bfcb-fc1144278194-konnectivity-ca\") pod \"konnectivity-agent-mrkbz\" (UID: \"c81f2766-91c6-479a-bfcb-fc1144278194\") " pod="kube-system/konnectivity-agent-mrkbz"
Jun 02 15:45:53.883579 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.882776    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/becbbb00-675f-4263-a999-884ce0d688b2-ovn-node-metrics-cert\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.883579 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.882951    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c81f2766-91c6-479a-bfcb-fc1144278194-agent-certs\") pod \"konnectivity-agent-mrkbz\" (UID: \"c81f2766-91c6-479a-bfcb-fc1144278194\") " pod="kube-system/konnectivity-agent-mrkbz"
Jun 02 15:45:53.885466 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:53.885447    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 02 15:45:53.885466 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:53.885468    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 02 15:45:53.885628 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:53.885481    2578 projected.go:194] Error preparing data for projected volume kube-api-access-mmjq2 for pod openshift-network-diagnostics/network-check-target-smdm5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:45:53.885628 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:53.885540    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a664bf1a-e1bf-4107-9ebb-076ddd6b6234-kube-api-access-mmjq2 podName:a664bf1a-e1bf-4107-9ebb-076ddd6b6234 nodeName:}" failed. No retries permitted until 2026-06-02 15:45:54.38552194 +0000 UTC m=+3.087985456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mmjq2" (UniqueName: "kubernetes.io/projected/a664bf1a-e1bf-4107-9ebb-076ddd6b6234-kube-api-access-mmjq2") pod "network-check-target-smdm5" (UID: "a664bf1a-e1bf-4107-9ebb-076ddd6b6234") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:45:53.891106 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.891048    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7csqb\" (UniqueName: \"kubernetes.io/projected/47c31aee-a2ce-4b2a-808d-eca33a9c31fc-kube-api-access-7csqb\") pod \"iptables-alerter-2pmfl\" (UID: \"47c31aee-a2ce-4b2a-808d-eca33a9c31fc\") " pod="openshift-network-operator/iptables-alerter-2pmfl"
Jun 02 15:45:53.891537 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.891487    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkgb2\" (UniqueName: \"kubernetes.io/projected/c8a1a07e-d72f-407a-a361-9ac7ed55178f-kube-api-access-mkgb2\") pod \"network-metrics-daemon-8bxt7\" (UID: \"c8a1a07e-d72f-407a-a361-9ac7ed55178f\") " pod="openshift-multus/network-metrics-daemon-8bxt7"
Jun 02 15:45:53.891537 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.891523    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wst8s\" (UniqueName: \"kubernetes.io/projected/4b0f9d3d-2024-423c-9afd-dc5094e82d2f-kube-api-access-wst8s\") pod \"multus-additional-cni-plugins-wr9zh\" (UID: \"4b0f9d3d-2024-423c-9afd-dc5094e82d2f\") " pod="openshift-multus/multus-additional-cni-plugins-wr9zh"
Jun 02 15:45:53.892015 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.891967    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz6vx\" (UniqueName: \"kubernetes.io/projected/05212ac5-a6ea-4c85-9a26-c518523c3fee-kube-api-access-kz6vx\") pod \"aws-ebs-csi-driver-node-572sg\" (UID: \"05212ac5-a6ea-4c85-9a26-c518523c3fee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-572sg"
Jun 02 15:45:53.892619 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.892597    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw64s\" (UniqueName: \"kubernetes.io/projected/becbbb00-675f-4263-a999-884ce0d688b2-kube-api-access-hw64s\") pod \"ovnkube-node-l5gxc\" (UID: \"becbbb00-675f-4263-a999-884ce0d688b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:53.892682 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.892603    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqqkq\" (UniqueName: \"kubernetes.io/projected/22d16389-ab0a-4b82-97ae-e548268c773e-kube-api-access-bqqkq\") pod \"multus-nnqmn\" (UID: \"22d16389-ab0a-4b82-97ae-e548268c773e\") " pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:53.895486 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.895465    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f56b6\" (UniqueName: \"kubernetes.io/projected/d15a2da2-4507-43f8-913a-0414ef633f40-kube-api-access-f56b6\") pod \"node-ca-csg6c\" (UID: \"d15a2da2-4507-43f8-913a-0414ef633f40\") " pod="openshift-image-registry/node-ca-csg6c"
Jun 02 15:45:53.978162 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.978134    2578 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 02 15:45:53.980808 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.980786    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6d6a55c1-8ee3-4dae-862e-1715d8d050d3-run\") pod \"tuned-89hj9\" (UID: \"6d6a55c1-8ee3-4dae-862e-1715d8d050d3\") " pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:53.980989 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.980817    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d6a55c1-8ee3-4dae-862e-1715d8d050d3-sys\") pod \"tuned-89hj9\" (UID: \"6d6a55c1-8ee3-4dae-862e-1715d8d050d3\") " pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:53.980989 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.980844    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sjzjp\" (UniqueName: \"kubernetes.io/projected/6d6a55c1-8ee3-4dae-862e-1715d8d050d3-kube-api-access-sjzjp\") pod \"tuned-89hj9\" (UID: \"6d6a55c1-8ee3-4dae-862e-1715d8d050d3\") " pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:53.980989 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.980878    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d6a55c1-8ee3-4dae-862e-1715d8d050d3-var-lib-kubelet\") pod \"tuned-89hj9\" (UID: \"6d6a55c1-8ee3-4dae-862e-1715d8d050d3\") " pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:53.980989 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.980901    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6d6a55c1-8ee3-4dae-862e-1715d8d050d3-host\") pod \"tuned-89hj9\" (UID: \"6d6a55c1-8ee3-4dae-862e-1715d8d050d3\") " pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:53.980989 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.980902    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6d6a55c1-8ee3-4dae-862e-1715d8d050d3-run\") pod \"tuned-89hj9\" (UID: \"6d6a55c1-8ee3-4dae-862e-1715d8d050d3\") " pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:53.980989 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.980928    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6d6a55c1-8ee3-4dae-862e-1715d8d050d3-etc-modprobe-d\") pod \"tuned-89hj9\" (UID: \"6d6a55c1-8ee3-4dae-862e-1715d8d050d3\") " pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:53.980989 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.980929    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d6a55c1-8ee3-4dae-862e-1715d8d050d3-sys\") pod \"tuned-89hj9\" (UID: \"6d6a55c1-8ee3-4dae-862e-1715d8d050d3\") " pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:53.980989 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.980957    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d6a55c1-8ee3-4dae-862e-1715d8d050d3-etc-kubernetes\") pod \"tuned-89hj9\" (UID: \"6d6a55c1-8ee3-4dae-862e-1715d8d050d3\") " pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:53.980989 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.980969    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6d6a55c1-8ee3-4dae-862e-1715d8d050d3-host\") pod \"tuned-89hj9\" (UID: \"6d6a55c1-8ee3-4dae-862e-1715d8d050d3\") " pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:53.981405 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.981026    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d6a55c1-8ee3-4dae-862e-1715d8d050d3-etc-kubernetes\") pod \"tuned-89hj9\" (UID: \"6d6a55c1-8ee3-4dae-862e-1715d8d050d3\") " pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:53.981405 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.981034    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d6a55c1-8ee3-4dae-862e-1715d8d050d3-tmp\") pod \"tuned-89hj9\" (UID: \"6d6a55c1-8ee3-4dae-862e-1715d8d050d3\") " pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:53.981405 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.981069    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6d6a55c1-8ee3-4dae-862e-1715d8d050d3-etc-modprobe-d\") pod \"tuned-89hj9\" (UID: \"6d6a55c1-8ee3-4dae-862e-1715d8d050d3\") " pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:53.981405 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.981080    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6d6a55c1-8ee3-4dae-862e-1715d8d050d3-etc-sysconfig\") pod \"tuned-89hj9\" (UID: \"6d6a55c1-8ee3-4dae-862e-1715d8d050d3\") " pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:53.981405 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.981026    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d6a55c1-8ee3-4dae-862e-1715d8d050d3-var-lib-kubelet\") pod \"tuned-89hj9\" (UID: \"6d6a55c1-8ee3-4dae-862e-1715d8d050d3\") " pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:53.981405 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.981110    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6d6a55c1-8ee3-4dae-862e-1715d8d050d3-etc-sysctl-d\") pod \"tuned-89hj9\" (UID: \"6d6a55c1-8ee3-4dae-862e-1715d8d050d3\") " pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:53.981405 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.981117    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6d6a55c1-8ee3-4dae-862e-1715d8d050d3-etc-sysconfig\") pod \"tuned-89hj9\" (UID: \"6d6a55c1-8ee3-4dae-862e-1715d8d050d3\") " pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:53.981405 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.981131    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6d6a55c1-8ee3-4dae-862e-1715d8d050d3-etc-systemd\") pod \"tuned-89hj9\" (UID: \"6d6a55c1-8ee3-4dae-862e-1715d8d050d3\") " pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:53.981405 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.981165    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6d6a55c1-8ee3-4dae-862e-1715d8d050d3-etc-sysctl-conf\") pod \"tuned-89hj9\" (UID: \"6d6a55c1-8ee3-4dae-862e-1715d8d050d3\") " pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:53.981405 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.981184    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6d6a55c1-8ee3-4dae-862e-1715d8d050d3-lib-modules\") pod \"tuned-89hj9\" (UID: \"6d6a55c1-8ee3-4dae-862e-1715d8d050d3\") " pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:53.981405 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.981207    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6d6a55c1-8ee3-4dae-862e-1715d8d050d3-etc-tuned\") pod \"tuned-89hj9\" (UID: \"6d6a55c1-8ee3-4dae-862e-1715d8d050d3\") " pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:53.981405 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.981229    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6d6a55c1-8ee3-4dae-862e-1715d8d050d3-etc-systemd\") pod \"tuned-89hj9\" (UID: \"6d6a55c1-8ee3-4dae-862e-1715d8d050d3\") " pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:53.981405 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.981206    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6d6a55c1-8ee3-4dae-862e-1715d8d050d3-etc-sysctl-d\") pod \"tuned-89hj9\" (UID: \"6d6a55c1-8ee3-4dae-862e-1715d8d050d3\") " pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:53.981405 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.981289    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6d6a55c1-8ee3-4dae-862e-1715d8d050d3-lib-modules\") pod \"tuned-89hj9\" (UID: \"6d6a55c1-8ee3-4dae-862e-1715d8d050d3\") " pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:53.981405 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.981343    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6d6a55c1-8ee3-4dae-862e-1715d8d050d3-etc-sysctl-conf\") pod \"tuned-89hj9\" (UID: \"6d6a55c1-8ee3-4dae-862e-1715d8d050d3\") " pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:53.983340 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.983315    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d6a55c1-8ee3-4dae-862e-1715d8d050d3-tmp\") pod \"tuned-89hj9\" (UID: \"6d6a55c1-8ee3-4dae-862e-1715d8d050d3\") " pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:53.983664 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.983646    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6d6a55c1-8ee3-4dae-862e-1715d8d050d3-etc-tuned\") pod \"tuned-89hj9\" (UID: \"6d6a55c1-8ee3-4dae-862e-1715d8d050d3\") " pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:53.989896 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:53.989875    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjzjp\" (UniqueName: \"kubernetes.io/projected/6d6a55c1-8ee3-4dae-862e-1715d8d050d3-kube-api-access-sjzjp\") pod \"tuned-89hj9\" (UID: \"6d6a55c1-8ee3-4dae-862e-1715d8d050d3\") " pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:54.064962 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:54.064881    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:45:54.071755 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:54.071727    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mrkbz"
Jun 02 15:45:54.080271 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:54.080251    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-572sg"
Jun 02 15:45:54.082820 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:54.082800    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wr9zh"
Jun 02 15:45:54.089371 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:54.089353    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-csg6c"
Jun 02 15:45:54.093843 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:54.093826    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nnqmn"
Jun 02 15:45:54.101364 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:54.101349    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2pmfl"
Jun 02 15:45:54.107892 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:54.107875    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-89hj9"
Jun 02 15:45:54.292921 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:54.292891    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05212ac5_a6ea_4c85_9a26_c518523c3fee.slice/crio-d102116e7c6ba7a5a0a3fb4066121770743487850ca6bfe5ec22f62a3988fb2c WatchSource:0}: Error finding container d102116e7c6ba7a5a0a3fb4066121770743487850ca6bfe5ec22f62a3988fb2c: Status 404 returned error can't find the container with id d102116e7c6ba7a5a0a3fb4066121770743487850ca6bfe5ec22f62a3988fb2c
Jun 02 15:45:54.294090 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:54.294020    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d6a55c1_8ee3_4dae_862e_1715d8d050d3.slice/crio-f2cd493c02df0f5545d7a5c585df7d485fa94d50d48dec54c042ff4e66eb18d5 WatchSource:0}: Error finding container f2cd493c02df0f5545d7a5c585df7d485fa94d50d48dec54c042ff4e66eb18d5: Status 404 returned error can't find the container with id f2cd493c02df0f5545d7a5c585df7d485fa94d50d48dec54c042ff4e66eb18d5
Jun 02 15:45:54.295953 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:54.295885    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd15a2da2_4507_43f8_913a_0414ef633f40.slice/crio-85f42fffbe86dff390c6a59c256c9d11720c84a48995f8b62bb27951d34ac751 WatchSource:0}: Error finding container 85f42fffbe86dff390c6a59c256c9d11720c84a48995f8b62bb27951d34ac751: Status 404 returned error can't find the container with id 85f42fffbe86dff390c6a59c256c9d11720c84a48995f8b62bb27951d34ac751
Jun 02 15:45:54.298726 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:54.298688    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc81f2766_91c6_479a_bfcb_fc1144278194.slice/crio-3980c38640bd72d7073e991a5075b853f89b03dbf6b7997e0906f72fa0f93d80 WatchSource:0}: Error finding container 3980c38640bd72d7073e991a5075b853f89b03dbf6b7997e0906f72fa0f93d80: Status 404 returned error can't find the container with id 3980c38640bd72d7073e991a5075b853f89b03dbf6b7997e0906f72fa0f93d80
Jun 02 15:45:54.298943 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:54.298923    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22d16389_ab0a_4b82_97ae_e548268c773e.slice/crio-8e92543b43e7f8c57e67514b752eb581f5a34a4c0004b3b6d4032a11f1a6c2d8 WatchSource:0}: Error finding container 8e92543b43e7f8c57e67514b752eb581f5a34a4c0004b3b6d4032a11f1a6c2d8: Status 404 returned error can't find the container with id 8e92543b43e7f8c57e67514b752eb581f5a34a4c0004b3b6d4032a11f1a6c2d8
Jun 02 15:45:54.299955 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:54.299931    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47c31aee_a2ce_4b2a_808d_eca33a9c31fc.slice/crio-ea31b6844344322d70cfdda7a549e17dcd873ad392f5b3940c0c78e65b7c4560 WatchSource:0}: Error finding container ea31b6844344322d70cfdda7a549e17dcd873ad392f5b3940c0c78e65b7c4560: Status 404 returned error can't find the container with id ea31b6844344322d70cfdda7a549e17dcd873ad392f5b3940c0c78e65b7c4560
Jun 02 15:45:54.301731 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:45:54.301645    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbecbbb00_675f_4263_a999_884ce0d688b2.slice/crio-e0e9aa48036c886e32735cdf2aaddce6a27588abc346c2a2a57befbfb8ac848d WatchSource:0}: Error finding container e0e9aa48036c886e32735cdf2aaddce6a27588abc346c2a2a57befbfb8ac848d: Status 404 returned error can't find the container with id e0e9aa48036c886e32735cdf2aaddce6a27588abc346c2a2a57befbfb8ac848d
Jun 02 15:45:54.384140 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:54.383967    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8a1a07e-d72f-407a-a361-9ac7ed55178f-metrics-certs\") pod \"network-metrics-daemon-8bxt7\" (UID: \"c8a1a07e-d72f-407a-a361-9ac7ed55178f\") " pod="openshift-multus/network-metrics-daemon-8bxt7"
Jun 02 15:45:54.384254 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:54.384093    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:45:54.384254 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:54.384239    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8a1a07e-d72f-407a-a361-9ac7ed55178f-metrics-certs podName:c8a1a07e-d72f-407a-a361-9ac7ed55178f nodeName:}" failed. No retries permitted until 2026-06-02 15:45:55.384222686 +0000 UTC m=+4.086686182 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c8a1a07e-d72f-407a-a361-9ac7ed55178f-metrics-certs") pod "network-metrics-daemon-8bxt7" (UID: "c8a1a07e-d72f-407a-a361-9ac7ed55178f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:45:54.484440 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:54.484402    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mmjq2\" (UniqueName: \"kubernetes.io/projected/a664bf1a-e1bf-4107-9ebb-076ddd6b6234-kube-api-access-mmjq2\") pod \"network-check-target-smdm5\" (UID: \"a664bf1a-e1bf-4107-9ebb-076ddd6b6234\") " pod="openshift-network-diagnostics/network-check-target-smdm5"
Jun 02 15:45:54.484615 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:54.484556    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 02 15:45:54.484615 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:54.484577    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 02 15:45:54.484615 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:54.484587    2578 projected.go:194] Error preparing data for projected volume kube-api-access-mmjq2 for pod openshift-network-diagnostics/network-check-target-smdm5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:45:54.484756 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:54.484653    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a664bf1a-e1bf-4107-9ebb-076ddd6b6234-kube-api-access-mmjq2 podName:a664bf1a-e1bf-4107-9ebb-076ddd6b6234 nodeName:}" failed. No retries permitted until 2026-06-02 15:45:55.484632851 +0000 UTC m=+4.187096348 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mmjq2" (UniqueName: "kubernetes.io/projected/a664bf1a-e1bf-4107-9ebb-076ddd6b6234-kube-api-access-mmjq2") pod "network-check-target-smdm5" (UID: "a664bf1a-e1bf-4107-9ebb-076ddd6b6234") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:45:54.810463 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:54.810391    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-01 15:40:52 +0000 UTC" deadline="2028-01-03 19:15:02.453203224 +0000 UTC"
Jun 02 15:45:54.810463 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:54.810429    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13923h29m7.642777772s"
Jun 02 15:45:54.927692 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:54.927622    2578 generic.go:358] "Generic (PLEG): container finished" podID="f52a050f57e8707b036e889aacbf12b0" containerID="200e5b24e95732527491aae45d83936f72d016f58474e3fba1586600fb14dd06" exitCode=0
Jun 02 15:45:54.928009 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:54.927966    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.ec2.internal" event={"ID":"f52a050f57e8707b036e889aacbf12b0","Type":"ContainerDied","Data":"200e5b24e95732527491aae45d83936f72d016f58474e3fba1586600fb14dd06"}
Jun 02 15:45:54.935324 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:54.935275    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wr9zh" event={"ID":"4b0f9d3d-2024-423c-9afd-dc5094e82d2f","Type":"ContainerStarted","Data":"e8553a7ed500b593d7e64235665cb440de79905679f2744109b9491a238cfefa"}
Jun 02 15:45:54.944273 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:54.944238    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc" event={"ID":"becbbb00-675f-4263-a999-884ce0d688b2","Type":"ContainerStarted","Data":"e0e9aa48036c886e32735cdf2aaddce6a27588abc346c2a2a57befbfb8ac848d"}
Jun 02 15:45:54.951020 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:54.950949    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2pmfl" event={"ID":"47c31aee-a2ce-4b2a-808d-eca33a9c31fc","Type":"ContainerStarted","Data":"ea31b6844344322d70cfdda7a549e17dcd873ad392f5b3940c0c78e65b7c4560"}
Jun 02 15:45:54.955151 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:54.955124    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mrkbz" event={"ID":"c81f2766-91c6-479a-bfcb-fc1144278194","Type":"ContainerStarted","Data":"3980c38640bd72d7073e991a5075b853f89b03dbf6b7997e0906f72fa0f93d80"}
Jun 02 15:45:54.956851 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:54.956827    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-csg6c" event={"ID":"d15a2da2-4507-43f8-913a-0414ef633f40","Type":"ContainerStarted","Data":"85f42fffbe86dff390c6a59c256c9d11720c84a48995f8b62bb27951d34ac751"}
Jun 02 15:45:54.965112 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:54.965082    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nnqmn" event={"ID":"22d16389-ab0a-4b82-97ae-e548268c773e","Type":"ContainerStarted","Data":"8e92543b43e7f8c57e67514b752eb581f5a34a4c0004b3b6d4032a11f1a6c2d8"}
Jun 02 15:45:54.976761 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:54.976716    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-89hj9" event={"ID":"6d6a55c1-8ee3-4dae-862e-1715d8d050d3","Type":"ContainerStarted","Data":"f2cd493c02df0f5545d7a5c585df7d485fa94d50d48dec54c042ff4e66eb18d5"}
Jun 02 15:45:54.994295 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:54.994259    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-572sg" event={"ID":"05212ac5-a6ea-4c85-9a26-c518523c3fee","Type":"ContainerStarted","Data":"d102116e7c6ba7a5a0a3fb4066121770743487850ca6bfe5ec22f62a3988fb2c"}
Jun 02 15:45:55.005696 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:55.005662    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-246.ec2.internal" event={"ID":"36fed07468a4db314a589b059de2ede5","Type":"ContainerStarted","Data":"47cf76f60abdebffd4acf821b8b645183662d9b258c13222c9b3d6cce46d6839"}
Jun 02 15:45:55.393680 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:55.393585    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8a1a07e-d72f-407a-a361-9ac7ed55178f-metrics-certs\") pod \"network-metrics-daemon-8bxt7\" (UID: \"c8a1a07e-d72f-407a-a361-9ac7ed55178f\") " pod="openshift-multus/network-metrics-daemon-8bxt7"
Jun 02 15:45:55.393838 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:55.393759    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:45:55.393838 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:55.393822    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8a1a07e-d72f-407a-a361-9ac7ed55178f-metrics-certs podName:c8a1a07e-d72f-407a-a361-9ac7ed55178f nodeName:}" failed. No retries permitted until 2026-06-02 15:45:57.393804003 +0000 UTC m=+6.096267505 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c8a1a07e-d72f-407a-a361-9ac7ed55178f-metrics-certs") pod "network-metrics-daemon-8bxt7" (UID: "c8a1a07e-d72f-407a-a361-9ac7ed55178f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:45:55.494854 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:55.494815    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mmjq2\" (UniqueName: \"kubernetes.io/projected/a664bf1a-e1bf-4107-9ebb-076ddd6b6234-kube-api-access-mmjq2\") pod \"network-check-target-smdm5\" (UID: \"a664bf1a-e1bf-4107-9ebb-076ddd6b6234\") " pod="openshift-network-diagnostics/network-check-target-smdm5"
Jun 02 15:45:55.495073 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:55.495045    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 02 15:45:55.495073 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:55.495064    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 02 15:45:55.495191 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:55.495076    2578 projected.go:194] Error preparing data for projected volume kube-api-access-mmjq2 for pod openshift-network-diagnostics/network-check-target-smdm5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:45:55.495191 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:55.495134    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a664bf1a-e1bf-4107-9ebb-076ddd6b6234-kube-api-access-mmjq2 podName:a664bf1a-e1bf-4107-9ebb-076ddd6b6234 nodeName:}" failed. No retries permitted until 2026-06-02 15:45:57.495116092 +0000 UTC m=+6.197579595 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mmjq2" (UniqueName: "kubernetes.io/projected/a664bf1a-e1bf-4107-9ebb-076ddd6b6234-kube-api-access-mmjq2") pod "network-check-target-smdm5" (UID: "a664bf1a-e1bf-4107-9ebb-076ddd6b6234") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:45:55.904383 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:55.903890    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bxt7"
Jun 02 15:45:55.904383 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:55.904053    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-smdm5"
Jun 02 15:45:55.904383 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:55.904152    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-smdm5" podUID="a664bf1a-e1bf-4107-9ebb-076ddd6b6234"
Jun 02 15:45:55.904383 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:55.904050    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bxt7" podUID="c8a1a07e-d72f-407a-a361-9ac7ed55178f"
Jun 02 15:45:56.018735 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:56.018643    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.ec2.internal" event={"ID":"f52a050f57e8707b036e889aacbf12b0","Type":"ContainerStarted","Data":"08c808f9afd066edeeda63d7492b377ba25b8df850f8b476354b3a41e7656720"}
Jun 02 15:45:56.030952 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:56.030874    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-246.ec2.internal" podStartSLOduration=4.030853112 podStartE2EDuration="4.030853112s" podCreationTimestamp="2026-06-02 15:45:52 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 15:45:55.020500906 +0000 UTC m=+3.722964426" watchObservedRunningTime="2026-06-02 15:45:56.030853112 +0000 UTC m=+4.733316632"
Jun 02 15:45:56.031294 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:56.031264    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.ec2.internal" podStartSLOduration=4.03125205 podStartE2EDuration="4.03125205s" podCreationTimestamp="2026-06-02 15:45:52 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 15:45:56.031056098 +0000 UTC m=+4.733519616" watchObservedRunningTime="2026-06-02 15:45:56.03125205 +0000 UTC m=+4.733715570"
Jun 02 15:45:57.410754 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:57.410699    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8a1a07e-d72f-407a-a361-9ac7ed55178f-metrics-certs\") pod \"network-metrics-daemon-8bxt7\" (UID: \"c8a1a07e-d72f-407a-a361-9ac7ed55178f\") " pod="openshift-multus/network-metrics-daemon-8bxt7"
Jun 02 15:45:57.411255 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:57.410837    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:45:57.411255 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:57.410915    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8a1a07e-d72f-407a-a361-9ac7ed55178f-metrics-certs podName:c8a1a07e-d72f-407a-a361-9ac7ed55178f nodeName:}" failed. No retries permitted until 2026-06-02 15:46:01.4108931 +0000 UTC m=+10.113356598 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c8a1a07e-d72f-407a-a361-9ac7ed55178f-metrics-certs") pod "network-metrics-daemon-8bxt7" (UID: "c8a1a07e-d72f-407a-a361-9ac7ed55178f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:45:57.511780 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:57.511748    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mmjq2\" (UniqueName: \"kubernetes.io/projected/a664bf1a-e1bf-4107-9ebb-076ddd6b6234-kube-api-access-mmjq2\") pod \"network-check-target-smdm5\" (UID: \"a664bf1a-e1bf-4107-9ebb-076ddd6b6234\") " pod="openshift-network-diagnostics/network-check-target-smdm5"
Jun 02 15:45:57.511961 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:57.511882    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 02 15:45:57.511961 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:57.511900    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 02 15:45:57.511961 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:57.511912    2578 projected.go:194] Error preparing data for projected volume kube-api-access-mmjq2 for pod openshift-network-diagnostics/network-check-target-smdm5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:45:57.512189 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:57.511971    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a664bf1a-e1bf-4107-9ebb-076ddd6b6234-kube-api-access-mmjq2 podName:a664bf1a-e1bf-4107-9ebb-076ddd6b6234 nodeName:}" failed. No retries permitted until 2026-06-02 15:46:01.511952763 +0000 UTC m=+10.214416273 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mmjq2" (UniqueName: "kubernetes.io/projected/a664bf1a-e1bf-4107-9ebb-076ddd6b6234-kube-api-access-mmjq2") pod "network-check-target-smdm5" (UID: "a664bf1a-e1bf-4107-9ebb-076ddd6b6234") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:45:57.904049 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:57.904015    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bxt7"
Jun 02 15:45:57.904226 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:57.904034    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-smdm5"
Jun 02 15:45:57.904226 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:57.904184    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bxt7" podUID="c8a1a07e-d72f-407a-a361-9ac7ed55178f"
Jun 02 15:45:57.904346 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:57.904275    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-smdm5" podUID="a664bf1a-e1bf-4107-9ebb-076ddd6b6234"
Jun 02 15:45:59.904283 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:59.904176    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-smdm5"
Jun 02 15:45:59.904737 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:59.904303    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-smdm5" podUID="a664bf1a-e1bf-4107-9ebb-076ddd6b6234"
Jun 02 15:45:59.904737 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:45:59.904417    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bxt7"
Jun 02 15:45:59.904737 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:45:59.904535    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bxt7" podUID="c8a1a07e-d72f-407a-a361-9ac7ed55178f"
Jun 02 15:46:01.446827 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:01.446781    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8a1a07e-d72f-407a-a361-9ac7ed55178f-metrics-certs\") pod \"network-metrics-daemon-8bxt7\" (UID: \"c8a1a07e-d72f-407a-a361-9ac7ed55178f\") " pod="openshift-multus/network-metrics-daemon-8bxt7"
Jun 02 15:46:01.447338 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:01.446936    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:01.447338 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:01.447032    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8a1a07e-d72f-407a-a361-9ac7ed55178f-metrics-certs podName:c8a1a07e-d72f-407a-a361-9ac7ed55178f nodeName:}" failed. No retries permitted until 2026-06-02 15:46:09.447009026 +0000 UTC m=+18.149472538 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c8a1a07e-d72f-407a-a361-9ac7ed55178f-metrics-certs") pod "network-metrics-daemon-8bxt7" (UID: "c8a1a07e-d72f-407a-a361-9ac7ed55178f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:01.547658 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:01.547589    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mmjq2\" (UniqueName: \"kubernetes.io/projected/a664bf1a-e1bf-4107-9ebb-076ddd6b6234-kube-api-access-mmjq2\") pod \"network-check-target-smdm5\" (UID: \"a664bf1a-e1bf-4107-9ebb-076ddd6b6234\") " pod="openshift-network-diagnostics/network-check-target-smdm5"
Jun 02 15:46:01.547837 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:01.547740    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 02 15:46:01.547837 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:01.547771    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 02 15:46:01.547837 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:01.547783    2578 projected.go:194] Error preparing data for projected volume kube-api-access-mmjq2 for pod openshift-network-diagnostics/network-check-target-smdm5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:01.548004 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:01.547850    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a664bf1a-e1bf-4107-9ebb-076ddd6b6234-kube-api-access-mmjq2 podName:a664bf1a-e1bf-4107-9ebb-076ddd6b6234 nodeName:}" failed. No retries permitted until 2026-06-02 15:46:09.54782949 +0000 UTC m=+18.250292994 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mmjq2" (UniqueName: "kubernetes.io/projected/a664bf1a-e1bf-4107-9ebb-076ddd6b6234-kube-api-access-mmjq2") pod "network-check-target-smdm5" (UID: "a664bf1a-e1bf-4107-9ebb-076ddd6b6234") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:01.905006 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:01.904960    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bxt7"
Jun 02 15:46:01.905269 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:01.905101    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bxt7" podUID="c8a1a07e-d72f-407a-a361-9ac7ed55178f"
Jun 02 15:46:01.905572 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:01.905553    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-smdm5"
Jun 02 15:46:01.905665 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:01.905644    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-smdm5" podUID="a664bf1a-e1bf-4107-9ebb-076ddd6b6234"
Jun 02 15:46:03.904383 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:03.904343    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bxt7"
Jun 02 15:46:03.904806 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:03.904347    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-smdm5"
Jun 02 15:46:03.904806 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:03.904497    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bxt7" podUID="c8a1a07e-d72f-407a-a361-9ac7ed55178f"
Jun 02 15:46:03.904806 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:03.904621    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-smdm5" podUID="a664bf1a-e1bf-4107-9ebb-076ddd6b6234"
Jun 02 15:46:05.904335 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:05.904298    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bxt7"
Jun 02 15:46:05.904792 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:05.904441    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bxt7" podUID="c8a1a07e-d72f-407a-a361-9ac7ed55178f"
Jun 02 15:46:05.904792 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:05.904498    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-smdm5"
Jun 02 15:46:05.904792 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:05.904597    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-smdm5" podUID="a664bf1a-e1bf-4107-9ebb-076ddd6b6234"
Jun 02 15:46:07.903855 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:07.903819    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bxt7"
Jun 02 15:46:07.904342 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:07.903989    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bxt7" podUID="c8a1a07e-d72f-407a-a361-9ac7ed55178f"
Jun 02 15:46:07.904342 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:07.904050    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-smdm5"
Jun 02 15:46:07.904342 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:07.904154    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-smdm5" podUID="a664bf1a-e1bf-4107-9ebb-076ddd6b6234"
Jun 02 15:46:09.501955 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:09.501909    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8a1a07e-d72f-407a-a361-9ac7ed55178f-metrics-certs\") pod \"network-metrics-daemon-8bxt7\" (UID: \"c8a1a07e-d72f-407a-a361-9ac7ed55178f\") " pod="openshift-multus/network-metrics-daemon-8bxt7"
Jun 02 15:46:09.502500 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:09.502059    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:09.502500 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:09.502122    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8a1a07e-d72f-407a-a361-9ac7ed55178f-metrics-certs podName:c8a1a07e-d72f-407a-a361-9ac7ed55178f nodeName:}" failed. No retries permitted until 2026-06-02 15:46:25.502102614 +0000 UTC m=+34.204566114 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c8a1a07e-d72f-407a-a361-9ac7ed55178f-metrics-certs") pod "network-metrics-daemon-8bxt7" (UID: "c8a1a07e-d72f-407a-a361-9ac7ed55178f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:09.603263 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:09.603225    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mmjq2\" (UniqueName: \"kubernetes.io/projected/a664bf1a-e1bf-4107-9ebb-076ddd6b6234-kube-api-access-mmjq2\") pod \"network-check-target-smdm5\" (UID: \"a664bf1a-e1bf-4107-9ebb-076ddd6b6234\") " pod="openshift-network-diagnostics/network-check-target-smdm5"
Jun 02 15:46:09.603446 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:09.603373    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 02 15:46:09.603446 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:09.603387    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 02 15:46:09.603446 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:09.603395    2578 projected.go:194] Error preparing data for projected volume kube-api-access-mmjq2 for pod openshift-network-diagnostics/network-check-target-smdm5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:09.603446 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:09.603448    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a664bf1a-e1bf-4107-9ebb-076ddd6b6234-kube-api-access-mmjq2 podName:a664bf1a-e1bf-4107-9ebb-076ddd6b6234 nodeName:}" failed. No retries permitted until 2026-06-02 15:46:25.60343243 +0000 UTC m=+34.305895929 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mmjq2" (UniqueName: "kubernetes.io/projected/a664bf1a-e1bf-4107-9ebb-076ddd6b6234-kube-api-access-mmjq2") pod "network-check-target-smdm5" (UID: "a664bf1a-e1bf-4107-9ebb-076ddd6b6234") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:09.904103 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:09.904065    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bxt7"
Jun 02 15:46:09.904267 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:09.904108    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-smdm5"
Jun 02 15:46:09.904267 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:09.904200    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bxt7" podUID="c8a1a07e-d72f-407a-a361-9ac7ed55178f"
Jun 02 15:46:09.904377 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:09.904305    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-smdm5" podUID="a664bf1a-e1bf-4107-9ebb-076ddd6b6234"
Jun 02 15:46:11.904303 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:11.903977    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-smdm5"
Jun 02 15:46:11.905068 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:11.904049    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bxt7"
Jun 02 15:46:11.905068 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:11.904372    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-smdm5" podUID="a664bf1a-e1bf-4107-9ebb-076ddd6b6234"
Jun 02 15:46:11.905068 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:11.904451    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bxt7" podUID="c8a1a07e-d72f-407a-a361-9ac7ed55178f"
Jun 02 15:46:12.043755 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:12.043728    2578 generic.go:358] "Generic (PLEG): container finished" podID="4b0f9d3d-2024-423c-9afd-dc5094e82d2f" containerID="8b1afc2632ef5a54b4026f3f4d77f328474819d223678bd61d2f5516c374118f" exitCode=0
Jun 02 15:46:12.043865 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:12.043781    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wr9zh" event={"ID":"4b0f9d3d-2024-423c-9afd-dc5094e82d2f","Type":"ContainerDied","Data":"8b1afc2632ef5a54b4026f3f4d77f328474819d223678bd61d2f5516c374118f"}
Jun 02 15:46:12.046212 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:12.046192    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc" event={"ID":"becbbb00-675f-4263-a999-884ce0d688b2","Type":"ContainerStarted","Data":"52f7dbf270939720184c90cb885172b341b002233a476e66a67565c3167cfb9e"}
Jun 02 15:46:12.046212 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:12.046222    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc" event={"ID":"becbbb00-675f-4263-a999-884ce0d688b2","Type":"ContainerStarted","Data":"debca3031cb47941121a774197bd12d1cdd0495a6c56fca61cf5bde42c451009"}
Jun 02 15:46:12.046350 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:12.046232    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc" event={"ID":"becbbb00-675f-4263-a999-884ce0d688b2","Type":"ContainerStarted","Data":"d766dd20ffc9a29baa27240f8d2e46fe8468a1b77213d4ad8afbed8616c2a9f1"}
Jun 02 15:46:12.046350 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:12.046244    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc" event={"ID":"becbbb00-675f-4263-a999-884ce0d688b2","Type":"ContainerStarted","Data":"975b5564c61a3e4c73e72a1928587232e4046c0429b5576670a0e7393031cbb2"}
Jun 02 15:46:12.046350 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:12.046255    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc" event={"ID":"becbbb00-675f-4263-a999-884ce0d688b2","Type":"ContainerStarted","Data":"35daa2e4c3bed74d9a34a3a93cd083448e722fe95b0a7be3cd11758162ccf933"}
Jun 02 15:46:12.047637 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:12.047617    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mrkbz" event={"ID":"c81f2766-91c6-479a-bfcb-fc1144278194","Type":"ContainerStarted","Data":"f55a403690cd35b4c951dda90c78e25e71c3fca07602f630ab714266a6c96bee"}
Jun 02 15:46:12.048981 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:12.048956    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-csg6c" event={"ID":"d15a2da2-4507-43f8-913a-0414ef633f40","Type":"ContainerStarted","Data":"a5e4ed06e90b7a354d215649bfbee8be5b6d7d316ac0b2204184e573e082fb8e"}
Jun 02 15:46:12.050188 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:12.050160    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nnqmn" event={"ID":"22d16389-ab0a-4b82-97ae-e548268c773e","Type":"ContainerStarted","Data":"c4f393c2efafa248bacc145bb51138140c3d3e7ea793b006c08b09e20583aa88"}
Jun 02 15:46:12.051297 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:12.051276    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-89hj9" event={"ID":"6d6a55c1-8ee3-4dae-862e-1715d8d050d3","Type":"ContainerStarted","Data":"0a029e29639d8f207644ffed5eaae70708421f4350c9ed8672ea63be29e455e7"}
Jun 02 15:46:12.052416 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:12.052399    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-572sg" event={"ID":"05212ac5-a6ea-4c85-9a26-c518523c3fee","Type":"ContainerStarted","Data":"0bc746c88fab2d8a7cb59dd71f6bf5794b59df67b8f8f3091e4172f5f280a901"}
Jun 02 15:46:12.080552 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:12.080518    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mrkbz" podStartSLOduration=8.416262517 podStartE2EDuration="21.08050593s" podCreationTimestamp="2026-06-02 15:45:51 +0000 UTC" firstStartedPulling="2026-06-02 15:45:54.300946355 +0000 UTC m=+3.003409855" lastFinishedPulling="2026-06-02 15:46:06.965189771 +0000 UTC m=+15.667653268" observedRunningTime="2026-06-02 15:46:12.080478279 +0000 UTC m=+20.782941797" watchObservedRunningTime="2026-06-02 15:46:12.08050593 +0000 UTC m=+20.782969447"
Jun 02 15:46:12.094902 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:12.094863    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-csg6c" podStartSLOduration=2.866195505 podStartE2EDuration="20.094851065s" podCreationTimestamp="2026-06-02 15:45:52 +0000 UTC" firstStartedPulling="2026-06-02 15:45:54.298274265 +0000 UTC m=+3.000737766" lastFinishedPulling="2026-06-02 15:46:11.526929829 +0000 UTC m=+20.229393326" observedRunningTime="2026-06-02 15:46:12.094711771 +0000 UTC m=+20.797175274" watchObservedRunningTime="2026-06-02 15:46:12.094851065 +0000 UTC m=+20.797314582"
Jun 02 15:46:12.112228 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:12.112155    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-nnqmn" podStartSLOduration=2.825987241 podStartE2EDuration="20.112144175s" podCreationTimestamp="2026-06-02 15:45:52 +0000 UTC" firstStartedPulling="2026-06-02 15:45:54.302040967 +0000 UTC m=+3.004504478" lastFinishedPulling="2026-06-02 15:46:11.588197905 +0000 UTC m=+20.290661412" observedRunningTime="2026-06-02 15:46:12.111899807 +0000 UTC m=+20.814363347" watchObservedRunningTime="2026-06-02 15:46:12.112144175 +0000 UTC m=+20.814607692"
Jun 02 15:46:12.128062 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:12.127988    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-89hj9" podStartSLOduration=2.896072324 podStartE2EDuration="20.12797128s" podCreationTimestamp="2026-06-02 15:45:52 +0000 UTC" firstStartedPulling="2026-06-02 15:45:54.296481495 +0000 UTC m=+2.998945009" lastFinishedPulling="2026-06-02 15:46:11.528380466 +0000 UTC m=+20.230843965" observedRunningTime="2026-06-02 15:46:12.127518981 +0000 UTC m=+20.829982498" watchObservedRunningTime="2026-06-02 15:46:12.12797128 +0000 UTC m=+20.830434803"
Jun 02 15:46:12.630076 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:12.630044    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jun 02 15:46:12.831945 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:12.831845    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-06-02T15:46:12.630066631Z","UUID":"6f2b56ef-bfa4-41bd-8e6b-5b85d4c1882a","Handler":null,"Name":"","Endpoint":""}
Jun 02 15:46:12.833441 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:12.833422    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jun 02 15:46:12.833557 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:12.833450    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jun 02 15:46:13.057734 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:13.057694    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc" event={"ID":"becbbb00-675f-4263-a999-884ce0d688b2","Type":"ContainerStarted","Data":"d5e08f3e13df3ae75cc6128246ffcd082bdc602a85efc275bf7627d0142d6188"}
Jun 02 15:46:13.059123 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:13.059091    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2pmfl" event={"ID":"47c31aee-a2ce-4b2a-808d-eca33a9c31fc","Type":"ContainerStarted","Data":"0c9fbd820431ff01348bcfd239119358849cf6b15ca2fa1d74fd224a971945a1"}
Jun 02 15:46:13.060897 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:13.060781    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-572sg" event={"ID":"05212ac5-a6ea-4c85-9a26-c518523c3fee","Type":"ContainerStarted","Data":"39fdb870d0ff234a28877aa6bcb209e277d539fe08b1f328cb973bfebf6ecf3f"}
Jun 02 15:46:13.075069 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:13.075028    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2pmfl" podStartSLOduration=3.8518268879999997 podStartE2EDuration="21.075015447s" podCreationTimestamp="2026-06-02 15:45:52 +0000 UTC" firstStartedPulling="2026-06-02 15:45:54.303674861 +0000 UTC m=+3.006138357" lastFinishedPulling="2026-06-02 15:46:11.526863402 +0000 UTC m=+20.229326916" observedRunningTime="2026-06-02 15:46:13.074670084 +0000 UTC m=+21.777133603" watchObservedRunningTime="2026-06-02 15:46:13.075015447 +0000 UTC m=+21.777478966"
Jun 02 15:46:13.903806 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:13.903774    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-smdm5"
Jun 02 15:46:13.904018 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:13.903895    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-smdm5" podUID="a664bf1a-e1bf-4107-9ebb-076ddd6b6234"
Jun 02 15:46:13.904018 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:13.903930    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bxt7"
Jun 02 15:46:13.907773 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:13.904427    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bxt7" podUID="c8a1a07e-d72f-407a-a361-9ac7ed55178f"
Jun 02 15:46:14.065241 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:14.065146    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-572sg" event={"ID":"05212ac5-a6ea-4c85-9a26-c518523c3fee","Type":"ContainerStarted","Data":"7af2a3e64270ba4b07bd08ea6a4cb62b4b73e8bcfb4db51a741e7af3d35f2c9f"}
Jun 02 15:46:14.084765 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:14.084701    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-572sg" podStartSLOduration=3.650687721 podStartE2EDuration="23.084682418s" podCreationTimestamp="2026-06-02 15:45:51 +0000 UTC" firstStartedPulling="2026-06-02 15:45:54.294774419 +0000 UTC m=+2.997237919" lastFinishedPulling="2026-06-02 15:46:13.728769116 +0000 UTC m=+22.431232616" observedRunningTime="2026-06-02 15:46:14.084263921 +0000 UTC m=+22.786727449" watchObservedRunningTime="2026-06-02 15:46:14.084682418 +0000 UTC m=+22.787145937"
Jun 02 15:46:14.760814 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:14.760773    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mrkbz"
Jun 02 15:46:14.761381 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:14.761358    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mrkbz"
Jun 02 15:46:15.070949 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:15.070868    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc" event={"ID":"becbbb00-675f-4263-a999-884ce0d688b2","Type":"ContainerStarted","Data":"2498c5f010ce903d2bde944ed489badd205386bd7ad0ff9f5d51e7092bac1035"}
Jun 02 15:46:15.071470 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:15.071141    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mrkbz"
Jun 02 15:46:15.071830 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:15.071814    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mrkbz"
Jun 02 15:46:15.111798 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:15.111765    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-4tfzk"]
Jun 02 15:46:15.118242 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:15.118219    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4tfzk"
Jun 02 15:46:15.120169 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:15.120148    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jun 02 15:46:15.120299 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:15.120151    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jun 02 15:46:15.120402 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:15.120382    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9v94l\""
Jun 02 15:46:15.248896 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:15.248860    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zggkx\" (UniqueName: \"kubernetes.io/projected/97fe90a6-eb72-4661-b376-fba7f85330e3-kube-api-access-zggkx\") pod \"node-resolver-4tfzk\" (UID: \"97fe90a6-eb72-4661-b376-fba7f85330e3\") " pod="openshift-dns/node-resolver-4tfzk"
Jun 02 15:46:15.249110 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:15.248906    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97fe90a6-eb72-4661-b376-fba7f85330e3-tmp-dir\") pod \"node-resolver-4tfzk\" (UID: \"97fe90a6-eb72-4661-b376-fba7f85330e3\") " pod="openshift-dns/node-resolver-4tfzk"
Jun 02 15:46:15.249110 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:15.248957    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/97fe90a6-eb72-4661-b376-fba7f85330e3-hosts-file\") pod \"node-resolver-4tfzk\" (UID: \"97fe90a6-eb72-4661-b376-fba7f85330e3\") " pod="openshift-dns/node-resolver-4tfzk"
Jun 02 15:46:15.350307 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:15.350221    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/97fe90a6-eb72-4661-b376-fba7f85330e3-hosts-file\") pod \"node-resolver-4tfzk\" (UID: \"97fe90a6-eb72-4661-b376-fba7f85330e3\") " pod="openshift-dns/node-resolver-4tfzk"
Jun 02 15:46:15.350307 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:15.350293    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zggkx\" (UniqueName: \"kubernetes.io/projected/97fe90a6-eb72-4661-b376-fba7f85330e3-kube-api-access-zggkx\") pod \"node-resolver-4tfzk\" (UID: \"97fe90a6-eb72-4661-b376-fba7f85330e3\") " pod="openshift-dns/node-resolver-4tfzk"
Jun 02 15:46:15.350522 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:15.350328    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97fe90a6-eb72-4661-b376-fba7f85330e3-tmp-dir\") pod \"node-resolver-4tfzk\" (UID: \"97fe90a6-eb72-4661-b376-fba7f85330e3\") " pod="openshift-dns/node-resolver-4tfzk"
Jun 02 15:46:15.350522 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:15.350346    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/97fe90a6-eb72-4661-b376-fba7f85330e3-hosts-file\") pod \"node-resolver-4tfzk\" (UID: \"97fe90a6-eb72-4661-b376-fba7f85330e3\") " pod="openshift-dns/node-resolver-4tfzk"
Jun 02 15:46:15.350716 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:15.350679    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97fe90a6-eb72-4661-b376-fba7f85330e3-tmp-dir\") pod \"node-resolver-4tfzk\" (UID: \"97fe90a6-eb72-4661-b376-fba7f85330e3\") " pod="openshift-dns/node-resolver-4tfzk"
Jun 02 15:46:15.365743 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:15.365711    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zggkx\" (UniqueName: \"kubernetes.io/projected/97fe90a6-eb72-4661-b376-fba7f85330e3-kube-api-access-zggkx\") pod \"node-resolver-4tfzk\" (UID: \"97fe90a6-eb72-4661-b376-fba7f85330e3\") " pod="openshift-dns/node-resolver-4tfzk"
Jun 02 15:46:15.428789 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:15.428741    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4tfzk"
Jun 02 15:46:15.903766 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:15.903564    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bxt7"
Jun 02 15:46:15.903954 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:15.903875    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bxt7" podUID="c8a1a07e-d72f-407a-a361-9ac7ed55178f"
Jun 02 15:46:15.904136 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:15.904112    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-smdm5"
Jun 02 15:46:15.904241 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:15.904216    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-smdm5" podUID="a664bf1a-e1bf-4107-9ebb-076ddd6b6234"
Jun 02 15:46:16.111847 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:46:16.111810    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97fe90a6_eb72_4661_b376_fba7f85330e3.slice/crio-8e1189ee9bd755000d6a65da9323ec169eb60802e039099a315af5a8100884d5 WatchSource:0}: Error finding container 8e1189ee9bd755000d6a65da9323ec169eb60802e039099a315af5a8100884d5: Status 404 returned error can't find the container with id 8e1189ee9bd755000d6a65da9323ec169eb60802e039099a315af5a8100884d5
Jun 02 15:46:17.076398 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:17.076211    2578 generic.go:358] "Generic (PLEG): container finished" podID="4b0f9d3d-2024-423c-9afd-dc5094e82d2f" containerID="7df8fd48fbfef591b69af82b6055431c3b36676cecfbcfad0324442f9a5eb980" exitCode=0
Jun 02 15:46:17.076602 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:17.076302    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wr9zh" event={"ID":"4b0f9d3d-2024-423c-9afd-dc5094e82d2f","Type":"ContainerDied","Data":"7df8fd48fbfef591b69af82b6055431c3b36676cecfbcfad0324442f9a5eb980"}
Jun 02 15:46:17.079457 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:17.079432    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc" event={"ID":"becbbb00-675f-4263-a999-884ce0d688b2","Type":"ContainerStarted","Data":"120739d43d1d2b2bc8ed09b4356287a40e11248bc5e4f995befeace8a815b340"}
Jun 02 15:46:17.079709 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:17.079684    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:46:17.079709 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:17.079711    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:46:17.080532 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:17.080511    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4tfzk" event={"ID":"97fe90a6-eb72-4661-b376-fba7f85330e3","Type":"ContainerStarted","Data":"2a08db12702e49de09ce8721aa9848a4bd008cc7db024ea04e82886dac078630"}
Jun 02 15:46:17.080618 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:17.080535    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4tfzk" event={"ID":"97fe90a6-eb72-4661-b376-fba7f85330e3","Type":"ContainerStarted","Data":"8e1189ee9bd755000d6a65da9323ec169eb60802e039099a315af5a8100884d5"}
Jun 02 15:46:17.095032 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:17.095007    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:46:17.127719 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:17.127666    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc" podStartSLOduration=8.853404559 podStartE2EDuration="26.127651013s" podCreationTimestamp="2026-06-02 15:45:51 +0000 UTC" firstStartedPulling="2026-06-02 15:45:54.303645661 +0000 UTC m=+3.006109171" lastFinishedPulling="2026-06-02 15:46:11.577892116 +0000 UTC m=+20.280355625" observedRunningTime="2026-06-02 15:46:17.127366075 +0000 UTC m=+25.829829594" watchObservedRunningTime="2026-06-02 15:46:17.127651013 +0000 UTC m=+25.830114531"
Jun 02 15:46:17.142009 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:17.141950    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-4tfzk" podStartSLOduration=2.141937311 podStartE2EDuration="2.141937311s" podCreationTimestamp="2026-06-02 15:46:15 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 15:46:17.141901677 +0000 UTC m=+25.844365206" watchObservedRunningTime="2026-06-02 15:46:17.141937311 +0000 UTC m=+25.844400828"
Jun 02 15:46:17.904353 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:17.904321    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-smdm5"
Jun 02 15:46:17.904511 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:17.904332    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bxt7"
Jun 02 15:46:17.904511 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:17.904433    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-smdm5" podUID="a664bf1a-e1bf-4107-9ebb-076ddd6b6234"
Jun 02 15:46:17.904609 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:17.904544    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bxt7" podUID="c8a1a07e-d72f-407a-a361-9ac7ed55178f"
Jun 02 15:46:18.085351 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:18.085319    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:46:18.102432 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:18.102401    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:46:18.542803 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:18.542728    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-smdm5"]
Jun 02 15:46:18.543295 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:18.542847    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-smdm5"
Jun 02 15:46:18.543295 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:18.542921    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-smdm5" podUID="a664bf1a-e1bf-4107-9ebb-076ddd6b6234"
Jun 02 15:46:18.559296 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:18.559264    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8bxt7"]
Jun 02 15:46:18.559424 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:18.559390    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bxt7"
Jun 02 15:46:18.559525 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:18.559499    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bxt7" podUID="c8a1a07e-d72f-407a-a361-9ac7ed55178f"
Jun 02 15:46:19.085239 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:19.085197    2578 generic.go:358] "Generic (PLEG): container finished" podID="4b0f9d3d-2024-423c-9afd-dc5094e82d2f" containerID="61a9c436985ba9508a36db693686edb38698c745d983d22e2096aac21dbf8b8d" exitCode=0
Jun 02 15:46:19.085393 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:19.085270    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wr9zh" event={"ID":"4b0f9d3d-2024-423c-9afd-dc5094e82d2f","Type":"ContainerDied","Data":"61a9c436985ba9508a36db693686edb38698c745d983d22e2096aac21dbf8b8d"}
Jun 02 15:46:19.906604 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:19.906571    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-smdm5"
Jun 02 15:46:19.907056 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:19.906687    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-smdm5" podUID="a664bf1a-e1bf-4107-9ebb-076ddd6b6234"
Jun 02 15:46:20.089440 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:20.089406    2578 generic.go:358] "Generic (PLEG): container finished" podID="4b0f9d3d-2024-423c-9afd-dc5094e82d2f" containerID="f38b2c914d32bcc9160bc015b05a6c2a0761690ce69c462ef51c098234751818" exitCode=0
Jun 02 15:46:20.089604 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:20.089494    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wr9zh" event={"ID":"4b0f9d3d-2024-423c-9afd-dc5094e82d2f","Type":"ContainerDied","Data":"f38b2c914d32bcc9160bc015b05a6c2a0761690ce69c462ef51c098234751818"}
Jun 02 15:46:20.903950 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:20.903915    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bxt7"
Jun 02 15:46:20.904130 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:20.904065    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bxt7" podUID="c8a1a07e-d72f-407a-a361-9ac7ed55178f"
Jun 02 15:46:21.907867 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:21.907684    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-smdm5"
Jun 02 15:46:21.908268 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:21.907967    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-smdm5" podUID="a664bf1a-e1bf-4107-9ebb-076ddd6b6234"
Jun 02 15:46:22.904079 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:22.904038    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bxt7"
Jun 02 15:46:22.904367 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:22.904170    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bxt7" podUID="c8a1a07e-d72f-407a-a361-9ac7ed55178f"
Jun 02 15:46:23.907205 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:23.907168    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-smdm5"
Jun 02 15:46:23.907671 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:23.907291    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-smdm5" podUID="a664bf1a-e1bf-4107-9ebb-076ddd6b6234"
Jun 02 15:46:24.143153 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.143119    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-246.ec2.internal" event="NodeReady"
Jun 02 15:46:24.143332 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.143264    2578 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
Jun 02 15:46:24.184371 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.184277    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7749958bb7-2xtcw"]
Jun 02 15:46:24.217318 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.217280    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5pxm2"]
Jun 02 15:46:24.217642 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.217617    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7749958bb7-2xtcw"
Jun 02 15:46:24.220632 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.220345    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jun 02 15:46:24.220632 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.220367    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jun 02 15:46:24.220632 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.220396    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jun 02 15:46:24.220632 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.220368    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rvmhh\""
Jun 02 15:46:24.225476 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.225448    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jun 02 15:46:24.231222 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.231200    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-l7cml"]
Jun 02 15:46:24.231388 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.231373    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5pxm2"
Jun 02 15:46:24.233776 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.233755    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jun 02 15:46:24.233879 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.233763    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cpsk2\""
Jun 02 15:46:24.233879 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.233796    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jun 02 15:46:24.249394 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.249360    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7749958bb7-2xtcw"]
Jun 02 15:46:24.249394 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.249396    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5pxm2"]
Jun 02 15:46:24.249570 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.249409    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-l7cml"]
Jun 02 15:46:24.249570 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.249409    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-l7cml"
Jun 02 15:46:24.251751 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.251730    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jun 02 15:46:24.251866 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.251747    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jun 02 15:46:24.251866 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.251748    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jun 02 15:46:24.252059 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.252035    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-s8ml8\""
Jun 02 15:46:24.314715 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.314671    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2d910de6-b7ee-49ae-b0f8-2948726ae27f-config-volume\") pod \"dns-default-5pxm2\" (UID: \"2d910de6-b7ee-49ae-b0f8-2948726ae27f\") " pod="openshift-dns/dns-default-5pxm2"
Jun 02 15:46:24.314891 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.314725    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2c23e202-3ecd-4fff-ac9b-1b633ee32679-trusted-ca\") pod \"image-registry-7749958bb7-2xtcw\" (UID: \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\") " pod="openshift-image-registry/image-registry-7749958bb7-2xtcw"
Jun 02 15:46:24.314891 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.314753    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m8t7\" (UniqueName: \"kubernetes.io/projected/2d910de6-b7ee-49ae-b0f8-2948726ae27f-kube-api-access-4m8t7\") pod \"dns-default-5pxm2\" (UID: \"2d910de6-b7ee-49ae-b0f8-2948726ae27f\") " pod="openshift-dns/dns-default-5pxm2"
Jun 02 15:46:24.314891 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.314780    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w2rv\" (UniqueName: \"kubernetes.io/projected/2c23e202-3ecd-4fff-ac9b-1b633ee32679-kube-api-access-2w2rv\") pod \"image-registry-7749958bb7-2xtcw\" (UID: \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\") " pod="openshift-image-registry/image-registry-7749958bb7-2xtcw"
Jun 02 15:46:24.314891 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.314810    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2c23e202-3ecd-4fff-ac9b-1b633ee32679-image-registry-private-configuration\") pod \"image-registry-7749958bb7-2xtcw\" (UID: \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\") " pod="openshift-image-registry/image-registry-7749958bb7-2xtcw"
Jun 02 15:46:24.314891 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.314832    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2c23e202-3ecd-4fff-ac9b-1b633ee32679-registry-certificates\") pod \"image-registry-7749958bb7-2xtcw\" (UID: \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\") " pod="openshift-image-registry/image-registry-7749958bb7-2xtcw"
Jun 02 15:46:24.314891 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.314852    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2c23e202-3ecd-4fff-ac9b-1b633ee32679-installation-pull-secrets\") pod \"image-registry-7749958bb7-2xtcw\" (UID: \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\") " pod="openshift-image-registry/image-registry-7749958bb7-2xtcw"
Jun 02 15:46:24.314891 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.314876    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c23e202-3ecd-4fff-ac9b-1b633ee32679-bound-sa-token\") pod \"image-registry-7749958bb7-2xtcw\" (UID: \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\") " pod="openshift-image-registry/image-registry-7749958bb7-2xtcw"
Jun 02 15:46:24.315201 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.314927    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d910de6-b7ee-49ae-b0f8-2948726ae27f-metrics-tls\") pod \"dns-default-5pxm2\" (UID: \"2d910de6-b7ee-49ae-b0f8-2948726ae27f\") " pod="openshift-dns/dns-default-5pxm2"
Jun 02 15:46:24.315201 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.314949    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d910de6-b7ee-49ae-b0f8-2948726ae27f-tmp-dir\") pod \"dns-default-5pxm2\" (UID: \"2d910de6-b7ee-49ae-b0f8-2948726ae27f\") " pod="openshift-dns/dns-default-5pxm2"
Jun 02 15:46:24.315201 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.315024    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2c23e202-3ecd-4fff-ac9b-1b633ee32679-registry-tls\") pod \"image-registry-7749958bb7-2xtcw\" (UID: \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\") " pod="openshift-image-registry/image-registry-7749958bb7-2xtcw"
Jun 02 15:46:24.315201 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.315064    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2c23e202-3ecd-4fff-ac9b-1b633ee32679-ca-trust-extracted\") pod \"image-registry-7749958bb7-2xtcw\" (UID: \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\") " pod="openshift-image-registry/image-registry-7749958bb7-2xtcw"
Jun 02 15:46:24.416055 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.416015    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2c23e202-3ecd-4fff-ac9b-1b633ee32679-registry-tls\") pod \"image-registry-7749958bb7-2xtcw\" (UID: \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\") " pod="openshift-image-registry/image-registry-7749958bb7-2xtcw"
Jun 02 15:46:24.416055 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.416065    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2l2r\" (UniqueName: \"kubernetes.io/projected/8601b504-c1ef-44de-853b-9d130969f742-kube-api-access-t2l2r\") pod \"ingress-canary-l7cml\" (UID: \"8601b504-c1ef-44de-853b-9d130969f742\") " pod="openshift-ingress-canary/ingress-canary-l7cml"
Jun 02 15:46:24.416306 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.416083    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8601b504-c1ef-44de-853b-9d130969f742-cert\") pod \"ingress-canary-l7cml\" (UID: \"8601b504-c1ef-44de-853b-9d130969f742\") " pod="openshift-ingress-canary/ingress-canary-l7cml"
Jun 02 15:46:24.416306 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.416112    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2c23e202-3ecd-4fff-ac9b-1b633ee32679-ca-trust-extracted\") pod \"image-registry-7749958bb7-2xtcw\" (UID: \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\") " pod="openshift-image-registry/image-registry-7749958bb7-2xtcw"
Jun 02 15:46:24.416306 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.416134    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2d910de6-b7ee-49ae-b0f8-2948726ae27f-config-volume\") pod \"dns-default-5pxm2\" (UID: \"2d910de6-b7ee-49ae-b0f8-2948726ae27f\") " pod="openshift-dns/dns-default-5pxm2"
Jun 02 15:46:24.416306 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.416159    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2c23e202-3ecd-4fff-ac9b-1b633ee32679-trusted-ca\") pod \"image-registry-7749958bb7-2xtcw\" (UID: \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\") " pod="openshift-image-registry/image-registry-7749958bb7-2xtcw"
Jun 02 15:46:24.416306 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:24.416182    2578 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jun 02 15:46:24.416306 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:24.416209    2578 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7749958bb7-2xtcw: secret "image-registry-tls" not found
Jun 02 15:46:24.416306 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:24.416273    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c23e202-3ecd-4fff-ac9b-1b633ee32679-registry-tls podName:2c23e202-3ecd-4fff-ac9b-1b633ee32679 nodeName:}" failed. No retries permitted until 2026-06-02 15:46:24.916242038 +0000 UTC m=+33.618705533 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2c23e202-3ecd-4fff-ac9b-1b633ee32679-registry-tls") pod "image-registry-7749958bb7-2xtcw" (UID: "2c23e202-3ecd-4fff-ac9b-1b633ee32679") : secret "image-registry-tls" not found
Jun 02 15:46:24.416306 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.416186    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4m8t7\" (UniqueName: \"kubernetes.io/projected/2d910de6-b7ee-49ae-b0f8-2948726ae27f-kube-api-access-4m8t7\") pod \"dns-default-5pxm2\" (UID: \"2d910de6-b7ee-49ae-b0f8-2948726ae27f\") " pod="openshift-dns/dns-default-5pxm2"
Jun 02 15:46:24.416718 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.416327    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2w2rv\" (UniqueName: \"kubernetes.io/projected/2c23e202-3ecd-4fff-ac9b-1b633ee32679-kube-api-access-2w2rv\") pod \"image-registry-7749958bb7-2xtcw\" (UID: \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\") " pod="openshift-image-registry/image-registry-7749958bb7-2xtcw"
Jun 02 15:46:24.416718 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.416365    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2c23e202-3ecd-4fff-ac9b-1b633ee32679-image-registry-private-configuration\") pod \"image-registry-7749958bb7-2xtcw\" (UID: \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\") " pod="openshift-image-registry/image-registry-7749958bb7-2xtcw"
Jun 02 15:46:24.416718 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.416400    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2c23e202-3ecd-4fff-ac9b-1b633ee32679-registry-certificates\") pod \"image-registry-7749958bb7-2xtcw\" (UID: \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\") " pod="openshift-image-registry/image-registry-7749958bb7-2xtcw"
Jun 02 15:46:24.416718 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.416427    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2c23e202-3ecd-4fff-ac9b-1b633ee32679-installation-pull-secrets\") pod \"image-registry-7749958bb7-2xtcw\" (UID: \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\") " pod="openshift-image-registry/image-registry-7749958bb7-2xtcw"
Jun 02 15:46:24.416718 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.416451    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c23e202-3ecd-4fff-ac9b-1b633ee32679-bound-sa-token\") pod \"image-registry-7749958bb7-2xtcw\" (UID: \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\") " pod="openshift-image-registry/image-registry-7749958bb7-2xtcw"
Jun 02 15:46:24.416718 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.416499    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d910de6-b7ee-49ae-b0f8-2948726ae27f-metrics-tls\") pod \"dns-default-5pxm2\" (UID: \"2d910de6-b7ee-49ae-b0f8-2948726ae27f\") " pod="openshift-dns/dns-default-5pxm2"
Jun 02 15:46:24.416718 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.416522    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d910de6-b7ee-49ae-b0f8-2948726ae27f-tmp-dir\") pod \"dns-default-5pxm2\" (UID: \"2d910de6-b7ee-49ae-b0f8-2948726ae27f\") " pod="openshift-dns/dns-default-5pxm2"
Jun 02 15:46:24.416718 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.416633    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2c23e202-3ecd-4fff-ac9b-1b633ee32679-ca-trust-extracted\") pod \"image-registry-7749958bb7-2xtcw\" (UID: \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\") " pod="openshift-image-registry/image-registry-7749958bb7-2xtcw"
Jun 02 15:46:24.417121 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.416759    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2d910de6-b7ee-49ae-b0f8-2948726ae27f-config-volume\") pod \"dns-default-5pxm2\" (UID: \"2d910de6-b7ee-49ae-b0f8-2948726ae27f\") " pod="openshift-dns/dns-default-5pxm2"
Jun 02 15:46:24.417121 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.416807    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d910de6-b7ee-49ae-b0f8-2948726ae27f-tmp-dir\") pod \"dns-default-5pxm2\" (UID: \"2d910de6-b7ee-49ae-b0f8-2948726ae27f\") " pod="openshift-dns/dns-default-5pxm2"
Jun 02 15:46:24.417121 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:24.416894    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:46:24.417121 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:24.416938    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d910de6-b7ee-49ae-b0f8-2948726ae27f-metrics-tls podName:2d910de6-b7ee-49ae-b0f8-2948726ae27f nodeName:}" failed. No retries permitted until 2026-06-02 15:46:24.916921971 +0000 UTC m=+33.619385486 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2d910de6-b7ee-49ae-b0f8-2948726ae27f-metrics-tls") pod "dns-default-5pxm2" (UID: "2d910de6-b7ee-49ae-b0f8-2948726ae27f") : secret "dns-default-metrics-tls" not found
Jun 02 15:46:24.417390 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.417370    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2c23e202-3ecd-4fff-ac9b-1b633ee32679-trusted-ca\") pod \"image-registry-7749958bb7-2xtcw\" (UID: \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\") " pod="openshift-image-registry/image-registry-7749958bb7-2xtcw"
Jun 02 15:46:24.417806 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.417775    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2c23e202-3ecd-4fff-ac9b-1b633ee32679-registry-certificates\") pod \"image-registry-7749958bb7-2xtcw\" (UID: \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\") " pod="openshift-image-registry/image-registry-7749958bb7-2xtcw"
Jun 02 15:46:24.421408 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.421362    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2c23e202-3ecd-4fff-ac9b-1b633ee32679-installation-pull-secrets\") pod \"image-registry-7749958bb7-2xtcw\" (UID: \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\") " pod="openshift-image-registry/image-registry-7749958bb7-2xtcw"
Jun 02 15:46:24.421408 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.421362    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2c23e202-3ecd-4fff-ac9b-1b633ee32679-image-registry-private-configuration\") pod \"image-registry-7749958bb7-2xtcw\" (UID: \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\") " pod="openshift-image-registry/image-registry-7749958bb7-2xtcw"
Jun 02 15:46:24.425822 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.425782    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c23e202-3ecd-4fff-ac9b-1b633ee32679-bound-sa-token\") pod \"image-registry-7749958bb7-2xtcw\" (UID: \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\") " pod="openshift-image-registry/image-registry-7749958bb7-2xtcw"
Jun 02 15:46:24.426019 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.425865    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w2rv\" (UniqueName: \"kubernetes.io/projected/2c23e202-3ecd-4fff-ac9b-1b633ee32679-kube-api-access-2w2rv\") pod \"image-registry-7749958bb7-2xtcw\" (UID: \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\") " pod="openshift-image-registry/image-registry-7749958bb7-2xtcw"
Jun 02 15:46:24.426246 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.426221    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m8t7\" (UniqueName: \"kubernetes.io/projected/2d910de6-b7ee-49ae-b0f8-2948726ae27f-kube-api-access-4m8t7\") pod \"dns-default-5pxm2\" (UID: \"2d910de6-b7ee-49ae-b0f8-2948726ae27f\") " pod="openshift-dns/dns-default-5pxm2"
Jun 02 15:46:24.517612 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.517512    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t2l2r\" (UniqueName: \"kubernetes.io/projected/8601b504-c1ef-44de-853b-9d130969f742-kube-api-access-t2l2r\") pod \"ingress-canary-l7cml\" (UID: \"8601b504-c1ef-44de-853b-9d130969f742\") " pod="openshift-ingress-canary/ingress-canary-l7cml"
Jun 02 15:46:24.517612 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.517566    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8601b504-c1ef-44de-853b-9d130969f742-cert\") pod \"ingress-canary-l7cml\" (UID: \"8601b504-c1ef-44de-853b-9d130969f742\") " pod="openshift-ingress-canary/ingress-canary-l7cml"
Jun 02 15:46:24.517842 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:24.517675    2578 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:46:24.517842 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:24.517752    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8601b504-c1ef-44de-853b-9d130969f742-cert podName:8601b504-c1ef-44de-853b-9d130969f742 nodeName:}" failed. No retries permitted until 2026-06-02 15:46:25.017729294 +0000 UTC m=+33.720192809 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8601b504-c1ef-44de-853b-9d130969f742-cert") pod "ingress-canary-l7cml" (UID: "8601b504-c1ef-44de-853b-9d130969f742") : secret "canary-serving-cert" not found
Jun 02 15:46:24.527722 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.527691    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2l2r\" (UniqueName: \"kubernetes.io/projected/8601b504-c1ef-44de-853b-9d130969f742-kube-api-access-t2l2r\") pod \"ingress-canary-l7cml\" (UID: \"8601b504-c1ef-44de-853b-9d130969f742\") " pod="openshift-ingress-canary/ingress-canary-l7cml"
Jun 02 15:46:24.904062 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.904024    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bxt7"
Jun 02 15:46:24.907348 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.907321    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jun 02 15:46:24.907879 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.907323    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wnhfh\""
Jun 02 15:46:24.921798 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.921762    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d910de6-b7ee-49ae-b0f8-2948726ae27f-metrics-tls\") pod \"dns-default-5pxm2\" (UID: \"2d910de6-b7ee-49ae-b0f8-2948726ae27f\") " pod="openshift-dns/dns-default-5pxm2"
Jun 02 15:46:24.921798 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:24.921811    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2c23e202-3ecd-4fff-ac9b-1b633ee32679-registry-tls\") pod \"image-registry-7749958bb7-2xtcw\" (UID: \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\") " pod="openshift-image-registry/image-registry-7749958bb7-2xtcw"
Jun 02 15:46:24.922014 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:24.921917    2578 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jun 02 15:46:24.922014 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:24.921932    2578 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7749958bb7-2xtcw: secret "image-registry-tls" not found
Jun 02 15:46:24.922014 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:24.921947    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:46:24.922014 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:24.922012    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c23e202-3ecd-4fff-ac9b-1b633ee32679-registry-tls podName:2c23e202-3ecd-4fff-ac9b-1b633ee32679 nodeName:}" failed. No retries permitted until 2026-06-02 15:46:25.921979221 +0000 UTC m=+34.624442718 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2c23e202-3ecd-4fff-ac9b-1b633ee32679-registry-tls") pod "image-registry-7749958bb7-2xtcw" (UID: "2c23e202-3ecd-4fff-ac9b-1b633ee32679") : secret "image-registry-tls" not found
Jun 02 15:46:24.922201 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:24.922040    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d910de6-b7ee-49ae-b0f8-2948726ae27f-metrics-tls podName:2d910de6-b7ee-49ae-b0f8-2948726ae27f nodeName:}" failed. No retries permitted until 2026-06-02 15:46:25.92201932 +0000 UTC m=+34.624482821 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2d910de6-b7ee-49ae-b0f8-2948726ae27f-metrics-tls") pod "dns-default-5pxm2" (UID: "2d910de6-b7ee-49ae-b0f8-2948726ae27f") : secret "dns-default-metrics-tls" not found
Jun 02 15:46:25.023177 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:25.023140    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8601b504-c1ef-44de-853b-9d130969f742-cert\") pod \"ingress-canary-l7cml\" (UID: \"8601b504-c1ef-44de-853b-9d130969f742\") " pod="openshift-ingress-canary/ingress-canary-l7cml"
Jun 02 15:46:25.023359 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:25.023287    2578 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:46:25.023422 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:25.023369    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8601b504-c1ef-44de-853b-9d130969f742-cert podName:8601b504-c1ef-44de-853b-9d130969f742 nodeName:}" failed. No retries permitted until 2026-06-02 15:46:26.023346451 +0000 UTC m=+34.725809952 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8601b504-c1ef-44de-853b-9d130969f742-cert") pod "ingress-canary-l7cml" (UID: "8601b504-c1ef-44de-853b-9d130969f742") : secret "canary-serving-cert" not found
Jun 02 15:46:25.528521 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:25.528487    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8a1a07e-d72f-407a-a361-9ac7ed55178f-metrics-certs\") pod \"network-metrics-daemon-8bxt7\" (UID: \"c8a1a07e-d72f-407a-a361-9ac7ed55178f\") " pod="openshift-multus/network-metrics-daemon-8bxt7"
Jun 02 15:46:25.528674 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:25.528640    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jun 02 15:46:25.528729 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:25.528702    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8a1a07e-d72f-407a-a361-9ac7ed55178f-metrics-certs podName:c8a1a07e-d72f-407a-a361-9ac7ed55178f nodeName:}" failed. No retries permitted until 2026-06-02 15:46:57.528683821 +0000 UTC m=+66.231147334 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c8a1a07e-d72f-407a-a361-9ac7ed55178f-metrics-certs") pod "network-metrics-daemon-8bxt7" (UID: "c8a1a07e-d72f-407a-a361-9ac7ed55178f") : secret "metrics-daemon-secret" not found
Jun 02 15:46:25.629692 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:25.629656    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mmjq2\" (UniqueName: \"kubernetes.io/projected/a664bf1a-e1bf-4107-9ebb-076ddd6b6234-kube-api-access-mmjq2\") pod \"network-check-target-smdm5\" (UID: \"a664bf1a-e1bf-4107-9ebb-076ddd6b6234\") " pod="openshift-network-diagnostics/network-check-target-smdm5"
Jun 02 15:46:25.629868 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:25.629818    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 02 15:46:25.629868 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:25.629843    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 02 15:46:25.629868 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:25.629853    2578 projected.go:194] Error preparing data for projected volume kube-api-access-mmjq2 for pod openshift-network-diagnostics/network-check-target-smdm5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:25.629973 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:25.629905    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a664bf1a-e1bf-4107-9ebb-076ddd6b6234-kube-api-access-mmjq2 podName:a664bf1a-e1bf-4107-9ebb-076ddd6b6234 nodeName:}" failed. No retries permitted until 2026-06-02 15:46:57.629891161 +0000 UTC m=+66.332354662 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mmjq2" (UniqueName: "kubernetes.io/projected/a664bf1a-e1bf-4107-9ebb-076ddd6b6234-kube-api-access-mmjq2") pod "network-check-target-smdm5" (UID: "a664bf1a-e1bf-4107-9ebb-076ddd6b6234") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:25.904334 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:25.904308    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-smdm5"
Jun 02 15:46:25.906954 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:25.906930    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jun 02 15:46:25.907272 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:25.907255    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jun 02 15:46:25.907366 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:25.907262    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4lmvm\""
Jun 02 15:46:25.931747 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:25.931716    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d910de6-b7ee-49ae-b0f8-2948726ae27f-metrics-tls\") pod \"dns-default-5pxm2\" (UID: \"2d910de6-b7ee-49ae-b0f8-2948726ae27f\") " pod="openshift-dns/dns-default-5pxm2"
Jun 02 15:46:25.931881 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:25.931771    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2c23e202-3ecd-4fff-ac9b-1b633ee32679-registry-tls\") pod \"image-registry-7749958bb7-2xtcw\" (UID: \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\") " pod="openshift-image-registry/image-registry-7749958bb7-2xtcw"
Jun 02 15:46:25.931958 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:25.931939    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:46:25.932149 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:25.932125    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d910de6-b7ee-49ae-b0f8-2948726ae27f-metrics-tls podName:2d910de6-b7ee-49ae-b0f8-2948726ae27f nodeName:}" failed. No retries permitted until 2026-06-02 15:46:27.932082125 +0000 UTC m=+36.634545636 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2d910de6-b7ee-49ae-b0f8-2948726ae27f-metrics-tls") pod "dns-default-5pxm2" (UID: "2d910de6-b7ee-49ae-b0f8-2948726ae27f") : secret "dns-default-metrics-tls" not found
Jun 02 15:46:25.932252 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:25.932162    2578 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jun 02 15:46:25.932252 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:25.932181    2578 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7749958bb7-2xtcw: secret "image-registry-tls" not found
Jun 02 15:46:25.932252 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:25.932236    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c23e202-3ecd-4fff-ac9b-1b633ee32679-registry-tls podName:2c23e202-3ecd-4fff-ac9b-1b633ee32679 nodeName:}" failed. No retries permitted until 2026-06-02 15:46:27.932219606 +0000 UTC m=+36.634683107 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2c23e202-3ecd-4fff-ac9b-1b633ee32679-registry-tls") pod "image-registry-7749958bb7-2xtcw" (UID: "2c23e202-3ecd-4fff-ac9b-1b633ee32679") : secret "image-registry-tls" not found
Jun 02 15:46:26.032536 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:26.032508    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8601b504-c1ef-44de-853b-9d130969f742-cert\") pod \"ingress-canary-l7cml\" (UID: \"8601b504-c1ef-44de-853b-9d130969f742\") " pod="openshift-ingress-canary/ingress-canary-l7cml"
Jun 02 15:46:26.032684 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:26.032618    2578 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:46:26.032684 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:26.032667    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8601b504-c1ef-44de-853b-9d130969f742-cert podName:8601b504-c1ef-44de-853b-9d130969f742 nodeName:}" failed. No retries permitted until 2026-06-02 15:46:28.032648101 +0000 UTC m=+36.735111597 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8601b504-c1ef-44de-853b-9d130969f742-cert") pod "ingress-canary-l7cml" (UID: "8601b504-c1ef-44de-853b-9d130969f742") : secret "canary-serving-cert" not found
Jun 02 15:46:26.103880 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:26.103851    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wr9zh" event={"ID":"4b0f9d3d-2024-423c-9afd-dc5094e82d2f","Type":"ContainerStarted","Data":"8ad0f06d04181859634259db3ec567fb3ce20c44d5350892149b99f7c53c5ee1"}
Jun 02 15:46:27.107755 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:27.107718    2578 generic.go:358] "Generic (PLEG): container finished" podID="4b0f9d3d-2024-423c-9afd-dc5094e82d2f" containerID="8ad0f06d04181859634259db3ec567fb3ce20c44d5350892149b99f7c53c5ee1" exitCode=0
Jun 02 15:46:27.108305 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:27.107766    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wr9zh" event={"ID":"4b0f9d3d-2024-423c-9afd-dc5094e82d2f","Type":"ContainerDied","Data":"8ad0f06d04181859634259db3ec567fb3ce20c44d5350892149b99f7c53c5ee1"}
Jun 02 15:46:27.948091 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:27.948058    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d910de6-b7ee-49ae-b0f8-2948726ae27f-metrics-tls\") pod \"dns-default-5pxm2\" (UID: \"2d910de6-b7ee-49ae-b0f8-2948726ae27f\") " pod="openshift-dns/dns-default-5pxm2"
Jun 02 15:46:27.948249 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:27.948105    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2c23e202-3ecd-4fff-ac9b-1b633ee32679-registry-tls\") pod \"image-registry-7749958bb7-2xtcw\" (UID: \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\") " pod="openshift-image-registry/image-registry-7749958bb7-2xtcw"
Jun 02 15:46:27.948249 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:27.948209    2578 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jun 02 15:46:27.948249 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:27.948211    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:46:27.948360 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:27.948275    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d910de6-b7ee-49ae-b0f8-2948726ae27f-metrics-tls podName:2d910de6-b7ee-49ae-b0f8-2948726ae27f nodeName:}" failed. No retries permitted until 2026-06-02 15:46:31.948257341 +0000 UTC m=+40.650720837 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2d910de6-b7ee-49ae-b0f8-2948726ae27f-metrics-tls") pod "dns-default-5pxm2" (UID: "2d910de6-b7ee-49ae-b0f8-2948726ae27f") : secret "dns-default-metrics-tls" not found
Jun 02 15:46:27.948360 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:27.948220    2578 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7749958bb7-2xtcw: secret "image-registry-tls" not found
Jun 02 15:46:27.948360 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:27.948329    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c23e202-3ecd-4fff-ac9b-1b633ee32679-registry-tls podName:2c23e202-3ecd-4fff-ac9b-1b633ee32679 nodeName:}" failed. No retries permitted until 2026-06-02 15:46:31.948317661 +0000 UTC m=+40.650781157 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2c23e202-3ecd-4fff-ac9b-1b633ee32679-registry-tls") pod "image-registry-7749958bb7-2xtcw" (UID: "2c23e202-3ecd-4fff-ac9b-1b633ee32679") : secret "image-registry-tls" not found
Jun 02 15:46:28.049191 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:28.049152    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8601b504-c1ef-44de-853b-9d130969f742-cert\") pod \"ingress-canary-l7cml\" (UID: \"8601b504-c1ef-44de-853b-9d130969f742\") " pod="openshift-ingress-canary/ingress-canary-l7cml"
Jun 02 15:46:28.049343 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:28.049326    2578 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:46:28.049396 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:28.049386    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8601b504-c1ef-44de-853b-9d130969f742-cert podName:8601b504-c1ef-44de-853b-9d130969f742 nodeName:}" failed. No retries permitted until 2026-06-02 15:46:32.049368332 +0000 UTC m=+40.751831839 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8601b504-c1ef-44de-853b-9d130969f742-cert") pod "ingress-canary-l7cml" (UID: "8601b504-c1ef-44de-853b-9d130969f742") : secret "canary-serving-cert" not found
Jun 02 15:46:28.111851 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:28.111821    2578 generic.go:358] "Generic (PLEG): container finished" podID="4b0f9d3d-2024-423c-9afd-dc5094e82d2f" containerID="02ff10d4d1aa6d3196eda7806059812dfd5abf7515261f94d8dd74d11f77f20f" exitCode=0
Jun 02 15:46:28.112302 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:28.111873    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wr9zh" event={"ID":"4b0f9d3d-2024-423c-9afd-dc5094e82d2f","Type":"ContainerDied","Data":"02ff10d4d1aa6d3196eda7806059812dfd5abf7515261f94d8dd74d11f77f20f"}
Jun 02 15:46:29.117238 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:29.116951    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wr9zh" event={"ID":"4b0f9d3d-2024-423c-9afd-dc5094e82d2f","Type":"ContainerStarted","Data":"3ab5e7d99c943dcc58b5efb25d72601bd4c5020c617bb2f51308846461bc6749"}
Jun 02 15:46:29.140765 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:29.140623    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wr9zh" podStartSLOduration=5.558007678 podStartE2EDuration="37.140605534s" podCreationTimestamp="2026-06-02 15:45:52 +0000 UTC" firstStartedPulling="2026-06-02 15:45:54.305131974 +0000 UTC m=+3.007595473" lastFinishedPulling="2026-06-02 15:46:25.887729824 +0000 UTC m=+34.590193329" observedRunningTime="2026-06-02 15:46:29.139884232 +0000 UTC m=+37.842347750" watchObservedRunningTime="2026-06-02 15:46:29.140605534 +0000 UTC m=+37.843069053"
Jun 02 15:46:31.978924 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:31.978887    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2c23e202-3ecd-4fff-ac9b-1b633ee32679-registry-tls\") pod \"image-registry-7749958bb7-2xtcw\" (UID: \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\") " pod="openshift-image-registry/image-registry-7749958bb7-2xtcw"
Jun 02 15:46:31.979302 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:31.979016    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d910de6-b7ee-49ae-b0f8-2948726ae27f-metrics-tls\") pod \"dns-default-5pxm2\" (UID: \"2d910de6-b7ee-49ae-b0f8-2948726ae27f\") " pod="openshift-dns/dns-default-5pxm2"
Jun 02 15:46:31.979302 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:31.979063    2578 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jun 02 15:46:31.979302 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:31.979081    2578 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7749958bb7-2xtcw: secret "image-registry-tls" not found
Jun 02 15:46:31.979302 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:31.979107    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:46:31.979302 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:31.979163    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c23e202-3ecd-4fff-ac9b-1b633ee32679-registry-tls podName:2c23e202-3ecd-4fff-ac9b-1b633ee32679 nodeName:}" failed. No retries permitted until 2026-06-02 15:46:39.9791464 +0000 UTC m=+48.681609908 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2c23e202-3ecd-4fff-ac9b-1b633ee32679-registry-tls") pod "image-registry-7749958bb7-2xtcw" (UID: "2c23e202-3ecd-4fff-ac9b-1b633ee32679") : secret "image-registry-tls" not found
Jun 02 15:46:31.979302 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:31.979178    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d910de6-b7ee-49ae-b0f8-2948726ae27f-metrics-tls podName:2d910de6-b7ee-49ae-b0f8-2948726ae27f nodeName:}" failed. No retries permitted until 2026-06-02 15:46:39.979173059 +0000 UTC m=+48.681636555 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2d910de6-b7ee-49ae-b0f8-2948726ae27f-metrics-tls") pod "dns-default-5pxm2" (UID: "2d910de6-b7ee-49ae-b0f8-2948726ae27f") : secret "dns-default-metrics-tls" not found
Jun 02 15:46:32.079873 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:32.079840    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8601b504-c1ef-44de-853b-9d130969f742-cert\") pod \"ingress-canary-l7cml\" (UID: \"8601b504-c1ef-44de-853b-9d130969f742\") " pod="openshift-ingress-canary/ingress-canary-l7cml"
Jun 02 15:46:32.080067 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:32.079957    2578 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:46:32.080067 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:32.080027    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8601b504-c1ef-44de-853b-9d130969f742-cert podName:8601b504-c1ef-44de-853b-9d130969f742 nodeName:}" failed. No retries permitted until 2026-06-02 15:46:40.080010997 +0000 UTC m=+48.782474509 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8601b504-c1ef-44de-853b-9d130969f742-cert") pod "ingress-canary-l7cml" (UID: "8601b504-c1ef-44de-853b-9d130969f742") : secret "canary-serving-cert" not found
Jun 02 15:46:36.147179 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:36.147151    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-685df94d57-j5dw8"]
Jun 02 15:46:36.187510 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:36.187469    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-685df94d57-j5dw8"]
Jun 02 15:46:36.187661 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:36.187599    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-685df94d57-j5dw8"
Jun 02 15:46:36.190433 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:36.190414    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jun 02 15:46:36.190433 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:36.190428    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jun 02 15:46:36.190601 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:36.190432    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jun 02 15:46:36.190877 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:36.190860    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jun 02 15:46:36.190970 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:36.190888    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-bbhj7\""
Jun 02 15:46:36.208648 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:36.208620    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9cf51e90-bb6a-48e6-b9ee-acb8c9d2145b-signing-cabundle\") pod \"service-ca-685df94d57-j5dw8\" (UID: \"9cf51e90-bb6a-48e6-b9ee-acb8c9d2145b\") " pod="openshift-service-ca/service-ca-685df94d57-j5dw8"
Jun 02 15:46:36.208772 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:36.208653    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9cf51e90-bb6a-48e6-b9ee-acb8c9d2145b-signing-key\") pod \"service-ca-685df94d57-j5dw8\" (UID: \"9cf51e90-bb6a-48e6-b9ee-acb8c9d2145b\") " pod="openshift-service-ca/service-ca-685df94d57-j5dw8"
Jun 02 15:46:36.208772 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:36.208677    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqsbt\" (UniqueName: \"kubernetes.io/projected/9cf51e90-bb6a-48e6-b9ee-acb8c9d2145b-kube-api-access-wqsbt\") pod \"service-ca-685df94d57-j5dw8\" (UID: \"9cf51e90-bb6a-48e6-b9ee-acb8c9d2145b\") " pod="openshift-service-ca/service-ca-685df94d57-j5dw8"
Jun 02 15:46:36.309258 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:36.309216    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9cf51e90-bb6a-48e6-b9ee-acb8c9d2145b-signing-cabundle\") pod \"service-ca-685df94d57-j5dw8\" (UID: \"9cf51e90-bb6a-48e6-b9ee-acb8c9d2145b\") " pod="openshift-service-ca/service-ca-685df94d57-j5dw8"
Jun 02 15:46:36.309258 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:36.309261    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9cf51e90-bb6a-48e6-b9ee-acb8c9d2145b-signing-key\") pod \"service-ca-685df94d57-j5dw8\" (UID: \"9cf51e90-bb6a-48e6-b9ee-acb8c9d2145b\") " pod="openshift-service-ca/service-ca-685df94d57-j5dw8"
Jun 02 15:46:36.309462 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:36.309290    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wqsbt\" (UniqueName: \"kubernetes.io/projected/9cf51e90-bb6a-48e6-b9ee-acb8c9d2145b-kube-api-access-wqsbt\") pod \"service-ca-685df94d57-j5dw8\" (UID: \"9cf51e90-bb6a-48e6-b9ee-acb8c9d2145b\") " pod="openshift-service-ca/service-ca-685df94d57-j5dw8"
Jun 02 15:46:36.309986 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:36.309966    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9cf51e90-bb6a-48e6-b9ee-acb8c9d2145b-signing-cabundle\") pod \"service-ca-685df94d57-j5dw8\" (UID: \"9cf51e90-bb6a-48e6-b9ee-acb8c9d2145b\") " pod="openshift-service-ca/service-ca-685df94d57-j5dw8"
Jun 02 15:46:36.312182 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:36.312156    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9cf51e90-bb6a-48e6-b9ee-acb8c9d2145b-signing-key\") pod \"service-ca-685df94d57-j5dw8\" (UID: \"9cf51e90-bb6a-48e6-b9ee-acb8c9d2145b\") " pod="openshift-service-ca/service-ca-685df94d57-j5dw8"
Jun 02 15:46:36.318175 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:36.318150    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqsbt\" (UniqueName: \"kubernetes.io/projected/9cf51e90-bb6a-48e6-b9ee-acb8c9d2145b-kube-api-access-wqsbt\") pod \"service-ca-685df94d57-j5dw8\" (UID: \"9cf51e90-bb6a-48e6-b9ee-acb8c9d2145b\") " pod="openshift-service-ca/service-ca-685df94d57-j5dw8"
Jun 02 15:46:36.351252 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:36.351230    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4tfzk_97fe90a6-eb72-4661-b376-fba7f85330e3/dns-node-resolver/0.log"
Jun 02 15:46:36.497897 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:36.497808    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-685df94d57-j5dw8"
Jun 02 15:46:36.656048 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:36.655980    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-685df94d57-j5dw8"]
Jun 02 15:46:36.659151 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:46:36.659127    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cf51e90_bb6a_48e6_b9ee_acb8c9d2145b.slice/crio-b3f6b96cfbe2fc069e1703fa53af0b8a5825a4d3d4ec77dae27807102d71b933 WatchSource:0}: Error finding container b3f6b96cfbe2fc069e1703fa53af0b8a5825a4d3d4ec77dae27807102d71b933: Status 404 returned error can't find the container with id b3f6b96cfbe2fc069e1703fa53af0b8a5825a4d3d4ec77dae27807102d71b933
Jun 02 15:46:37.132200 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:37.132167    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-685df94d57-j5dw8" event={"ID":"9cf51e90-bb6a-48e6-b9ee-acb8c9d2145b","Type":"ContainerStarted","Data":"b3f6b96cfbe2fc069e1703fa53af0b8a5825a4d3d4ec77dae27807102d71b933"}
Jun 02 15:46:37.351403 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:37.351375    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-csg6c_d15a2da2-4507-43f8-913a-0414ef633f40/node-ca/0.log"
Jun 02 15:46:40.039455 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:40.039411    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d910de6-b7ee-49ae-b0f8-2948726ae27f-metrics-tls\") pod \"dns-default-5pxm2\" (UID: \"2d910de6-b7ee-49ae-b0f8-2948726ae27f\") " pod="openshift-dns/dns-default-5pxm2"
Jun 02 15:46:40.039455 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:40.039464    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2c23e202-3ecd-4fff-ac9b-1b633ee32679-registry-tls\") pod \"image-registry-7749958bb7-2xtcw\" (UID: \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\") " pod="openshift-image-registry/image-registry-7749958bb7-2xtcw"
Jun 02 15:46:40.039874 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:40.039564    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:46:40.039874 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:40.039631    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d910de6-b7ee-49ae-b0f8-2948726ae27f-metrics-tls podName:2d910de6-b7ee-49ae-b0f8-2948726ae27f nodeName:}" failed. No retries permitted until 2026-06-02 15:46:56.039614053 +0000 UTC m=+64.742077550 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2d910de6-b7ee-49ae-b0f8-2948726ae27f-metrics-tls") pod "dns-default-5pxm2" (UID: "2d910de6-b7ee-49ae-b0f8-2948726ae27f") : secret "dns-default-metrics-tls" not found
Jun 02 15:46:40.039874 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:40.039570    2578 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jun 02 15:46:40.039874 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:40.039669    2578 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7749958bb7-2xtcw: secret "image-registry-tls" not found
Jun 02 15:46:40.039874 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:40.039714    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c23e202-3ecd-4fff-ac9b-1b633ee32679-registry-tls podName:2c23e202-3ecd-4fff-ac9b-1b633ee32679 nodeName:}" failed. No retries permitted until 2026-06-02 15:46:56.039702662 +0000 UTC m=+64.742166158 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2c23e202-3ecd-4fff-ac9b-1b633ee32679-registry-tls") pod "image-registry-7749958bb7-2xtcw" (UID: "2c23e202-3ecd-4fff-ac9b-1b633ee32679") : secret "image-registry-tls" not found
Jun 02 15:46:40.139574 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:40.139537    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-685df94d57-j5dw8" event={"ID":"9cf51e90-bb6a-48e6-b9ee-acb8c9d2145b","Type":"ContainerStarted","Data":"c9652970fc2ecbf6ed4aea8a981bf9a2060757320c5e19c59b6672f8fd37d236"}
Jun 02 15:46:40.139813 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:40.139797    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8601b504-c1ef-44de-853b-9d130969f742-cert\") pod \"ingress-canary-l7cml\" (UID: \"8601b504-c1ef-44de-853b-9d130969f742\") " pod="openshift-ingress-canary/ingress-canary-l7cml"
Jun 02 15:46:40.139935 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:40.139923    2578 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:46:40.139983 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:46:40.139973    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8601b504-c1ef-44de-853b-9d130969f742-cert podName:8601b504-c1ef-44de-853b-9d130969f742 nodeName:}" failed. No retries permitted until 2026-06-02 15:46:56.139957631 +0000 UTC m=+64.842421128 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8601b504-c1ef-44de-853b-9d130969f742-cert") pod "ingress-canary-l7cml" (UID: "8601b504-c1ef-44de-853b-9d130969f742") : secret "canary-serving-cert" not found
Jun 02 15:46:50.099947 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:50.099919    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l5gxc"
Jun 02 15:46:50.131454 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:50.131400    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca/service-ca-685df94d57-j5dw8" podStartSLOduration=11.569838644 podStartE2EDuration="14.1313868s" podCreationTimestamp="2026-06-02 15:46:36 +0000 UTC" firstStartedPulling="2026-06-02 15:46:36.660911159 +0000 UTC m=+45.363374655" lastFinishedPulling="2026-06-02 15:46:39.222459313 +0000 UTC m=+47.924922811" observedRunningTime="2026-06-02 15:46:40.159672057 +0000 UTC m=+48.862135574" watchObservedRunningTime="2026-06-02 15:46:50.1313868 +0000 UTC m=+58.833850318"
Jun 02 15:46:56.059921 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:56.059876    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d910de6-b7ee-49ae-b0f8-2948726ae27f-metrics-tls\") pod \"dns-default-5pxm2\" (UID: \"2d910de6-b7ee-49ae-b0f8-2948726ae27f\") " pod="openshift-dns/dns-default-5pxm2"
Jun 02 15:46:56.059921 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:56.059927    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2c23e202-3ecd-4fff-ac9b-1b633ee32679-registry-tls\") pod \"image-registry-7749958bb7-2xtcw\" (UID: \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\") " pod="openshift-image-registry/image-registry-7749958bb7-2xtcw"
Jun 02 15:46:56.062181 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:56.062152    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d910de6-b7ee-49ae-b0f8-2948726ae27f-metrics-tls\") pod \"dns-default-5pxm2\" (UID: \"2d910de6-b7ee-49ae-b0f8-2948726ae27f\") " pod="openshift-dns/dns-default-5pxm2"
Jun 02 15:46:56.062329 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:56.062305    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2c23e202-3ecd-4fff-ac9b-1b633ee32679-registry-tls\") pod \"image-registry-7749958bb7-2xtcw\" (UID: \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\") " pod="openshift-image-registry/image-registry-7749958bb7-2xtcw"
Jun 02 15:46:56.160597 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:56.160563    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8601b504-c1ef-44de-853b-9d130969f742-cert\") pod \"ingress-canary-l7cml\" (UID: \"8601b504-c1ef-44de-853b-9d130969f742\") " pod="openshift-ingress-canary/ingress-canary-l7cml"
Jun 02 15:46:56.162815 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:56.162798    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8601b504-c1ef-44de-853b-9d130969f742-cert\") pod \"ingress-canary-l7cml\" (UID: \"8601b504-c1ef-44de-853b-9d130969f742\") " pod="openshift-ingress-canary/ingress-canary-l7cml"
Jun 02 15:46:56.331811 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:56.331728    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rvmhh\""
Jun 02 15:46:56.339679 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:56.339659    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7749958bb7-2xtcw"
Jun 02 15:46:56.342539 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:56.342524    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cpsk2\""
Jun 02 15:46:56.351284 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:56.351260    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5pxm2"
Jun 02 15:46:56.362256 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:56.362235    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-s8ml8\""
Jun 02 15:46:56.370875 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:56.370852    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-l7cml"
Jun 02 15:46:56.503691 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:56.503629    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7749958bb7-2xtcw"]
Jun 02 15:46:56.506951 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:46:56.506915    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c23e202_3ecd_4fff_ac9b_1b633ee32679.slice/crio-8fb00773eaa9ee0c881bd9af46cbc854f64d01b2427f395deaf57148a14ba468 WatchSource:0}: Error finding container 8fb00773eaa9ee0c881bd9af46cbc854f64d01b2427f395deaf57148a14ba468: Status 404 returned error can't find the container with id 8fb00773eaa9ee0c881bd9af46cbc854f64d01b2427f395deaf57148a14ba468
Jun 02 15:46:56.514337 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:56.514314    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5pxm2"]
Jun 02 15:46:56.517174 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:46:56.517148    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d910de6_b7ee_49ae_b0f8_2948726ae27f.slice/crio-e8b27349130e417d033af3dccf18a9ec6ffc8ba435f72c2e2e76d34073d83463 WatchSource:0}: Error finding container e8b27349130e417d033af3dccf18a9ec6ffc8ba435f72c2e2e76d34073d83463: Status 404 returned error can't find the container with id e8b27349130e417d033af3dccf18a9ec6ffc8ba435f72c2e2e76d34073d83463
Jun 02 15:46:56.538784 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:56.538754    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-l7cml"]
Jun 02 15:46:56.545392 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:46:56.545369    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8601b504_c1ef_44de_853b_9d130969f742.slice/crio-9202a57f9fbfb56cace800b7e85e3afd4c172afb45ae4cdc1b04cc4eb539d51d WatchSource:0}: Error finding container 9202a57f9fbfb56cace800b7e85e3afd4c172afb45ae4cdc1b04cc4eb539d51d: Status 404 returned error can't find the container with id 9202a57f9fbfb56cace800b7e85e3afd4c172afb45ae4cdc1b04cc4eb539d51d
Jun 02 15:46:57.173743 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:57.173705    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5pxm2" event={"ID":"2d910de6-b7ee-49ae-b0f8-2948726ae27f","Type":"ContainerStarted","Data":"e8b27349130e417d033af3dccf18a9ec6ffc8ba435f72c2e2e76d34073d83463"}
Jun 02 15:46:57.174955 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:57.174928    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-l7cml" event={"ID":"8601b504-c1ef-44de-853b-9d130969f742","Type":"ContainerStarted","Data":"9202a57f9fbfb56cace800b7e85e3afd4c172afb45ae4cdc1b04cc4eb539d51d"}
Jun 02 15:46:57.176404 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:57.176382    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7749958bb7-2xtcw" event={"ID":"2c23e202-3ecd-4fff-ac9b-1b633ee32679","Type":"ContainerStarted","Data":"203da634b2be04cc053da9ce6aefbe879242997e1e410db12b1c5ca460a14760"}
Jun 02 15:46:57.176404 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:57.176409    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7749958bb7-2xtcw" event={"ID":"2c23e202-3ecd-4fff-ac9b-1b633ee32679","Type":"ContainerStarted","Data":"8fb00773eaa9ee0c881bd9af46cbc854f64d01b2427f395deaf57148a14ba468"}
Jun 02 15:46:57.176565 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:57.176481    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7749958bb7-2xtcw"
Jun 02 15:46:57.571845 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:57.571762    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8a1a07e-d72f-407a-a361-9ac7ed55178f-metrics-certs\") pod \"network-metrics-daemon-8bxt7\" (UID: \"c8a1a07e-d72f-407a-a361-9ac7ed55178f\") " pod="openshift-multus/network-metrics-daemon-8bxt7"
Jun 02 15:46:57.574538 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:57.574512    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8a1a07e-d72f-407a-a361-9ac7ed55178f-metrics-certs\") pod \"network-metrics-daemon-8bxt7\" (UID: \"c8a1a07e-d72f-407a-a361-9ac7ed55178f\") " pod="openshift-multus/network-metrics-daemon-8bxt7"
Jun 02 15:46:57.619779 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:57.619750    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wnhfh\""
Jun 02 15:46:57.625529 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:57.625501    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bxt7"
Jun 02 15:46:57.672618 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:57.672582    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mmjq2\" (UniqueName: \"kubernetes.io/projected/a664bf1a-e1bf-4107-9ebb-076ddd6b6234-kube-api-access-mmjq2\") pod \"network-check-target-smdm5\" (UID: \"a664bf1a-e1bf-4107-9ebb-076ddd6b6234\") " pod="openshift-network-diagnostics/network-check-target-smdm5"
Jun 02 15:46:57.675473 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:57.675451    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jun 02 15:46:57.685506 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:57.685482    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jun 02 15:46:57.696326 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:57.696298    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmjq2\" (UniqueName: \"kubernetes.io/projected/a664bf1a-e1bf-4107-9ebb-076ddd6b6234-kube-api-access-mmjq2\") pod \"network-check-target-smdm5\" (UID: \"a664bf1a-e1bf-4107-9ebb-076ddd6b6234\") " pod="openshift-network-diagnostics/network-check-target-smdm5"
Jun 02 15:46:57.715141 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:57.715111    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4lmvm\""
Jun 02 15:46:57.723009 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:57.722970    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-smdm5"
Jun 02 15:46:58.593808 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:58.593141    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7749958bb7-2xtcw" podStartSLOduration=44.593120423 podStartE2EDuration="44.593120423s" podCreationTimestamp="2026-06-02 15:46:14 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 15:46:57.202036843 +0000 UTC m=+65.904500363" watchObservedRunningTime="2026-06-02 15:46:58.593120423 +0000 UTC m=+67.295583947"
Jun 02 15:46:58.593808 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:58.593393    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-smdm5"]
Jun 02 15:46:58.621561 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:58.621529    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8bxt7"]
Jun 02 15:46:58.626120 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:46:58.626091    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8a1a07e_d72f_407a_a361_9ac7ed55178f.slice/crio-861297d3af43b2650bf161a41d0c97f4fece3aba87e0120e7e689059e955eb69 WatchSource:0}: Error finding container 861297d3af43b2650bf161a41d0c97f4fece3aba87e0120e7e689059e955eb69: Status 404 returned error can't find the container with id 861297d3af43b2650bf161a41d0c97f4fece3aba87e0120e7e689059e955eb69
Jun 02 15:46:59.182771 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:59.182725    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5pxm2" event={"ID":"2d910de6-b7ee-49ae-b0f8-2948726ae27f","Type":"ContainerStarted","Data":"5fd18875a0d160ae44ed43591e1de928303066479addd3891d02c73d8b2cf8f7"}
Jun 02 15:46:59.182771 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:59.182777    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5pxm2" event={"ID":"2d910de6-b7ee-49ae-b0f8-2948726ae27f","Type":"ContainerStarted","Data":"48627d3c2135faa9027244e48b49a8e7d3d796c93e726bf32c68dc9aae5ef676"}
Jun 02 15:46:59.183080 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:59.182858    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5pxm2"
Jun 02 15:46:59.184341 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:59.184304    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8bxt7" event={"ID":"c8a1a07e-d72f-407a-a361-9ac7ed55178f","Type":"ContainerStarted","Data":"861297d3af43b2650bf161a41d0c97f4fece3aba87e0120e7e689059e955eb69"}
Jun 02 15:46:59.185731 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:59.185706    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-l7cml" event={"ID":"8601b504-c1ef-44de-853b-9d130969f742","Type":"ContainerStarted","Data":"e3491f7d75f8ecd1cae901d0e32a5da5e00fe622f106d9882254a8ba1fac65ca"}
Jun 02 15:46:59.186948 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:59.186927    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-smdm5" event={"ID":"a664bf1a-e1bf-4107-9ebb-076ddd6b6234","Type":"ContainerStarted","Data":"32ca0ebedfa175ab1bebd1c8e5e7f5ce059d58e510751663eadd34b6e8eac638"}
Jun 02 15:46:59.201549 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:59.201502    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-5pxm2" podStartSLOduration=33.260306137 podStartE2EDuration="35.201488821s" podCreationTimestamp="2026-06-02 15:46:24 +0000 UTC" firstStartedPulling="2026-06-02 15:46:56.519042525 +0000 UTC m=+65.221506021" lastFinishedPulling="2026-06-02 15:46:58.460225203 +0000 UTC m=+67.162688705" observedRunningTime="2026-06-02 15:46:59.200283421 +0000 UTC m=+67.902746940" watchObservedRunningTime="2026-06-02 15:46:59.201488821 +0000 UTC m=+67.903952338"
Jun 02 15:46:59.218230 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:46:59.218170    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-l7cml" podStartSLOduration=33.300339883 podStartE2EDuration="35.218152907s" podCreationTimestamp="2026-06-02 15:46:24 +0000 UTC" firstStartedPulling="2026-06-02 15:46:56.547073919 +0000 UTC m=+65.249537415" lastFinishedPulling="2026-06-02 15:46:58.464886862 +0000 UTC m=+67.167350439" observedRunningTime="2026-06-02 15:46:59.217458299 +0000 UTC m=+67.919921827" watchObservedRunningTime="2026-06-02 15:46:59.218152907 +0000 UTC m=+67.920616425"
Jun 02 15:47:00.192261 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:00.192216    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8bxt7" event={"ID":"c8a1a07e-d72f-407a-a361-9ac7ed55178f","Type":"ContainerStarted","Data":"42ab01e5c00b4cf3b0b8a4101dd305c30aef6cc06b36cdce9e4bbe7eaaeabfcd"}
Jun 02 15:47:00.192261 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:00.192265    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8bxt7" event={"ID":"c8a1a07e-d72f-407a-a361-9ac7ed55178f","Type":"ContainerStarted","Data":"9d43f1ce69b658c724b385a444c4ba6ac5ba6a9896d0016d8878f8a6c5fe188a"}
Jun 02 15:47:00.210844 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:00.210777    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8bxt7" podStartSLOduration=67.280790809 podStartE2EDuration="1m8.210757993s" podCreationTimestamp="2026-06-02 15:45:52 +0000 UTC" firstStartedPulling="2026-06-02 15:46:58.630161827 +0000 UTC m=+67.332625337" lastFinishedPulling="2026-06-02 15:46:59.560129012 +0000 UTC m=+68.262592521" observedRunningTime="2026-06-02 15:47:00.209206716 +0000 UTC m=+68.911670233" watchObservedRunningTime="2026-06-02 15:47:00.210757993 +0000 UTC m=+68.913221511"
Jun 02 15:47:01.230987 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.230951    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pm7zw"]
Jun 02 15:47:01.234309 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.234287    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pm7zw"
Jun 02 15:47:01.237792 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.237769    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jun 02 15:47:01.238545 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.238388    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jun 02 15:47:01.238545 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.238428    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hqmcq\""
Jun 02 15:47:01.238545 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.238479    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jun 02 15:47:01.238545 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.238539    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jun 02 15:47:01.244983 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.244960    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pm7zw"]
Jun 02 15:47:01.265428 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.265381    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7749958bb7-2xtcw"]
Jun 02 15:47:01.295239 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.295149    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5167c18f-db19-477e-a673-6b8a6795bc6e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pm7zw\" (UID: \"5167c18f-db19-477e-a673-6b8a6795bc6e\") " pod="openshift-insights/insights-runtime-extractor-pm7zw"
Jun 02 15:47:01.295239 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.295197    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5167c18f-db19-477e-a673-6b8a6795bc6e-crio-socket\") pod \"insights-runtime-extractor-pm7zw\" (UID: \"5167c18f-db19-477e-a673-6b8a6795bc6e\") " pod="openshift-insights/insights-runtime-extractor-pm7zw"
Jun 02 15:47:01.295239 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.295222    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5167c18f-db19-477e-a673-6b8a6795bc6e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pm7zw\" (UID: \"5167c18f-db19-477e-a673-6b8a6795bc6e\") " pod="openshift-insights/insights-runtime-extractor-pm7zw"
Jun 02 15:47:01.295517 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.295261    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5167c18f-db19-477e-a673-6b8a6795bc6e-data-volume\") pod \"insights-runtime-extractor-pm7zw\" (UID: \"5167c18f-db19-477e-a673-6b8a6795bc6e\") " pod="openshift-insights/insights-runtime-extractor-pm7zw"
Jun 02 15:47:01.295517 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.295301    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxz8v\" (UniqueName: \"kubernetes.io/projected/5167c18f-db19-477e-a673-6b8a6795bc6e-kube-api-access-dxz8v\") pod \"insights-runtime-extractor-pm7zw\" (UID: \"5167c18f-db19-477e-a673-6b8a6795bc6e\") " pod="openshift-insights/insights-runtime-extractor-pm7zw"
Jun 02 15:47:01.324068 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.324031    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-c8f78dbd9-4qk6x"]
Jun 02 15:47:01.326620 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.326595    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c8f78dbd9-4qk6x"
Jun 02 15:47:01.342743 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.342704    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c8f78dbd9-4qk6x"]
Jun 02 15:47:01.395872 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.395850    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dxz8v\" (UniqueName: \"kubernetes.io/projected/5167c18f-db19-477e-a673-6b8a6795bc6e-kube-api-access-dxz8v\") pod \"insights-runtime-extractor-pm7zw\" (UID: \"5167c18f-db19-477e-a673-6b8a6795bc6e\") " pod="openshift-insights/insights-runtime-extractor-pm7zw"
Jun 02 15:47:01.395970 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.395883    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff4aa919-595a-4e50-8a64-0e182f6c8e13-trusted-ca\") pod \"image-registry-c8f78dbd9-4qk6x\" (UID: \"ff4aa919-595a-4e50-8a64-0e182f6c8e13\") " pod="openshift-image-registry/image-registry-c8f78dbd9-4qk6x"
Jun 02 15:47:01.395970 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.395905    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff4aa919-595a-4e50-8a64-0e182f6c8e13-registry-certificates\") pod \"image-registry-c8f78dbd9-4qk6x\" (UID: \"ff4aa919-595a-4e50-8a64-0e182f6c8e13\") " pod="openshift-image-registry/image-registry-c8f78dbd9-4qk6x"
Jun 02 15:47:01.395970 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.395940    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5167c18f-db19-477e-a673-6b8a6795bc6e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pm7zw\" (UID: \"5167c18f-db19-477e-a673-6b8a6795bc6e\") " pod="openshift-insights/insights-runtime-extractor-pm7zw"
Jun 02 15:47:01.395970 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.395964    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff4aa919-595a-4e50-8a64-0e182f6c8e13-ca-trust-extracted\") pod \"image-registry-c8f78dbd9-4qk6x\" (UID: \"ff4aa919-595a-4e50-8a64-0e182f6c8e13\") " pod="openshift-image-registry/image-registry-c8f78dbd9-4qk6x"
Jun 02 15:47:01.396130 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.395983    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5167c18f-db19-477e-a673-6b8a6795bc6e-data-volume\") pod \"insights-runtime-extractor-pm7zw\" (UID: \"5167c18f-db19-477e-a673-6b8a6795bc6e\") " pod="openshift-insights/insights-runtime-extractor-pm7zw"
Jun 02 15:47:01.396130 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.396051    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff4aa919-595a-4e50-8a64-0e182f6c8e13-registry-tls\") pod \"image-registry-c8f78dbd9-4qk6x\" (UID: \"ff4aa919-595a-4e50-8a64-0e182f6c8e13\") " pod="openshift-image-registry/image-registry-c8f78dbd9-4qk6x"
Jun 02 15:47:01.396130 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.396087    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff4aa919-595a-4e50-8a64-0e182f6c8e13-installation-pull-secrets\") pod \"image-registry-c8f78dbd9-4qk6x\" (UID: \"ff4aa919-595a-4e50-8a64-0e182f6c8e13\") " pod="openshift-image-registry/image-registry-c8f78dbd9-4qk6x"
Jun 02 15:47:01.396299 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.396151    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5167c18f-db19-477e-a673-6b8a6795bc6e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pm7zw\" (UID: \"5167c18f-db19-477e-a673-6b8a6795bc6e\") " pod="openshift-insights/insights-runtime-extractor-pm7zw"
Jun 02 15:47:01.396299 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.396197    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcbvl\" (UniqueName: \"kubernetes.io/projected/ff4aa919-595a-4e50-8a64-0e182f6c8e13-kube-api-access-qcbvl\") pod \"image-registry-c8f78dbd9-4qk6x\" (UID: \"ff4aa919-595a-4e50-8a64-0e182f6c8e13\") " pod="openshift-image-registry/image-registry-c8f78dbd9-4qk6x"
Jun 02 15:47:01.396299 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.396233    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff4aa919-595a-4e50-8a64-0e182f6c8e13-image-registry-private-configuration\") pod \"image-registry-c8f78dbd9-4qk6x\" (UID: \"ff4aa919-595a-4e50-8a64-0e182f6c8e13\") " pod="openshift-image-registry/image-registry-c8f78dbd9-4qk6x"
Jun 02 15:47:01.396299 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.396266    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5167c18f-db19-477e-a673-6b8a6795bc6e-crio-socket\") pod \"insights-runtime-extractor-pm7zw\" (UID: \"5167c18f-db19-477e-a673-6b8a6795bc6e\") " pod="openshift-insights/insights-runtime-extractor-pm7zw"
Jun 02 15:47:01.396470 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.396305    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff4aa919-595a-4e50-8a64-0e182f6c8e13-bound-sa-token\") pod \"image-registry-c8f78dbd9-4qk6x\" (UID: \"ff4aa919-595a-4e50-8a64-0e182f6c8e13\") " pod="openshift-image-registry/image-registry-c8f78dbd9-4qk6x"
Jun 02 15:47:01.396470 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.396367    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5167c18f-db19-477e-a673-6b8a6795bc6e-crio-socket\") pod \"insights-runtime-extractor-pm7zw\" (UID: \"5167c18f-db19-477e-a673-6b8a6795bc6e\") " pod="openshift-insights/insights-runtime-extractor-pm7zw"
Jun 02 15:47:01.396470 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.396371    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5167c18f-db19-477e-a673-6b8a6795bc6e-data-volume\") pod \"insights-runtime-extractor-pm7zw\" (UID: \"5167c18f-db19-477e-a673-6b8a6795bc6e\") " pod="openshift-insights/insights-runtime-extractor-pm7zw"
Jun 02 15:47:01.396575 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.396559    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5167c18f-db19-477e-a673-6b8a6795bc6e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pm7zw\" (UID: \"5167c18f-db19-477e-a673-6b8a6795bc6e\") " pod="openshift-insights/insights-runtime-extractor-pm7zw"
Jun 02 15:47:01.398160 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.398143    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5167c18f-db19-477e-a673-6b8a6795bc6e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pm7zw\" (UID: \"5167c18f-db19-477e-a673-6b8a6795bc6e\") " pod="openshift-insights/insights-runtime-extractor-pm7zw"
Jun 02 15:47:01.403717 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.403700    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxz8v\" (UniqueName: \"kubernetes.io/projected/5167c18f-db19-477e-a673-6b8a6795bc6e-kube-api-access-dxz8v\") pod \"insights-runtime-extractor-pm7zw\" (UID: \"5167c18f-db19-477e-a673-6b8a6795bc6e\") " pod="openshift-insights/insights-runtime-extractor-pm7zw"
Jun 02 15:47:01.497230 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.497143    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff4aa919-595a-4e50-8a64-0e182f6c8e13-registry-tls\") pod \"image-registry-c8f78dbd9-4qk6x\" (UID: \"ff4aa919-595a-4e50-8a64-0e182f6c8e13\") " pod="openshift-image-registry/image-registry-c8f78dbd9-4qk6x"
Jun 02 15:47:01.497230 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.497193    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff4aa919-595a-4e50-8a64-0e182f6c8e13-installation-pull-secrets\") pod \"image-registry-c8f78dbd9-4qk6x\" (UID: \"ff4aa919-595a-4e50-8a64-0e182f6c8e13\") " pod="openshift-image-registry/image-registry-c8f78dbd9-4qk6x"
Jun 02 15:47:01.497230 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.497220    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qcbvl\" (UniqueName: \"kubernetes.io/projected/ff4aa919-595a-4e50-8a64-0e182f6c8e13-kube-api-access-qcbvl\") pod \"image-registry-c8f78dbd9-4qk6x\" (UID: \"ff4aa919-595a-4e50-8a64-0e182f6c8e13\") " pod="openshift-image-registry/image-registry-c8f78dbd9-4qk6x"
Jun 02 15:47:01.497511 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.497237    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff4aa919-595a-4e50-8a64-0e182f6c8e13-image-registry-private-configuration\") pod \"image-registry-c8f78dbd9-4qk6x\" (UID: \"ff4aa919-595a-4e50-8a64-0e182f6c8e13\") " pod="openshift-image-registry/image-registry-c8f78dbd9-4qk6x"
Jun 02 15:47:01.497511 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.497269    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff4aa919-595a-4e50-8a64-0e182f6c8e13-bound-sa-token\") pod \"image-registry-c8f78dbd9-4qk6x\" (UID: \"ff4aa919-595a-4e50-8a64-0e182f6c8e13\") " pod="openshift-image-registry/image-registry-c8f78dbd9-4qk6x"
Jun 02 15:47:01.497511 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.497317    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff4aa919-595a-4e50-8a64-0e182f6c8e13-trusted-ca\") pod \"image-registry-c8f78dbd9-4qk6x\" (UID: \"ff4aa919-595a-4e50-8a64-0e182f6c8e13\") " pod="openshift-image-registry/image-registry-c8f78dbd9-4qk6x"
Jun 02 15:47:01.497511 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.497351    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff4aa919-595a-4e50-8a64-0e182f6c8e13-registry-certificates\") pod \"image-registry-c8f78dbd9-4qk6x\" (UID: \"ff4aa919-595a-4e50-8a64-0e182f6c8e13\") " pod="openshift-image-registry/image-registry-c8f78dbd9-4qk6x"
Jun 02 15:47:01.497511 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.497391    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff4aa919-595a-4e50-8a64-0e182f6c8e13-ca-trust-extracted\") pod \"image-registry-c8f78dbd9-4qk6x\" (UID: \"ff4aa919-595a-4e50-8a64-0e182f6c8e13\") " pod="openshift-image-registry/image-registry-c8f78dbd9-4qk6x"
Jun 02 15:47:01.497923 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.497884    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff4aa919-595a-4e50-8a64-0e182f6c8e13-ca-trust-extracted\") pod \"image-registry-c8f78dbd9-4qk6x\" (UID: \"ff4aa919-595a-4e50-8a64-0e182f6c8e13\") " pod="openshift-image-registry/image-registry-c8f78dbd9-4qk6x"
Jun 02 15:47:01.498338 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.498317    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff4aa919-595a-4e50-8a64-0e182f6c8e13-trusted-ca\") pod \"image-registry-c8f78dbd9-4qk6x\" (UID: \"ff4aa919-595a-4e50-8a64-0e182f6c8e13\") " pod="openshift-image-registry/image-registry-c8f78dbd9-4qk6x"
Jun 02 15:47:01.498444 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.498336    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff4aa919-595a-4e50-8a64-0e182f6c8e13-registry-certificates\") pod \"image-registry-c8f78dbd9-4qk6x\" (UID: \"ff4aa919-595a-4e50-8a64-0e182f6c8e13\") " pod="openshift-image-registry/image-registry-c8f78dbd9-4qk6x"
Jun 02 15:47:01.499742 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.499719    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff4aa919-595a-4e50-8a64-0e182f6c8e13-image-registry-private-configuration\") pod \"image-registry-c8f78dbd9-4qk6x\" (UID: \"ff4aa919-595a-4e50-8a64-0e182f6c8e13\") " pod="openshift-image-registry/image-registry-c8f78dbd9-4qk6x"
Jun 02 15:47:01.499742 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.499738    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff4aa919-595a-4e50-8a64-0e182f6c8e13-registry-tls\") pod \"image-registry-c8f78dbd9-4qk6x\" (UID: \"ff4aa919-595a-4e50-8a64-0e182f6c8e13\") " pod="openshift-image-registry/image-registry-c8f78dbd9-4qk6x"
Jun 02 15:47:01.499866 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.499805    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff4aa919-595a-4e50-8a64-0e182f6c8e13-installation-pull-secrets\") pod \"image-registry-c8f78dbd9-4qk6x\" (UID: \"ff4aa919-595a-4e50-8a64-0e182f6c8e13\") " pod="openshift-image-registry/image-registry-c8f78dbd9-4qk6x"
Jun 02 15:47:01.505597 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.505572    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff4aa919-595a-4e50-8a64-0e182f6c8e13-bound-sa-token\") pod \"image-registry-c8f78dbd9-4qk6x\" (UID: \"ff4aa919-595a-4e50-8a64-0e182f6c8e13\") " pod="openshift-image-registry/image-registry-c8f78dbd9-4qk6x"
Jun 02 15:47:01.505841 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.505824    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcbvl\" (UniqueName: \"kubernetes.io/projected/ff4aa919-595a-4e50-8a64-0e182f6c8e13-kube-api-access-qcbvl\") pod \"image-registry-c8f78dbd9-4qk6x\" (UID: \"ff4aa919-595a-4e50-8a64-0e182f6c8e13\") " pod="openshift-image-registry/image-registry-c8f78dbd9-4qk6x"
Jun 02 15:47:01.545914 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.545879    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pm7zw"
Jun 02 15:47:01.635412 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.635366    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c8f78dbd9-4qk6x"
Jun 02 15:47:01.680360 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.680333    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pm7zw"]
Jun 02 15:47:01.690747 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:47:01.690700    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5167c18f_db19_477e_a673_6b8a6795bc6e.slice/crio-2ae06a501bf7a71bb3787226b2d67ce1fc2b38c0a4328a65bcf1322c215d219d WatchSource:0}: Error finding container 2ae06a501bf7a71bb3787226b2d67ce1fc2b38c0a4328a65bcf1322c215d219d: Status 404 returned error can't find the container with id 2ae06a501bf7a71bb3787226b2d67ce1fc2b38c0a4328a65bcf1322c215d219d
Jun 02 15:47:01.775261 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:01.775212    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c8f78dbd9-4qk6x"]
Jun 02 15:47:02.199157 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:02.199120    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pm7zw" event={"ID":"5167c18f-db19-477e-a673-6b8a6795bc6e","Type":"ContainerStarted","Data":"16a3f9ae475d6b0aea4212c79e8d7b364b076abeddc1d21ad4163e7cbba5c1c9"}
Jun 02 15:47:02.199157 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:02.199162    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pm7zw" event={"ID":"5167c18f-db19-477e-a673-6b8a6795bc6e","Type":"ContainerStarted","Data":"2ae06a501bf7a71bb3787226b2d67ce1fc2b38c0a4328a65bcf1322c215d219d"}
Jun 02 15:47:02.200411 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:02.200388    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-smdm5" event={"ID":"a664bf1a-e1bf-4107-9ebb-076ddd6b6234","Type":"ContainerStarted","Data":"784084de38bbb9b1883bf5be2f4969e8af8971d2cc23b776405f2fa9e2f2b9cf"}
Jun 02 15:47:02.200636 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:02.200613    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-smdm5"
Jun 02 15:47:02.201675 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:02.201653    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c8f78dbd9-4qk6x" event={"ID":"ff4aa919-595a-4e50-8a64-0e182f6c8e13","Type":"ContainerStarted","Data":"da09197a6053ec022563aec0d1d61a15042518fcefcb879117f1f5ed4c9d4dff"}
Jun 02 15:47:02.201755 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:02.201680    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c8f78dbd9-4qk6x" event={"ID":"ff4aa919-595a-4e50-8a64-0e182f6c8e13","Type":"ContainerStarted","Data":"c75e3a69df5fc07654cd3b1d47daf133d5b37876f18588cb016be19f2f88c170"}
Jun 02 15:47:02.201792 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:02.201771    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-c8f78dbd9-4qk6x"
Jun 02 15:47:02.219768 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:02.219717    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-smdm5" podStartSLOduration=68.447262597 podStartE2EDuration="1m11.219700696s" podCreationTimestamp="2026-06-02 15:45:51 +0000 UTC" firstStartedPulling="2026-06-02 15:46:58.604422857 +0000 UTC m=+67.306886370" lastFinishedPulling="2026-06-02 15:47:01.37686096 +0000 UTC m=+70.079324469" observedRunningTime="2026-06-02 15:47:02.219114645 +0000 UTC m=+70.921578162" watchObservedRunningTime="2026-06-02 15:47:02.219700696 +0000 UTC m=+70.922164216"
Jun 02 15:47:02.242718 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:02.242658    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-c8f78dbd9-4qk6x" podStartSLOduration=1.242635937 podStartE2EDuration="1.242635937s" podCreationTimestamp="2026-06-02 15:47:01 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 15:47:02.241269634 +0000 UTC m=+70.943733151" watchObservedRunningTime="2026-06-02 15:47:02.242635937 +0000 UTC m=+70.945099455"
Jun 02 15:47:03.206402 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:03.206365    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pm7zw" event={"ID":"5167c18f-db19-477e-a673-6b8a6795bc6e","Type":"ContainerStarted","Data":"6e7942b88ef4d0a8374febf0ac2c2a0b54994442f2676a0ab0b2083379ba158a"}
Jun 02 15:47:05.213525 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:05.213484    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pm7zw" event={"ID":"5167c18f-db19-477e-a673-6b8a6795bc6e","Type":"ContainerStarted","Data":"7157093d24b62229e5d1de5b246451159be04e5505c41eddc2fac549ad2fe588"}
Jun 02 15:47:05.230953 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:05.230907    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pm7zw" podStartSLOduration=1.620309196 podStartE2EDuration="4.230895482s" podCreationTimestamp="2026-06-02 15:47:01 +0000 UTC" firstStartedPulling="2026-06-02 15:47:01.747470044 +0000 UTC m=+70.449933540" lastFinishedPulling="2026-06-02 15:47:04.358056316 +0000 UTC m=+73.060519826" observedRunningTime="2026-06-02 15:47:05.230884768 +0000 UTC m=+73.933348287" watchObservedRunningTime="2026-06-02 15:47:05.230895482 +0000 UTC m=+73.933359000"
Jun 02 15:47:09.194776 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:09.194736    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5pxm2"
Jun 02 15:47:11.270596 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:11.270570    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7749958bb7-2xtcw"
Jun 02 15:47:22.107817 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:22.107782    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-59b7f8756b-vtgbm"]
Jun 02 15:47:22.110746 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:22.110730    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-59b7f8756b-vtgbm"
Jun 02 15:47:22.121713 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:22.121687    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-x2bsn\""
Jun 02 15:47:22.122957 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:22.122940    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jun 02 15:47:22.147111 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:22.147089    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-59b7f8756b-vtgbm"]
Jun 02 15:47:22.232234 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:22.232203    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8242da94-694d-499d-8c71-b3a479b8c726-tls-certificates\") pod \"prometheus-operator-admission-webhook-59b7f8756b-vtgbm\" (UID: \"8242da94-694d-499d-8c71-b3a479b8c726\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-59b7f8756b-vtgbm"
Jun 02 15:47:22.333083 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:22.333055    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8242da94-694d-499d-8c71-b3a479b8c726-tls-certificates\") pod \"prometheus-operator-admission-webhook-59b7f8756b-vtgbm\" (UID: \"8242da94-694d-499d-8c71-b3a479b8c726\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-59b7f8756b-vtgbm"
Jun 02 15:47:22.333257 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:47:22.333175    2578 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jun 02 15:47:22.333257 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:47:22.333225    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8242da94-694d-499d-8c71-b3a479b8c726-tls-certificates podName:8242da94-694d-499d-8c71-b3a479b8c726 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:22.833209304 +0000 UTC m=+91.535672800 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/8242da94-694d-499d-8c71-b3a479b8c726-tls-certificates") pod "prometheus-operator-admission-webhook-59b7f8756b-vtgbm" (UID: "8242da94-694d-499d-8c71-b3a479b8c726") : secret "prometheus-operator-admission-webhook-tls" not found
Jun 02 15:47:22.837318 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:22.837282    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8242da94-694d-499d-8c71-b3a479b8c726-tls-certificates\") pod \"prometheus-operator-admission-webhook-59b7f8756b-vtgbm\" (UID: \"8242da94-694d-499d-8c71-b3a479b8c726\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-59b7f8756b-vtgbm"
Jun 02 15:47:22.839702 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:22.839679    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8242da94-694d-499d-8c71-b3a479b8c726-tls-certificates\") pod \"prometheus-operator-admission-webhook-59b7f8756b-vtgbm\" (UID: \"8242da94-694d-499d-8c71-b3a479b8c726\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-59b7f8756b-vtgbm"
Jun 02 15:47:23.020602 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:23.020566    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-59b7f8756b-vtgbm"
Jun 02 15:47:23.161233 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:23.161212    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-59b7f8756b-vtgbm"]
Jun 02 15:47:23.163198 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:47:23.163173    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8242da94_694d_499d_8c71_b3a479b8c726.slice/crio-57f32b96427ed09712b208f57fed7b08dad8345e62ad0a40b52b91ec1c3cb971 WatchSource:0}: Error finding container 57f32b96427ed09712b208f57fed7b08dad8345e62ad0a40b52b91ec1c3cb971: Status 404 returned error can't find the container with id 57f32b96427ed09712b208f57fed7b08dad8345e62ad0a40b52b91ec1c3cb971
Jun 02 15:47:23.210587 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:23.210565    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-c8f78dbd9-4qk6x"
Jun 02 15:47:23.257185 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:23.257161    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-59b7f8756b-vtgbm" event={"ID":"8242da94-694d-499d-8c71-b3a479b8c726","Type":"ContainerStarted","Data":"57f32b96427ed09712b208f57fed7b08dad8345e62ad0a40b52b91ec1c3cb971"}
Jun 02 15:47:25.263614 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:25.263580    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-59b7f8756b-vtgbm" event={"ID":"8242da94-694d-499d-8c71-b3a479b8c726","Type":"ContainerStarted","Data":"08a517838a61dedc237a2607c645bfaa324928c7e2567a9c427f2f92168e16be"}
Jun 02 15:47:25.264102 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:25.263807    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-59b7f8756b-vtgbm"
Jun 02 15:47:25.268426 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:25.268404    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-59b7f8756b-vtgbm"
Jun 02 15:47:25.293706 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:25.293669    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-59b7f8756b-vtgbm" podStartSLOduration=2.144610472 podStartE2EDuration="3.293657476s" podCreationTimestamp="2026-06-02 15:47:22 +0000 UTC" firstStartedPulling="2026-06-02 15:47:23.165373955 +0000 UTC m=+91.867837455" lastFinishedPulling="2026-06-02 15:47:24.31442096 +0000 UTC m=+93.016884459" observedRunningTime="2026-06-02 15:47:25.292114532 +0000 UTC m=+93.994578050" watchObservedRunningTime="2026-06-02 15:47:25.293657476 +0000 UTC m=+93.996120993"
Jun 02 15:47:26.290277 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:26.290242    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7749958bb7-2xtcw" podUID="2c23e202-3ecd-4fff-ac9b-1b633ee32679" containerName="registry" containerID="cri-o://203da634b2be04cc053da9ce6aefbe879242997e1e410db12b1c5ca460a14760" gracePeriod=30
Jun 02 15:47:26.518513 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:26.518494    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7749958bb7-2xtcw"
Jun 02 15:47:26.563417 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:26.563349    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c23e202-3ecd-4fff-ac9b-1b633ee32679-bound-sa-token\") pod \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\" (UID: \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\") "
Jun 02 15:47:26.563417 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:26.563400    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2c23e202-3ecd-4fff-ac9b-1b633ee32679-trusted-ca\") pod \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\" (UID: \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\") "
Jun 02 15:47:26.563580 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:26.563443    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2w2rv\" (UniqueName: \"kubernetes.io/projected/2c23e202-3ecd-4fff-ac9b-1b633ee32679-kube-api-access-2w2rv\") pod \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\" (UID: \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\") "
Jun 02 15:47:26.563580 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:26.563473    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2c23e202-3ecd-4fff-ac9b-1b633ee32679-image-registry-private-configuration\") pod \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\" (UID: \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\") "
Jun 02 15:47:26.563580 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:26.563499    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2c23e202-3ecd-4fff-ac9b-1b633ee32679-ca-trust-extracted\") pod \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\" (UID: \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\") "
Jun 02 15:47:26.563580 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:26.563535    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2c23e202-3ecd-4fff-ac9b-1b633ee32679-registry-certificates\") pod \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\" (UID: \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\") "
Jun 02 15:47:26.563580 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:26.563560    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2c23e202-3ecd-4fff-ac9b-1b633ee32679-installation-pull-secrets\") pod \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\" (UID: \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\") "
Jun 02 15:47:26.563807 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:26.563588    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2c23e202-3ecd-4fff-ac9b-1b633ee32679-registry-tls\") pod \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\" (UID: \"2c23e202-3ecd-4fff-ac9b-1b633ee32679\") "
Jun 02 15:47:26.564152 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:26.563855    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c23e202-3ecd-4fff-ac9b-1b633ee32679-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "2c23e202-3ecd-4fff-ac9b-1b633ee32679" (UID: "2c23e202-3ecd-4fff-ac9b-1b633ee32679"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:47:26.564349 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:26.564307    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c23e202-3ecd-4fff-ac9b-1b633ee32679-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "2c23e202-3ecd-4fff-ac9b-1b633ee32679" (UID: "2c23e202-3ecd-4fff-ac9b-1b633ee32679"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:47:26.566118 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:26.566089    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c23e202-3ecd-4fff-ac9b-1b633ee32679-kube-api-access-2w2rv" (OuterVolumeSpecName: "kube-api-access-2w2rv") pod "2c23e202-3ecd-4fff-ac9b-1b633ee32679" (UID: "2c23e202-3ecd-4fff-ac9b-1b633ee32679"). InnerVolumeSpecName "kube-api-access-2w2rv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 15:47:26.566221 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:26.566133    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c23e202-3ecd-4fff-ac9b-1b633ee32679-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "2c23e202-3ecd-4fff-ac9b-1b633ee32679" (UID: "2c23e202-3ecd-4fff-ac9b-1b633ee32679"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 15:47:26.566421 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:26.566398    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c23e202-3ecd-4fff-ac9b-1b633ee32679-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "2c23e202-3ecd-4fff-ac9b-1b633ee32679" (UID: "2c23e202-3ecd-4fff-ac9b-1b633ee32679"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 15:47:26.566500 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:26.566480    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c23e202-3ecd-4fff-ac9b-1b633ee32679-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2c23e202-3ecd-4fff-ac9b-1b633ee32679" (UID: "2c23e202-3ecd-4fff-ac9b-1b633ee32679"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 15:47:26.567483 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:26.567459    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c23e202-3ecd-4fff-ac9b-1b633ee32679-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "2c23e202-3ecd-4fff-ac9b-1b633ee32679" (UID: "2c23e202-3ecd-4fff-ac9b-1b633ee32679"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 15:47:26.572516 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:26.572491    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c23e202-3ecd-4fff-ac9b-1b633ee32679-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "2c23e202-3ecd-4fff-ac9b-1b633ee32679" (UID: "2c23e202-3ecd-4fff-ac9b-1b633ee32679"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 15:47:26.664210 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:26.664186    2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2c23e202-3ecd-4fff-ac9b-1b633ee32679-registry-certificates\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 15:47:26.664210 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:26.664208    2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2c23e202-3ecd-4fff-ac9b-1b633ee32679-installation-pull-secrets\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 15:47:26.664344 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:26.664218    2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2c23e202-3ecd-4fff-ac9b-1b633ee32679-registry-tls\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 15:47:26.664344 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:26.664228    2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c23e202-3ecd-4fff-ac9b-1b633ee32679-bound-sa-token\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 15:47:26.664344 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:26.664236    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2c23e202-3ecd-4fff-ac9b-1b633ee32679-trusted-ca\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 15:47:26.664344 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:26.664243    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w2rv\" (UniqueName: \"kubernetes.io/projected/2c23e202-3ecd-4fff-ac9b-1b633ee32679-kube-api-access-2w2rv\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 15:47:26.664344 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:26.664253    2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2c23e202-3ecd-4fff-ac9b-1b633ee32679-image-registry-private-configuration\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 15:47:26.664344 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:26.664262    2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2c23e202-3ecd-4fff-ac9b-1b633ee32679-ca-trust-extracted\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 15:47:27.269812 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:27.269776    2578 generic.go:358] "Generic (PLEG): container finished" podID="2c23e202-3ecd-4fff-ac9b-1b633ee32679" containerID="203da634b2be04cc053da9ce6aefbe879242997e1e410db12b1c5ca460a14760" exitCode=0
Jun 02 15:47:27.270009 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:27.269856    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7749958bb7-2xtcw" event={"ID":"2c23e202-3ecd-4fff-ac9b-1b633ee32679","Type":"ContainerDied","Data":"203da634b2be04cc053da9ce6aefbe879242997e1e410db12b1c5ca460a14760"}
Jun 02 15:47:27.270009 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:27.269893    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7749958bb7-2xtcw" event={"ID":"2c23e202-3ecd-4fff-ac9b-1b633ee32679","Type":"ContainerDied","Data":"8fb00773eaa9ee0c881bd9af46cbc854f64d01b2427f395deaf57148a14ba468"}
Jun 02 15:47:27.270009 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:27.269908    2578 scope.go:117] "RemoveContainer" containerID="203da634b2be04cc053da9ce6aefbe879242997e1e410db12b1c5ca460a14760"
Jun 02 15:47:27.270009 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:27.269866    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7749958bb7-2xtcw"
Jun 02 15:47:27.277200 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:27.277182    2578 scope.go:117] "RemoveContainer" containerID="203da634b2be04cc053da9ce6aefbe879242997e1e410db12b1c5ca460a14760"
Jun 02 15:47:27.277466 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:47:27.277442    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"203da634b2be04cc053da9ce6aefbe879242997e1e410db12b1c5ca460a14760\": container with ID starting with 203da634b2be04cc053da9ce6aefbe879242997e1e410db12b1c5ca460a14760 not found: ID does not exist" containerID="203da634b2be04cc053da9ce6aefbe879242997e1e410db12b1c5ca460a14760"
Jun 02 15:47:27.277520 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:27.277479    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"203da634b2be04cc053da9ce6aefbe879242997e1e410db12b1c5ca460a14760"} err="failed to get container status \"203da634b2be04cc053da9ce6aefbe879242997e1e410db12b1c5ca460a14760\": rpc error: code = NotFound desc = could not find container \"203da634b2be04cc053da9ce6aefbe879242997e1e410db12b1c5ca460a14760\": container with ID starting with 203da634b2be04cc053da9ce6aefbe879242997e1e410db12b1c5ca460a14760 not found: ID does not exist"
Jun 02 15:47:27.297506 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:27.297483    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7749958bb7-2xtcw"]
Jun 02 15:47:27.307815 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:27.307793    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7749958bb7-2xtcw"]
Jun 02 15:47:27.906725 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:27.906703    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c23e202-3ecd-4fff-ac9b-1b633ee32679" path="/var/lib/kubelet/pods/2c23e202-3ecd-4fff-ac9b-1b633ee32679/volumes"
Jun 02 15:47:32.939097 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:32.939058    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-85658796db-nxv6k"]
Jun 02 15:47:32.939539 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:32.939380    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c23e202-3ecd-4fff-ac9b-1b633ee32679" containerName="registry"
Jun 02 15:47:32.939539 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:32.939395    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c23e202-3ecd-4fff-ac9b-1b633ee32679" containerName="registry"
Jun 02 15:47:32.939539 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:32.939443    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="2c23e202-3ecd-4fff-ac9b-1b633ee32679" containerName="registry"
Jun 02 15:47:32.941876 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:32.941859    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-85658796db-nxv6k"
Jun 02 15:47:32.946613 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:32.946589    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jun 02 15:47:32.946613 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:32.946611    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jun 02 15:47:32.946843 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:32.946670    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jun 02 15:47:32.947239 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:32.947221    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jun 02 15:47:32.947360 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:32.947241    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jun 02 15:47:32.947498 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:32.947483    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-h56jw\""
Jun 02 15:47:32.958787 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:32.958762    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-85658796db-nxv6k"]
Jun 02 15:47:32.971143 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:32.971123    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-8469c486bb-9b5kt"]
Jun 02 15:47:32.973777 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:32.973761    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-8469c486bb-9b5kt"
Jun 02 15:47:32.976031 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:32.975988    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jun 02 15:47:32.976127 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:32.976014    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jun 02 15:47:32.976181 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:32.976164    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-mkb6w\""
Jun 02 15:47:32.976449 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:32.976432    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jun 02 15:47:32.993449 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:32.993428    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-8469c486bb-9b5kt"]
Jun 02 15:47:33.006012 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.005980    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a074a268-b658-430e-83f0-d5e98f856d03-metrics-client-ca\") pod \"openshift-state-metrics-85658796db-nxv6k\" (UID: \"a074a268-b658-430e-83f0-d5e98f856d03\") " pod="openshift-monitoring/openshift-state-metrics-85658796db-nxv6k"
Jun 02 15:47:33.006118 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.006034    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a074a268-b658-430e-83f0-d5e98f856d03-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-85658796db-nxv6k\" (UID: \"a074a268-b658-430e-83f0-d5e98f856d03\") " pod="openshift-monitoring/openshift-state-metrics-85658796db-nxv6k"
Jun 02 15:47:33.006118 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.006066    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw6cl\" (UniqueName: \"kubernetes.io/projected/a074a268-b658-430e-83f0-d5e98f856d03-kube-api-access-xw6cl\") pod \"openshift-state-metrics-85658796db-nxv6k\" (UID: \"a074a268-b658-430e-83f0-d5e98f856d03\") " pod="openshift-monitoring/openshift-state-metrics-85658796db-nxv6k"
Jun 02 15:47:33.006221 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.006121    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a074a268-b658-430e-83f0-d5e98f856d03-openshift-state-metrics-tls\") pod \"openshift-state-metrics-85658796db-nxv6k\" (UID: \"a074a268-b658-430e-83f0-d5e98f856d03\") " pod="openshift-monitoring/openshift-state-metrics-85658796db-nxv6k"
Jun 02 15:47:33.017717 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.017698    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-j5z2h"]
Jun 02 15:47:33.020870 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.020854    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-j5z2h"
Jun 02 15:47:33.023248 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.023230    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jun 02 15:47:33.023603 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.023584    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jun 02 15:47:33.023686 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.023618    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6dctl\""
Jun 02 15:47:33.024062 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.024044    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jun 02 15:47:33.106569 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.106541    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xw6cl\" (UniqueName: \"kubernetes.io/projected/a074a268-b658-430e-83f0-d5e98f856d03-kube-api-access-xw6cl\") pod \"openshift-state-metrics-85658796db-nxv6k\" (UID: \"a074a268-b658-430e-83f0-d5e98f856d03\") " pod="openshift-monitoring/openshift-state-metrics-85658796db-nxv6k"
Jun 02 15:47:33.106722 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.106574    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c31d1126-4813-4f5d-9f89-23bac44e0919-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-8469c486bb-9b5kt\" (UID: \"c31d1126-4813-4f5d-9f89-23bac44e0919\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-9b5kt"
Jun 02 15:47:33.106722 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.106593    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b64cb704-5a89-4375-9a14-d6131bf4a0fb-node-exporter-tls\") pod \"node-exporter-j5z2h\" (UID: \"b64cb704-5a89-4375-9a14-d6131bf4a0fb\") " pod="openshift-monitoring/node-exporter-j5z2h"
Jun 02 15:47:33.106722 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.106640    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b64cb704-5a89-4375-9a14-d6131bf4a0fb-node-exporter-wtmp\") pod \"node-exporter-j5z2h\" (UID: \"b64cb704-5a89-4375-9a14-d6131bf4a0fb\") " pod="openshift-monitoring/node-exporter-j5z2h"
Jun 02 15:47:33.106890 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.106730    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c31d1126-4813-4f5d-9f89-23bac44e0919-kube-state-metrics-tls\") pod \"kube-state-metrics-8469c486bb-9b5kt\" (UID: \"c31d1126-4813-4f5d-9f89-23bac44e0919\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-9b5kt"
Jun 02 15:47:33.106890 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.106787    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a074a268-b658-430e-83f0-d5e98f856d03-openshift-state-metrics-tls\") pod \"openshift-state-metrics-85658796db-nxv6k\" (UID: \"a074a268-b658-430e-83f0-d5e98f856d03\") " pod="openshift-monitoring/openshift-state-metrics-85658796db-nxv6k"
Jun 02 15:47:33.106890 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.106845    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c31d1126-4813-4f5d-9f89-23bac44e0919-metrics-client-ca\") pod \"kube-state-metrics-8469c486bb-9b5kt\" (UID: \"c31d1126-4813-4f5d-9f89-23bac44e0919\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-9b5kt"
Jun 02 15:47:33.106890 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.106864    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c31d1126-4813-4f5d-9f89-23bac44e0919-volume-directive-shadow\") pod \"kube-state-metrics-8469c486bb-9b5kt\" (UID: \"c31d1126-4813-4f5d-9f89-23bac44e0919\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-9b5kt"
Jun 02 15:47:33.106890 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.106880    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b64cb704-5a89-4375-9a14-d6131bf4a0fb-metrics-client-ca\") pod \"node-exporter-j5z2h\" (UID: \"b64cb704-5a89-4375-9a14-d6131bf4a0fb\") " pod="openshift-monitoring/node-exporter-j5z2h"
Jun 02 15:47:33.107153 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.106905    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b64cb704-5a89-4375-9a14-d6131bf4a0fb-node-exporter-accelerators-collector-config\") pod \"node-exporter-j5z2h\" (UID: \"b64cb704-5a89-4375-9a14-d6131bf4a0fb\") " pod="openshift-monitoring/node-exporter-j5z2h"
Jun 02 15:47:33.107153 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.106942    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b64cb704-5a89-4375-9a14-d6131bf4a0fb-node-exporter-textfile\") pod \"node-exporter-j5z2h\" (UID: \"b64cb704-5a89-4375-9a14-d6131bf4a0fb\") " pod="openshift-monitoring/node-exporter-j5z2h"
Jun 02 15:47:33.107153 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.106964    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b64cb704-5a89-4375-9a14-d6131bf4a0fb-sys\") pod \"node-exporter-j5z2h\" (UID: \"b64cb704-5a89-4375-9a14-d6131bf4a0fb\") " pod="openshift-monitoring/node-exporter-j5z2h"
Jun 02 15:47:33.107153 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.106980    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b64cb704-5a89-4375-9a14-d6131bf4a0fb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j5z2h\" (UID: \"b64cb704-5a89-4375-9a14-d6131bf4a0fb\") " pod="openshift-monitoring/node-exporter-j5z2h"
Jun 02 15:47:33.107153 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.107024    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfsl2\" (UniqueName: \"kubernetes.io/projected/b64cb704-5a89-4375-9a14-d6131bf4a0fb-kube-api-access-cfsl2\") pod \"node-exporter-j5z2h\" (UID: \"b64cb704-5a89-4375-9a14-d6131bf4a0fb\") " pod="openshift-monitoring/node-exporter-j5z2h"
Jun 02 15:47:33.107153 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.107047    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c31d1126-4813-4f5d-9f89-23bac44e0919-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-8469c486bb-9b5kt\" (UID: \"c31d1126-4813-4f5d-9f89-23bac44e0919\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-9b5kt"
Jun 02 15:47:33.107153 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.107080    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zmz9\" (UniqueName: \"kubernetes.io/projected/c31d1126-4813-4f5d-9f89-23bac44e0919-kube-api-access-8zmz9\") pod \"kube-state-metrics-8469c486bb-9b5kt\" (UID: \"c31d1126-4813-4f5d-9f89-23bac44e0919\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-9b5kt"
Jun 02 15:47:33.107153 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.107146    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a074a268-b658-430e-83f0-d5e98f856d03-metrics-client-ca\") pod \"openshift-state-metrics-85658796db-nxv6k\" (UID: \"a074a268-b658-430e-83f0-d5e98f856d03\") " pod="openshift-monitoring/openshift-state-metrics-85658796db-nxv6k"
Jun 02 15:47:33.107524 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.107187    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b64cb704-5a89-4375-9a14-d6131bf4a0fb-root\") pod \"node-exporter-j5z2h\" (UID: \"b64cb704-5a89-4375-9a14-d6131bf4a0fb\") " pod="openshift-monitoring/node-exporter-j5z2h"
Jun 02 15:47:33.107524 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.107218    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a074a268-b658-430e-83f0-d5e98f856d03-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-85658796db-nxv6k\" (UID: \"a074a268-b658-430e-83f0-d5e98f856d03\") " pod="openshift-monitoring/openshift-state-metrics-85658796db-nxv6k"
Jun 02 15:47:33.107670 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.107647    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a074a268-b658-430e-83f0-d5e98f856d03-metrics-client-ca\") pod \"openshift-state-metrics-85658796db-nxv6k\" (UID: \"a074a268-b658-430e-83f0-d5e98f856d03\") " pod="openshift-monitoring/openshift-state-metrics-85658796db-nxv6k"
Jun 02 15:47:33.109195 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.109169    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a074a268-b658-430e-83f0-d5e98f856d03-openshift-state-metrics-tls\") pod \"openshift-state-metrics-85658796db-nxv6k\" (UID: \"a074a268-b658-430e-83f0-d5e98f856d03\") " pod="openshift-monitoring/openshift-state-metrics-85658796db-nxv6k"
Jun 02 15:47:33.109302 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.109246    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a074a268-b658-430e-83f0-d5e98f856d03-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-85658796db-nxv6k\" (UID: \"a074a268-b658-430e-83f0-d5e98f856d03\") " pod="openshift-monitoring/openshift-state-metrics-85658796db-nxv6k"
Jun 02 15:47:33.116181 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.116161    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw6cl\" (UniqueName: \"kubernetes.io/projected/a074a268-b658-430e-83f0-d5e98f856d03-kube-api-access-xw6cl\") pod \"openshift-state-metrics-85658796db-nxv6k\" (UID: \"a074a268-b658-430e-83f0-d5e98f856d03\") " pod="openshift-monitoring/openshift-state-metrics-85658796db-nxv6k"
Jun 02 15:47:33.207585 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.207509    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8zmz9\" (UniqueName: \"kubernetes.io/projected/c31d1126-4813-4f5d-9f89-23bac44e0919-kube-api-access-8zmz9\") pod \"kube-state-metrics-8469c486bb-9b5kt\" (UID: \"c31d1126-4813-4f5d-9f89-23bac44e0919\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-9b5kt"
Jun 02 15:47:33.207585 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.207561    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b64cb704-5a89-4375-9a14-d6131bf4a0fb-root\") pod \"node-exporter-j5z2h\" (UID: \"b64cb704-5a89-4375-9a14-d6131bf4a0fb\") " pod="openshift-monitoring/node-exporter-j5z2h"
Jun 02 15:47:33.207830 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.207593    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c31d1126-4813-4f5d-9f89-23bac44e0919-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-8469c486bb-9b5kt\" (UID: \"c31d1126-4813-4f5d-9f89-23bac44e0919\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-9b5kt"
Jun 02 15:47:33.207830 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.207620    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b64cb704-5a89-4375-9a14-d6131bf4a0fb-node-exporter-tls\") pod \"node-exporter-j5z2h\" (UID: \"b64cb704-5a89-4375-9a14-d6131bf4a0fb\") " pod="openshift-monitoring/node-exporter-j5z2h"
Jun 02 15:47:33.207830 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.207644    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b64cb704-5a89-4375-9a14-d6131bf4a0fb-node-exporter-wtmp\") pod \"node-exporter-j5z2h\" (UID: \"b64cb704-5a89-4375-9a14-d6131bf4a0fb\") " pod="openshift-monitoring/node-exporter-j5z2h"
Jun 02 15:47:33.207830 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.207653    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b64cb704-5a89-4375-9a14-d6131bf4a0fb-root\") pod \"node-exporter-j5z2h\" (UID: \"b64cb704-5a89-4375-9a14-d6131bf4a0fb\") " pod="openshift-monitoring/node-exporter-j5z2h"
Jun 02 15:47:33.207830 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.207668    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c31d1126-4813-4f5d-9f89-23bac44e0919-kube-state-metrics-tls\") pod \"kube-state-metrics-8469c486bb-9b5kt\" (UID: \"c31d1126-4813-4f5d-9f89-23bac44e0919\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-9b5kt"
Jun 02 15:47:33.207830 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:47:33.207757    2578 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jun 02 15:47:33.207830 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:47:33.207812    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b64cb704-5a89-4375-9a14-d6131bf4a0fb-node-exporter-tls podName:b64cb704-5a89-4375-9a14-d6131bf4a0fb nodeName:}" failed. No retries permitted until 2026-06-02 15:47:33.707791786 +0000 UTC m=+102.410255292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/b64cb704-5a89-4375-9a14-d6131bf4a0fb-node-exporter-tls") pod "node-exporter-j5z2h" (UID: "b64cb704-5a89-4375-9a14-d6131bf4a0fb") : secret "node-exporter-tls" not found
Jun 02 15:47:33.207830 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.207809    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b64cb704-5a89-4375-9a14-d6131bf4a0fb-node-exporter-wtmp\") pod \"node-exporter-j5z2h\" (UID: \"b64cb704-5a89-4375-9a14-d6131bf4a0fb\") " pod="openshift-monitoring/node-exporter-j5z2h"
Jun 02 15:47:33.207830 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.207828    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c31d1126-4813-4f5d-9f89-23bac44e0919-metrics-client-ca\") pod \"kube-state-metrics-8469c486bb-9b5kt\" (UID: \"c31d1126-4813-4f5d-9f89-23bac44e0919\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-9b5kt"
Jun 02 15:47:33.208364 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.207856    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c31d1126-4813-4f5d-9f89-23bac44e0919-volume-directive-shadow\") pod \"kube-state-metrics-8469c486bb-9b5kt\" (UID: \"c31d1126-4813-4f5d-9f89-23bac44e0919\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-9b5kt"
Jun 02 15:47:33.208364 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.207881    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b64cb704-5a89-4375-9a14-d6131bf4a0fb-metrics-client-ca\") pod \"node-exporter-j5z2h\" (UID: \"b64cb704-5a89-4375-9a14-d6131bf4a0fb\") " pod="openshift-monitoring/node-exporter-j5z2h"
Jun 02 15:47:33.208364 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.207905    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b64cb704-5a89-4375-9a14-d6131bf4a0fb-node-exporter-accelerators-collector-config\") pod \"node-exporter-j5z2h\" (UID: \"b64cb704-5a89-4375-9a14-d6131bf4a0fb\") " pod="openshift-monitoring/node-exporter-j5z2h"
Jun 02 15:47:33.208364 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.207933    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b64cb704-5a89-4375-9a14-d6131bf4a0fb-node-exporter-textfile\") pod \"node-exporter-j5z2h\" (UID: \"b64cb704-5a89-4375-9a14-d6131bf4a0fb\") " pod="openshift-monitoring/node-exporter-j5z2h"
Jun 02 15:47:33.208364 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.207959    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b64cb704-5a89-4375-9a14-d6131bf4a0fb-sys\") pod \"node-exporter-j5z2h\" (UID: \"b64cb704-5a89-4375-9a14-d6131bf4a0fb\") " pod="openshift-monitoring/node-exporter-j5z2h"
Jun 02 15:47:33.208364 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.207984    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b64cb704-5a89-4375-9a14-d6131bf4a0fb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j5z2h\" (UID: \"b64cb704-5a89-4375-9a14-d6131bf4a0fb\") " pod="openshift-monitoring/node-exporter-j5z2h"
Jun 02 15:47:33.208364 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.208024    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cfsl2\" (UniqueName: \"kubernetes.io/projected/b64cb704-5a89-4375-9a14-d6131bf4a0fb-kube-api-access-cfsl2\") pod \"node-exporter-j5z2h\" (UID: \"b64cb704-5a89-4375-9a14-d6131bf4a0fb\") " pod="openshift-monitoring/node-exporter-j5z2h"
Jun 02 15:47:33.208364 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.208052    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c31d1126-4813-4f5d-9f89-23bac44e0919-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-8469c486bb-9b5kt\" (UID: \"c31d1126-4813-4f5d-9f89-23bac44e0919\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-9b5kt"
Jun 02 15:47:33.208364 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.208319    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b64cb704-5a89-4375-9a14-d6131bf4a0fb-node-exporter-textfile\") pod \"node-exporter-j5z2h\" (UID: \"b64cb704-5a89-4375-9a14-d6131bf4a0fb\") " pod="openshift-monitoring/node-exporter-j5z2h"
Jun 02 15:47:33.208760 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.208472    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c31d1126-4813-4f5d-9f89-23bac44e0919-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-8469c486bb-9b5kt\" (UID: \"c31d1126-4813-4f5d-9f89-23bac44e0919\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-9b5kt"
Jun 02 15:47:33.208760 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.208616    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b64cb704-5a89-4375-9a14-d6131bf4a0fb-metrics-client-ca\") pod \"node-exporter-j5z2h\" (UID: \"b64cb704-5a89-4375-9a14-d6131bf4a0fb\") " pod="openshift-monitoring/node-exporter-j5z2h"
Jun 02 15:47:33.208863 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.208773    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b64cb704-5a89-4375-9a14-d6131bf4a0fb-node-exporter-accelerators-collector-config\") pod \"node-exporter-j5z2h\" (UID: \"b64cb704-5a89-4375-9a14-d6131bf4a0fb\") " pod="openshift-monitoring/node-exporter-j5z2h"
Jun 02 15:47:33.208926 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.208909    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-smdm5"
Jun 02 15:47:33.208926 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.208917    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c31d1126-4813-4f5d-9f89-23bac44e0919-volume-directive-shadow\") pod \"kube-state-metrics-8469c486bb-9b5kt\" (UID: \"c31d1126-4813-4f5d-9f89-23bac44e0919\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-9b5kt"
Jun 02 15:47:33.209129 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.209107    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c31d1126-4813-4f5d-9f89-23bac44e0919-metrics-client-ca\") pod \"kube-state-metrics-8469c486bb-9b5kt\" (UID: \"c31d1126-4813-4f5d-9f89-23bac44e0919\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-9b5kt"
Jun 02 15:47:33.209378 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.209361    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b64cb704-5a89-4375-9a14-d6131bf4a0fb-sys\") pod \"node-exporter-j5z2h\" (UID: \"b64cb704-5a89-4375-9a14-d6131bf4a0fb\") " pod="openshift-monitoring/node-exporter-j5z2h"
Jun 02 15:47:33.211076 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.211053    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c31d1126-4813-4f5d-9f89-23bac44e0919-kube-state-metrics-tls\") pod \"kube-state-metrics-8469c486bb-9b5kt\" (UID: \"c31d1126-4813-4f5d-9f89-23bac44e0919\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-9b5kt"
Jun 02 15:47:33.211488 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.211449    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b64cb704-5a89-4375-9a14-d6131bf4a0fb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j5z2h\" (UID: \"b64cb704-5a89-4375-9a14-d6131bf4a0fb\") " pod="openshift-monitoring/node-exporter-j5z2h"
Jun 02 15:47:33.211692 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.211672    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c31d1126-4813-4f5d-9f89-23bac44e0919-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-8469c486bb-9b5kt\" (UID: \"c31d1126-4813-4f5d-9f89-23bac44e0919\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-9b5kt"
Jun 02 15:47:33.217963 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.217944    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zmz9\" (UniqueName: \"kubernetes.io/projected/c31d1126-4813-4f5d-9f89-23bac44e0919-kube-api-access-8zmz9\") pod \"kube-state-metrics-8469c486bb-9b5kt\" (UID: \"c31d1126-4813-4f5d-9f89-23bac44e0919\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-9b5kt"
Jun 02 15:47:33.223284 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.223261    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfsl2\" (UniqueName: \"kubernetes.io/projected/b64cb704-5a89-4375-9a14-d6131bf4a0fb-kube-api-access-cfsl2\") pod \"node-exporter-j5z2h\" (UID: \"b64cb704-5a89-4375-9a14-d6131bf4a0fb\") " pod="openshift-monitoring/node-exporter-j5z2h"
Jun 02 15:47:33.250941 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.250919    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-85658796db-nxv6k"
Jun 02 15:47:33.282576 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.282322    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-8469c486bb-9b5kt"
Jun 02 15:47:33.442117 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.442084    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-85658796db-nxv6k"]
Jun 02 15:47:33.447438 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:47:33.447414    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda074a268_b658_430e_83f0_d5e98f856d03.slice/crio-cb4f980c1b6cf721d9d432d215940f522ec768d0a536c1a024038efed15b4793 WatchSource:0}: Error finding container cb4f980c1b6cf721d9d432d215940f522ec768d0a536c1a024038efed15b4793: Status 404 returned error can't find the container with id cb4f980c1b6cf721d9d432d215940f522ec768d0a536c1a024038efed15b4793
Jun 02 15:47:33.469629 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.469607    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-8469c486bb-9b5kt"]
Jun 02 15:47:33.474472 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:47:33.474442    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc31d1126_4813_4f5d_9f89_23bac44e0919.slice/crio-1922161fe554a903020f55e585677538dea873cfcd4fba9ae6cd8817f8fa4327 WatchSource:0}: Error finding container 1922161fe554a903020f55e585677538dea873cfcd4fba9ae6cd8817f8fa4327: Status 404 returned error can't find the container with id 1922161fe554a903020f55e585677538dea873cfcd4fba9ae6cd8817f8fa4327
Jun 02 15:47:33.712700 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:33.712613    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b64cb704-5a89-4375-9a14-d6131bf4a0fb-node-exporter-tls\") pod \"node-exporter-j5z2h\" (UID: \"b64cb704-5a89-4375-9a14-d6131bf4a0fb\") " pod="openshift-monitoring/node-exporter-j5z2h"
Jun 02 15:47:33.712870 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:47:33.712759    2578 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jun 02 15:47:33.712870 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:47:33.712820    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b64cb704-5a89-4375-9a14-d6131bf4a0fb-node-exporter-tls podName:b64cb704-5a89-4375-9a14-d6131bf4a0fb nodeName:}" failed. No retries permitted until 2026-06-02 15:47:34.712805043 +0000 UTC m=+103.415268539 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/b64cb704-5a89-4375-9a14-d6131bf4a0fb-node-exporter-tls") pod "node-exporter-j5z2h" (UID: "b64cb704-5a89-4375-9a14-d6131bf4a0fb") : secret "node-exporter-tls" not found
Jun 02 15:47:34.080927 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.080835    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jun 02 15:47:34.084968 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.084944    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:34.089531 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.089507    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jun 02 15:47:34.089644 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.089587    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jun 02 15:47:34.089644 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.089616    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jun 02 15:47:34.089833 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.089819    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wh5ph\""
Jun 02 15:47:34.090038 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.089989    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jun 02 15:47:34.090038 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.090010    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jun 02 15:47:34.090148 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.090064    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jun 02 15:47:34.092971 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.092955    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jun 02 15:47:34.093825 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.093810    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jun 02 15:47:34.093825 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.093820    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jun 02 15:47:34.119223 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.119200    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jun 02 15:47:34.215284 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.215252    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/20954fac-f654-426a-9b6a-007786d3d7b9-web-config\") pod \"alertmanager-main-0\" (UID: \"20954fac-f654-426a-9b6a-007786d3d7b9\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:34.215439 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.215296    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/20954fac-f654-426a-9b6a-007786d3d7b9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"20954fac-f654-426a-9b6a-007786d3d7b9\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:34.215439 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.215319    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/20954fac-f654-426a-9b6a-007786d3d7b9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"20954fac-f654-426a-9b6a-007786d3d7b9\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:34.215439 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.215378    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/20954fac-f654-426a-9b6a-007786d3d7b9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"20954fac-f654-426a-9b6a-007786d3d7b9\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:34.215603 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.215458    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/20954fac-f654-426a-9b6a-007786d3d7b9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"20954fac-f654-426a-9b6a-007786d3d7b9\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:34.215603 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.215507    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/20954fac-f654-426a-9b6a-007786d3d7b9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"20954fac-f654-426a-9b6a-007786d3d7b9\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:34.215603 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.215542    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/20954fac-f654-426a-9b6a-007786d3d7b9-config-out\") pod \"alertmanager-main-0\" (UID: \"20954fac-f654-426a-9b6a-007786d3d7b9\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:34.215603 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.215574    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/20954fac-f654-426a-9b6a-007786d3d7b9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"20954fac-f654-426a-9b6a-007786d3d7b9\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:34.215760 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.215604    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/20954fac-f654-426a-9b6a-007786d3d7b9-config-volume\") pod \"alertmanager-main-0\" (UID: \"20954fac-f654-426a-9b6a-007786d3d7b9\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:34.215760 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.215626    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/20954fac-f654-426a-9b6a-007786d3d7b9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"20954fac-f654-426a-9b6a-007786d3d7b9\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:34.215760 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.215664    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20954fac-f654-426a-9b6a-007786d3d7b9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"20954fac-f654-426a-9b6a-007786d3d7b9\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:34.215760 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.215725    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42mpb\" (UniqueName: \"kubernetes.io/projected/20954fac-f654-426a-9b6a-007786d3d7b9-kube-api-access-42mpb\") pod \"alertmanager-main-0\" (UID: \"20954fac-f654-426a-9b6a-007786d3d7b9\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:34.215760 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.215758    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20954fac-f654-426a-9b6a-007786d3d7b9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"20954fac-f654-426a-9b6a-007786d3d7b9\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:34.289380 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.289345    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-8469c486bb-9b5kt" event={"ID":"c31d1126-4813-4f5d-9f89-23bac44e0919","Type":"ContainerStarted","Data":"1922161fe554a903020f55e585677538dea873cfcd4fba9ae6cd8817f8fa4327"}
Jun 02 15:47:34.291069 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.291037    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-85658796db-nxv6k" event={"ID":"a074a268-b658-430e-83f0-d5e98f856d03","Type":"ContainerStarted","Data":"488bc99538a4a9b09915e11b3fb56a871102a2ac4930207a16f97d8c3a98fbca"}
Jun 02 15:47:34.291069 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.291071    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-85658796db-nxv6k" event={"ID":"a074a268-b658-430e-83f0-d5e98f856d03","Type":"ContainerStarted","Data":"d324a7552c3517ee55f9048d735f1da6ea6e46d0b409c5ff90e976429f95ffdb"}
Jun 02 15:47:34.291209 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.291084    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-85658796db-nxv6k" event={"ID":"a074a268-b658-430e-83f0-d5e98f856d03","Type":"ContainerStarted","Data":"cb4f980c1b6cf721d9d432d215940f522ec768d0a536c1a024038efed15b4793"}
Jun 02 15:47:34.316495 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.316468    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/20954fac-f654-426a-9b6a-007786d3d7b9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"20954fac-f654-426a-9b6a-007786d3d7b9\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:34.316616 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.316519    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/20954fac-f654-426a-9b6a-007786d3d7b9-config-volume\") pod \"alertmanager-main-0\" (UID: \"20954fac-f654-426a-9b6a-007786d3d7b9\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:34.316616 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.316548    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/20954fac-f654-426a-9b6a-007786d3d7b9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"20954fac-f654-426a-9b6a-007786d3d7b9\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:34.316755 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.316718    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20954fac-f654-426a-9b6a-007786d3d7b9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"20954fac-f654-426a-9b6a-007786d3d7b9\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:34.316900 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.316769    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-42mpb\" (UniqueName: \"kubernetes.io/projected/20954fac-f654-426a-9b6a-007786d3d7b9-kube-api-access-42mpb\") pod \"alertmanager-main-0\" (UID: \"20954fac-f654-426a-9b6a-007786d3d7b9\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:34.316900 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.316813    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20954fac-f654-426a-9b6a-007786d3d7b9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"20954fac-f654-426a-9b6a-007786d3d7b9\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:34.316900 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.316871    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/20954fac-f654-426a-9b6a-007786d3d7b9-web-config\") pod \"alertmanager-main-0\" (UID: \"20954fac-f654-426a-9b6a-007786d3d7b9\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:34.317074 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.316905    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/20954fac-f654-426a-9b6a-007786d3d7b9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"20954fac-f654-426a-9b6a-007786d3d7b9\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:34.317218 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.317197    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/20954fac-f654-426a-9b6a-007786d3d7b9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"20954fac-f654-426a-9b6a-007786d3d7b9\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:34.317288 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.317239    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/20954fac-f654-426a-9b6a-007786d3d7b9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"20954fac-f654-426a-9b6a-007786d3d7b9\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:34.317288 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.317283    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/20954fac-f654-426a-9b6a-007786d3d7b9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"20954fac-f654-426a-9b6a-007786d3d7b9\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:34.317385 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.317316    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/20954fac-f654-426a-9b6a-007786d3d7b9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"20954fac-f654-426a-9b6a-007786d3d7b9\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:34.317385 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.317351    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/20954fac-f654-426a-9b6a-007786d3d7b9-config-out\") pod \"alertmanager-main-0\" (UID: \"20954fac-f654-426a-9b6a-007786d3d7b9\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:34.318725 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.317760    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20954fac-f654-426a-9b6a-007786d3d7b9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"20954fac-f654-426a-9b6a-007786d3d7b9\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:34.318725 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.317781    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/20954fac-f654-426a-9b6a-007786d3d7b9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"20954fac-f654-426a-9b6a-007786d3d7b9\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:34.318725 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.318684    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20954fac-f654-426a-9b6a-007786d3d7b9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"20954fac-f654-426a-9b6a-007786d3d7b9\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:34.320467 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.320039    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/20954fac-f654-426a-9b6a-007786d3d7b9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"20954fac-f654-426a-9b6a-007786d3d7b9\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:34.320467 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.320146    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/20954fac-f654-426a-9b6a-007786d3d7b9-web-config\") pod \"alertmanager-main-0\" (UID: \"20954fac-f654-426a-9b6a-007786d3d7b9\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:34.320467 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.320186    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/20954fac-f654-426a-9b6a-007786d3d7b9-config-volume\") pod \"alertmanager-main-0\" (UID: \"20954fac-f654-426a-9b6a-007786d3d7b9\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:34.320467 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.320422    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/20954fac-f654-426a-9b6a-007786d3d7b9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"20954fac-f654-426a-9b6a-007786d3d7b9\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:34.320963 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.320942    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/20954fac-f654-426a-9b6a-007786d3d7b9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"20954fac-f654-426a-9b6a-007786d3d7b9\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:34.321178 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.321136    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/20954fac-f654-426a-9b6a-007786d3d7b9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"20954fac-f654-426a-9b6a-007786d3d7b9\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:34.321334 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.321315    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/20954fac-f654-426a-9b6a-007786d3d7b9-config-out\") pod \"alertmanager-main-0\" (UID: \"20954fac-f654-426a-9b6a-007786d3d7b9\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:34.322323 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.322297    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/20954fac-f654-426a-9b6a-007786d3d7b9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"20954fac-f654-426a-9b6a-007786d3d7b9\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:34.322406 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.322298    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/20954fac-f654-426a-9b6a-007786d3d7b9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"20954fac-f654-426a-9b6a-007786d3d7b9\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:34.330626 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.330608    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42mpb\" (UniqueName: \"kubernetes.io/projected/20954fac-f654-426a-9b6a-007786d3d7b9-kube-api-access-42mpb\") pod \"alertmanager-main-0\" (UID: \"20954fac-f654-426a-9b6a-007786d3d7b9\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:34.394329 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.394258    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:34.548788 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.548741    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jun 02 15:47:34.714879 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:47:34.714849    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20954fac_f654_426a_9b6a_007786d3d7b9.slice/crio-3a7f258fa909defd8bcf3e0599c85d8bc37ca80c0c5dd422021b34d8781bf363 WatchSource:0}: Error finding container 3a7f258fa909defd8bcf3e0599c85d8bc37ca80c0c5dd422021b34d8781bf363: Status 404 returned error can't find the container with id 3a7f258fa909defd8bcf3e0599c85d8bc37ca80c0c5dd422021b34d8781bf363
Jun 02 15:47:34.721469 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.721444    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b64cb704-5a89-4375-9a14-d6131bf4a0fb-node-exporter-tls\") pod \"node-exporter-j5z2h\" (UID: \"b64cb704-5a89-4375-9a14-d6131bf4a0fb\") " pod="openshift-monitoring/node-exporter-j5z2h"
Jun 02 15:47:34.723538 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.723516    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b64cb704-5a89-4375-9a14-d6131bf4a0fb-node-exporter-tls\") pod \"node-exporter-j5z2h\" (UID: \"b64cb704-5a89-4375-9a14-d6131bf4a0fb\") " pod="openshift-monitoring/node-exporter-j5z2h"
Jun 02 15:47:34.829721 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:34.829687    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-j5z2h"
Jun 02 15:47:35.034362 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:47:35.034337    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb64cb704_5a89_4375_9a14_d6131bf4a0fb.slice/crio-dce31031289c6d4efc294c7017a1c260bc72a12a0098b7603c5ac723cda55f06 WatchSource:0}: Error finding container dce31031289c6d4efc294c7017a1c260bc72a12a0098b7603c5ac723cda55f06: Status 404 returned error can't find the container with id dce31031289c6d4efc294c7017a1c260bc72a12a0098b7603c5ac723cda55f06
Jun 02 15:47:35.295421 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:35.295337    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"20954fac-f654-426a-9b6a-007786d3d7b9","Type":"ContainerStarted","Data":"3a7f258fa909defd8bcf3e0599c85d8bc37ca80c0c5dd422021b34d8781bf363"}
Jun 02 15:47:35.297878 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:35.297847    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-8469c486bb-9b5kt" event={"ID":"c31d1126-4813-4f5d-9f89-23bac44e0919","Type":"ContainerStarted","Data":"4e1ab3e094b918d1a6f953626e635ca903d0e40870690ee24237d555b3779848"}
Jun 02 15:47:35.298013 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:35.297884    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-8469c486bb-9b5kt" event={"ID":"c31d1126-4813-4f5d-9f89-23bac44e0919","Type":"ContainerStarted","Data":"1655fe3ed6b0427ed0f2e881411388e7490df5f1fb16d707b3d12aee2da99a80"}
Jun 02 15:47:35.298013 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:35.297897    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-8469c486bb-9b5kt" event={"ID":"c31d1126-4813-4f5d-9f89-23bac44e0919","Type":"ContainerStarted","Data":"a390a2002613b4332314628963797250bfbc3667da7c6327be5c11dbe8869cc9"}
Jun 02 15:47:35.298984 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:35.298961    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j5z2h" event={"ID":"b64cb704-5a89-4375-9a14-d6131bf4a0fb","Type":"ContainerStarted","Data":"dce31031289c6d4efc294c7017a1c260bc72a12a0098b7603c5ac723cda55f06"}
Jun 02 15:47:35.301077 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:35.301053    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-85658796db-nxv6k" event={"ID":"a074a268-b658-430e-83f0-d5e98f856d03","Type":"ContainerStarted","Data":"f8b82bee78c3f89e74b946dcf4cd143cf8f1cdc6c47901551a7e2935e9c9f038"}
Jun 02 15:47:35.321133 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:35.320817    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-8469c486bb-9b5kt" podStartSLOduration=1.7620679369999999 podStartE2EDuration="3.320802856s" podCreationTimestamp="2026-06-02 15:47:32 +0000 UTC" firstStartedPulling="2026-06-02 15:47:33.476871878 +0000 UTC m=+102.179335375" lastFinishedPulling="2026-06-02 15:47:35.035606795 +0000 UTC m=+103.738070294" observedRunningTime="2026-06-02 15:47:35.31962454 +0000 UTC m=+104.022088058" watchObservedRunningTime="2026-06-02 15:47:35.320802856 +0000 UTC m=+104.023266376"
Jun 02 15:47:35.340964 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:35.340913    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-85658796db-nxv6k" podStartSLOduration=1.860512748 podStartE2EDuration="3.340895741s" podCreationTimestamp="2026-06-02 15:47:32 +0000 UTC" firstStartedPulling="2026-06-02 15:47:33.55212079 +0000 UTC m=+102.254584286" lastFinishedPulling="2026-06-02 15:47:35.032503769 +0000 UTC m=+103.734967279" observedRunningTime="2026-06-02 15:47:35.339940861 +0000 UTC m=+104.042404401" watchObservedRunningTime="2026-06-02 15:47:35.340895741 +0000 UTC m=+104.043359260"
Jun 02 15:47:35.970741 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:35.970709    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-86d46bcd55-5mltm"]
Jun 02 15:47:35.995114 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:35.994989    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-86d46bcd55-5mltm"]
Jun 02 15:47:35.997728 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:35.997704    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-86d46bcd55-5mltm"
Jun 02 15:47:36.000265 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:36.000185    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jun 02 15:47:36.000954 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:36.000796    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jun 02 15:47:36.000954 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:36.000798    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-bplhc\""
Jun 02 15:47:36.000954 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:36.000904    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jun 02 15:47:36.001176 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:36.000989    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jun 02 15:47:36.003043 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:36.003011    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jun 02 15:47:36.004521 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:36.004504    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-a093m1o8si1nb\""
Jun 02 15:47:36.135242 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:36.135208    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c6a52535-c632-41ab-9a09-4f4494e7c2d8-secret-grpc-tls\") pod \"thanos-querier-86d46bcd55-5mltm\" (UID: \"c6a52535-c632-41ab-9a09-4f4494e7c2d8\") " pod="openshift-monitoring/thanos-querier-86d46bcd55-5mltm"
Jun 02 15:47:36.135417 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:36.135248    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c6a52535-c632-41ab-9a09-4f4494e7c2d8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86d46bcd55-5mltm\" (UID: \"c6a52535-c632-41ab-9a09-4f4494e7c2d8\") " pod="openshift-monitoring/thanos-querier-86d46bcd55-5mltm"
Jun 02 15:47:36.135417 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:36.135348    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6a52535-c632-41ab-9a09-4f4494e7c2d8-metrics-client-ca\") pod \"thanos-querier-86d46bcd55-5mltm\" (UID: \"c6a52535-c632-41ab-9a09-4f4494e7c2d8\") " pod="openshift-monitoring/thanos-querier-86d46bcd55-5mltm"
Jun 02 15:47:36.135417 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:36.135388    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c6a52535-c632-41ab-9a09-4f4494e7c2d8-secret-thanos-querier-tls\") pod \"thanos-querier-86d46bcd55-5mltm\" (UID: \"c6a52535-c632-41ab-9a09-4f4494e7c2d8\") " pod="openshift-monitoring/thanos-querier-86d46bcd55-5mltm"
Jun 02 15:47:36.135582 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:36.135437    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c6a52535-c632-41ab-9a09-4f4494e7c2d8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86d46bcd55-5mltm\" (UID: \"c6a52535-c632-41ab-9a09-4f4494e7c2d8\") " pod="openshift-monitoring/thanos-querier-86d46bcd55-5mltm"
Jun 02 15:47:36.135582 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:36.135478    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c6a52535-c632-41ab-9a09-4f4494e7c2d8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86d46bcd55-5mltm\" (UID: \"c6a52535-c632-41ab-9a09-4f4494e7c2d8\") " pod="openshift-monitoring/thanos-querier-86d46bcd55-5mltm"
Jun 02 15:47:36.135582 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:36.135512    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c6a52535-c632-41ab-9a09-4f4494e7c2d8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86d46bcd55-5mltm\" (UID: \"c6a52535-c632-41ab-9a09-4f4494e7c2d8\") " pod="openshift-monitoring/thanos-querier-86d46bcd55-5mltm"
Jun 02 15:47:36.135582 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:36.135556    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sv5q\" (UniqueName: \"kubernetes.io/projected/c6a52535-c632-41ab-9a09-4f4494e7c2d8-kube-api-access-6sv5q\") pod \"thanos-querier-86d46bcd55-5mltm\" (UID: \"c6a52535-c632-41ab-9a09-4f4494e7c2d8\") " pod="openshift-monitoring/thanos-querier-86d46bcd55-5mltm"
Jun 02 15:47:36.236908 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:36.236831    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c6a52535-c632-41ab-9a09-4f4494e7c2d8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86d46bcd55-5mltm\" (UID: \"c6a52535-c632-41ab-9a09-4f4494e7c2d8\") " pod="openshift-monitoring/thanos-querier-86d46bcd55-5mltm"
Jun 02 15:47:36.236908 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:36.236887    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c6a52535-c632-41ab-9a09-4f4494e7c2d8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86d46bcd55-5mltm\" (UID: \"c6a52535-c632-41ab-9a09-4f4494e7c2d8\") " pod="openshift-monitoring/thanos-querier-86d46bcd55-5mltm"
Jun 02 15:47:36.237112 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:36.236921    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c6a52535-c632-41ab-9a09-4f4494e7c2d8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86d46bcd55-5mltm\" (UID: \"c6a52535-c632-41ab-9a09-4f4494e7c2d8\") " pod="openshift-monitoring/thanos-querier-86d46bcd55-5mltm"
Jun 02 15:47:36.237112 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:36.236957    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6sv5q\" (UniqueName: \"kubernetes.io/projected/c6a52535-c632-41ab-9a09-4f4494e7c2d8-kube-api-access-6sv5q\") pod \"thanos-querier-86d46bcd55-5mltm\" (UID: \"c6a52535-c632-41ab-9a09-4f4494e7c2d8\") " pod="openshift-monitoring/thanos-querier-86d46bcd55-5mltm"
Jun 02 15:47:36.237112 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:36.236984    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c6a52535-c632-41ab-9a09-4f4494e7c2d8-secret-grpc-tls\") pod \"thanos-querier-86d46bcd55-5mltm\" (UID: \"c6a52535-c632-41ab-9a09-4f4494e7c2d8\") " pod="openshift-monitoring/thanos-querier-86d46bcd55-5mltm"
Jun 02 15:47:36.237112 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:36.237032    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c6a52535-c632-41ab-9a09-4f4494e7c2d8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86d46bcd55-5mltm\" (UID: \"c6a52535-c632-41ab-9a09-4f4494e7c2d8\") " pod="openshift-monitoring/thanos-querier-86d46bcd55-5mltm"
Jun 02 15:47:36.237112 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:36.237098    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6a52535-c632-41ab-9a09-4f4494e7c2d8-metrics-client-ca\") pod \"thanos-querier-86d46bcd55-5mltm\" (UID: \"c6a52535-c632-41ab-9a09-4f4494e7c2d8\") " pod="openshift-monitoring/thanos-querier-86d46bcd55-5mltm"
Jun 02 15:47:36.237350 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:36.237126    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c6a52535-c632-41ab-9a09-4f4494e7c2d8-secret-thanos-querier-tls\") pod \"thanos-querier-86d46bcd55-5mltm\" (UID: \"c6a52535-c632-41ab-9a09-4f4494e7c2d8\") " pod="openshift-monitoring/thanos-querier-86d46bcd55-5mltm"
Jun 02 15:47:36.237808 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:36.237776    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6a52535-c632-41ab-9a09-4f4494e7c2d8-metrics-client-ca\") pod \"thanos-querier-86d46bcd55-5mltm\" (UID: \"c6a52535-c632-41ab-9a09-4f4494e7c2d8\") " pod="openshift-monitoring/thanos-querier-86d46bcd55-5mltm"
Jun 02 15:47:36.239813 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:36.239785    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c6a52535-c632-41ab-9a09-4f4494e7c2d8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86d46bcd55-5mltm\" (UID: \"c6a52535-c632-41ab-9a09-4f4494e7c2d8\") " pod="openshift-monitoring/thanos-querier-86d46bcd55-5mltm"
Jun 02 15:47:36.239907 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:36.239845    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c6a52535-c632-41ab-9a09-4f4494e7c2d8-secret-thanos-querier-tls\") pod \"thanos-querier-86d46bcd55-5mltm\" (UID: \"c6a52535-c632-41ab-9a09-4f4494e7c2d8\") " pod="openshift-monitoring/thanos-querier-86d46bcd55-5mltm"
Jun 02 15:47:36.240067 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:36.240047    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c6a52535-c632-41ab-9a09-4f4494e7c2d8-secret-grpc-tls\") pod \"thanos-querier-86d46bcd55-5mltm\" (UID: \"c6a52535-c632-41ab-9a09-4f4494e7c2d8\") " pod="openshift-monitoring/thanos-querier-86d46bcd55-5mltm"
Jun 02 15:47:36.240132 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:36.240068    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c6a52535-c632-41ab-9a09-4f4494e7c2d8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86d46bcd55-5mltm\" (UID: \"c6a52535-c632-41ab-9a09-4f4494e7c2d8\") " pod="openshift-monitoring/thanos-querier-86d46bcd55-5mltm"
Jun 02 15:47:36.240181 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:36.240164    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c6a52535-c632-41ab-9a09-4f4494e7c2d8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86d46bcd55-5mltm\" (UID: \"c6a52535-c632-41ab-9a09-4f4494e7c2d8\") " pod="openshift-monitoring/thanos-querier-86d46bcd55-5mltm"
Jun 02 15:47:36.240231 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:36.240211    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c6a52535-c632-41ab-9a09-4f4494e7c2d8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86d46bcd55-5mltm\" (UID: \"c6a52535-c632-41ab-9a09-4f4494e7c2d8\") " pod="openshift-monitoring/thanos-querier-86d46bcd55-5mltm"
Jun 02 15:47:36.246927 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:36.246907    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sv5q\" (UniqueName: \"kubernetes.io/projected/c6a52535-c632-41ab-9a09-4f4494e7c2d8-kube-api-access-6sv5q\") pod \"thanos-querier-86d46bcd55-5mltm\" (UID: \"c6a52535-c632-41ab-9a09-4f4494e7c2d8\") " pod="openshift-monitoring/thanos-querier-86d46bcd55-5mltm"
Jun 02 15:47:36.308480 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:36.308457    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-86d46bcd55-5mltm"
Jun 02 15:47:36.722260 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:36.722083    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-86d46bcd55-5mltm"]
Jun 02 15:47:36.724214 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:47:36.724188    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6a52535_c632_41ab_9a09_4f4494e7c2d8.slice/crio-d0b870dc50c0679b030771aadfea31a95297ffe0d65c43a45845d209fc4dc352 WatchSource:0}: Error finding container d0b870dc50c0679b030771aadfea31a95297ffe0d65c43a45845d209fc4dc352: Status 404 returned error can't find the container with id d0b870dc50c0679b030771aadfea31a95297ffe0d65c43a45845d209fc4dc352
Jun 02 15:47:37.307847 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:37.307815    2578 generic.go:358] "Generic (PLEG): container finished" podID="20954fac-f654-426a-9b6a-007786d3d7b9" containerID="c72f8374d4cdaf504c63d85e501f32e8c84502883294f0e553ddd79f54491ed6" exitCode=0
Jun 02 15:47:37.308044 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:37.307905    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"20954fac-f654-426a-9b6a-007786d3d7b9","Type":"ContainerDied","Data":"c72f8374d4cdaf504c63d85e501f32e8c84502883294f0e553ddd79f54491ed6"}
Jun 02 15:47:37.309647 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:37.309625    2578 generic.go:358] "Generic (PLEG): container finished" podID="b64cb704-5a89-4375-9a14-d6131bf4a0fb" containerID="1b5de48567d033e933b007085345c004c5a6b2dff695de5117f743fd28f28ab7" exitCode=0
Jun 02 15:47:37.309954 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:37.309656    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j5z2h" event={"ID":"b64cb704-5a89-4375-9a14-d6131bf4a0fb","Type":"ContainerDied","Data":"1b5de48567d033e933b007085345c004c5a6b2dff695de5117f743fd28f28ab7"}
Jun 02 15:47:37.311040 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:37.310948    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86d46bcd55-5mltm" event={"ID":"c6a52535-c632-41ab-9a09-4f4494e7c2d8","Type":"ContainerStarted","Data":"d0b870dc50c0679b030771aadfea31a95297ffe0d65c43a45845d209fc4dc352"}
Jun 02 15:47:38.317019 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:38.316961    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j5z2h" event={"ID":"b64cb704-5a89-4375-9a14-d6131bf4a0fb","Type":"ContainerStarted","Data":"13db68d974f16abc609b977a3dcc24f0a24afd88251feda15ea0d6edd740f091"}
Jun 02 15:47:38.317473 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:38.317382    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j5z2h" event={"ID":"b64cb704-5a89-4375-9a14-d6131bf4a0fb","Type":"ContainerStarted","Data":"6a4014a9764197e90a247306660e2d54cd80c08bb6e5a24cfa8c99297632b70c"}
Jun 02 15:47:38.340624 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:38.340368    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-j5z2h" podStartSLOduration=4.785682897 podStartE2EDuration="6.340351499s" podCreationTimestamp="2026-06-02 15:47:32 +0000 UTC" firstStartedPulling="2026-06-02 15:47:35.035938034 +0000 UTC m=+103.738401530" lastFinishedPulling="2026-06-02 15:47:36.590606634 +0000 UTC m=+105.293070132" observedRunningTime="2026-06-02 15:47:38.338862137 +0000 UTC m=+107.041325656" watchObservedRunningTime="2026-06-02 15:47:38.340351499 +0000 UTC m=+107.042815019"
Jun 02 15:47:39.131020 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.130928    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jun 02 15:47:39.134912 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.134883    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.145510 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.145319    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jun 02 15:47:39.145510 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.145319    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jun 02 15:47:39.145510 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.145399    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jun 02 15:47:39.145745 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.145628    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jun 02 15:47:39.145745 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.145692    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jun 02 15:47:39.145745 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.145737    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jun 02 15:47:39.145898 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.145742    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jun 02 15:47:39.145898 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.145799    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-j5ds5\""
Jun 02 15:47:39.146395 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.146374    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jun 02 15:47:39.146510 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.146492    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jun 02 15:47:39.146570 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.146525    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jun 02 15:47:39.146636 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.146569    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jun 02 15:47:39.146907 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.146887    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2kifvavufd06i\""
Jun 02 15:47:39.146907 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.146903    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jun 02 15:47:39.150981 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.150961    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jun 02 15:47:39.158685 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.158663    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jun 02 15:47:39.261962 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.261924    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdb7335b-4f53-4632-8403-db5825ff15c6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.261962 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.261966    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cdb7335b-4f53-4632-8403-db5825ff15c6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.262199 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.262017    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cdb7335b-4f53-4632-8403-db5825ff15c6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.262199 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.262042    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cdb7335b-4f53-4632-8403-db5825ff15c6-web-config\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.262199 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.262078    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cdb7335b-4f53-4632-8403-db5825ff15c6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.262199 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.262145    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdb7335b-4f53-4632-8403-db5825ff15c6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.262199 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.262189    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlr7f\" (UniqueName: \"kubernetes.io/projected/cdb7335b-4f53-4632-8403-db5825ff15c6-kube-api-access-tlr7f\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.262387 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.262225    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cdb7335b-4f53-4632-8403-db5825ff15c6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.262387 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.262322    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cdb7335b-4f53-4632-8403-db5825ff15c6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.262387 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.262352    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdb7335b-4f53-4632-8403-db5825ff15c6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.262387 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.262381    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cdb7335b-4f53-4632-8403-db5825ff15c6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.262554 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.262441    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cdb7335b-4f53-4632-8403-db5825ff15c6-config\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.262554 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.262463    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cdb7335b-4f53-4632-8403-db5825ff15c6-config-out\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.262554 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.262487    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cdb7335b-4f53-4632-8403-db5825ff15c6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.262554 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.262516    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cdb7335b-4f53-4632-8403-db5825ff15c6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.262554 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.262537    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cdb7335b-4f53-4632-8403-db5825ff15c6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.262770 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.262564    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cdb7335b-4f53-4632-8403-db5825ff15c6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.262770 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.262595    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cdb7335b-4f53-4632-8403-db5825ff15c6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.322205 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.322167    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"20954fac-f654-426a-9b6a-007786d3d7b9","Type":"ContainerStarted","Data":"1118f3ac8a56e6be0bc14b5dae2f4783ea2a0b5d4fce32e5a8c4e737cf91129a"}
Jun 02 15:47:39.322205 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.322209    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"20954fac-f654-426a-9b6a-007786d3d7b9","Type":"ContainerStarted","Data":"dc0a29e78e6da0e08fd24cabbfbf2b3c5b234d424124d430adb4b4e5ee199029"}
Jun 02 15:47:39.322564 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.322222    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"20954fac-f654-426a-9b6a-007786d3d7b9","Type":"ContainerStarted","Data":"027e7952351b2d91e9f99399ecbc303bc7cef24427f877212ac76c4756370594"}
Jun 02 15:47:39.322564 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.322234    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"20954fac-f654-426a-9b6a-007786d3d7b9","Type":"ContainerStarted","Data":"b1c3d1b894c3cf1cb37ee1ef12d96e21f81070ad9f824b029a0ac30e6cf7e820"}
Jun 02 15:47:39.322564 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.322245    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"20954fac-f654-426a-9b6a-007786d3d7b9","Type":"ContainerStarted","Data":"d33aa417c3e0418323dca6c32ac26db4adecec9866326ff2f6b4704744a68d79"}
Jun 02 15:47:39.363675 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.363651    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cdb7335b-4f53-4632-8403-db5825ff15c6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.363780 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.363691    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdb7335b-4f53-4632-8403-db5825ff15c6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.363780 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.363716    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tlr7f\" (UniqueName: \"kubernetes.io/projected/cdb7335b-4f53-4632-8403-db5825ff15c6-kube-api-access-tlr7f\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.363970 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.363938    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cdb7335b-4f53-4632-8403-db5825ff15c6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.364097 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.364014    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cdb7335b-4f53-4632-8403-db5825ff15c6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.364097 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.364040    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdb7335b-4f53-4632-8403-db5825ff15c6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.364097 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.364066    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cdb7335b-4f53-4632-8403-db5825ff15c6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.364271 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.364109    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cdb7335b-4f53-4632-8403-db5825ff15c6-config\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.364271 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.364135    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cdb7335b-4f53-4632-8403-db5825ff15c6-config-out\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.364271 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.364159    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cdb7335b-4f53-4632-8403-db5825ff15c6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.364271 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.364185    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cdb7335b-4f53-4632-8403-db5825ff15c6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.364271 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.364210    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cdb7335b-4f53-4632-8403-db5825ff15c6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.364271 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.364239    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cdb7335b-4f53-4632-8403-db5825ff15c6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.364546 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.364276    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cdb7335b-4f53-4632-8403-db5825ff15c6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.364546 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.364323    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdb7335b-4f53-4632-8403-db5825ff15c6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.364546 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.364345    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cdb7335b-4f53-4632-8403-db5825ff15c6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.364546 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.364385    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cdb7335b-4f53-4632-8403-db5825ff15c6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.364546 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.364410    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cdb7335b-4f53-4632-8403-db5825ff15c6-web-config\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.365348 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.364383    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdb7335b-4f53-4632-8403-db5825ff15c6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.365438 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.365373    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdb7335b-4f53-4632-8403-db5825ff15c6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.365494 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.365432    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cdb7335b-4f53-4632-8403-db5825ff15c6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.367019 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.366494    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdb7335b-4f53-4632-8403-db5825ff15c6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.367019 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.366834    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cdb7335b-4f53-4632-8403-db5825ff15c6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.367520 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.367381    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cdb7335b-4f53-4632-8403-db5825ff15c6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.367520 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.367486    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cdb7335b-4f53-4632-8403-db5825ff15c6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.367520 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.367489    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cdb7335b-4f53-4632-8403-db5825ff15c6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.367715 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.367555    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cdb7335b-4f53-4632-8403-db5825ff15c6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.367715 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.367673    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cdb7335b-4f53-4632-8403-db5825ff15c6-web-config\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.368217 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.368186    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cdb7335b-4f53-4632-8403-db5825ff15c6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.368314 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.368187    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cdb7335b-4f53-4632-8403-db5825ff15c6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.368754 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.368733    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cdb7335b-4f53-4632-8403-db5825ff15c6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.368828 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.368784    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cdb7335b-4f53-4632-8403-db5825ff15c6-config-out\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.369491 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.369465    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cdb7335b-4f53-4632-8403-db5825ff15c6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.369555 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.369493    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/cdb7335b-4f53-4632-8403-db5825ff15c6-config\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.369791 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.369776    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cdb7335b-4f53-4632-8403-db5825ff15c6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.372114 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.372094    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlr7f\" (UniqueName: \"kubernetes.io/projected/cdb7335b-4f53-4632-8403-db5825ff15c6-kube-api-access-tlr7f\") pod \"prometheus-k8s-0\" (UID: \"cdb7335b-4f53-4632-8403-db5825ff15c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.446687 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.446667    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:39.607173 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:39.607070    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jun 02 15:47:39.611313 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:47:39.611285    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdb7335b_4f53_4632_8403_db5825ff15c6.slice/crio-cebfc424ff8a00f78074b29fa6c87939cb6b020c3b58a0cf4fa4f8e9a3df97e2 WatchSource:0}: Error finding container cebfc424ff8a00f78074b29fa6c87939cb6b020c3b58a0cf4fa4f8e9a3df97e2: Status 404 returned error can't find the container with id cebfc424ff8a00f78074b29fa6c87939cb6b020c3b58a0cf4fa4f8e9a3df97e2
Jun 02 15:47:40.326702 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:40.326618    2578 generic.go:358] "Generic (PLEG): container finished" podID="cdb7335b-4f53-4632-8403-db5825ff15c6" containerID="85d4cac6280d18e675a03ed25d0f45080fb24f3bf7c06439621ff40933a1abbf" exitCode=0
Jun 02 15:47:40.327150 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:40.326710    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cdb7335b-4f53-4632-8403-db5825ff15c6","Type":"ContainerDied","Data":"85d4cac6280d18e675a03ed25d0f45080fb24f3bf7c06439621ff40933a1abbf"}
Jun 02 15:47:40.327150 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:40.326754    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cdb7335b-4f53-4632-8403-db5825ff15c6","Type":"ContainerStarted","Data":"cebfc424ff8a00f78074b29fa6c87939cb6b020c3b58a0cf4fa4f8e9a3df97e2"}
Jun 02 15:47:40.329777 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:40.329758    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"20954fac-f654-426a-9b6a-007786d3d7b9","Type":"ContainerStarted","Data":"aeb307f900d38c6dd853c8fab36d92565bfe0d66a1410d4863ae56eb3151f54d"}
Jun 02 15:47:40.332146 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:40.332124    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86d46bcd55-5mltm" event={"ID":"c6a52535-c632-41ab-9a09-4f4494e7c2d8","Type":"ContainerStarted","Data":"928eaa57fb3203f769ecc030f5ec94aa707349c453a4a89cba06b59ab1a4e372"}
Jun 02 15:47:40.332229 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:40.332154    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86d46bcd55-5mltm" event={"ID":"c6a52535-c632-41ab-9a09-4f4494e7c2d8","Type":"ContainerStarted","Data":"c6ab3f79d404cdf215013caed901bbd5efdd0dc686a388682c5af0129b2407c7"}
Jun 02 15:47:40.332229 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:40.332168    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86d46bcd55-5mltm" event={"ID":"c6a52535-c632-41ab-9a09-4f4494e7c2d8","Type":"ContainerStarted","Data":"a56dee846599bb0acdc7b9df6ed3d2fb597d5d9ec33852807f9b95a7787896b4"}
Jun 02 15:47:40.332229 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:40.332209    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86d46bcd55-5mltm" event={"ID":"c6a52535-c632-41ab-9a09-4f4494e7c2d8","Type":"ContainerStarted","Data":"36ea70721dc7b1de78a50c5d23e84f70a1c6ed3e618b5edd14bc87171a6afa46"}
Jun 02 15:47:40.332229 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:40.332223    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86d46bcd55-5mltm" event={"ID":"c6a52535-c632-41ab-9a09-4f4494e7c2d8","Type":"ContainerStarted","Data":"6e18b0a34daef46e9ac0d87d7b562d0a22ae35659ad34dfffda89f26ec432240"}
Jun 02 15:47:40.332361 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:40.332236    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86d46bcd55-5mltm" event={"ID":"c6a52535-c632-41ab-9a09-4f4494e7c2d8","Type":"ContainerStarted","Data":"e006a62bbedcceaaf61fcfffe5e5dd0f72ae3a1f9436a6a27ba635095c45960e"}
Jun 02 15:47:40.332361 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:40.332288    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-86d46bcd55-5mltm"
Jun 02 15:47:40.397896 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:40.397842    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.251955098 podStartE2EDuration="6.397829328s" podCreationTimestamp="2026-06-02 15:47:34 +0000 UTC" firstStartedPulling="2026-06-02 15:47:34.716785697 +0000 UTC m=+103.419249194" lastFinishedPulling="2026-06-02 15:47:39.862659925 +0000 UTC m=+108.565123424" observedRunningTime="2026-06-02 15:47:40.396733085 +0000 UTC m=+109.099196616" watchObservedRunningTime="2026-06-02 15:47:40.397829328 +0000 UTC m=+109.100292845"
Jun 02 15:47:40.435937 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:40.435878    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-86d46bcd55-5mltm" podStartSLOduration=2.298864983 podStartE2EDuration="5.435859089s" podCreationTimestamp="2026-06-02 15:47:35 +0000 UTC" firstStartedPulling="2026-06-02 15:47:36.726200395 +0000 UTC m=+105.428663894" lastFinishedPulling="2026-06-02 15:47:39.863194504 +0000 UTC m=+108.565658000" observedRunningTime="2026-06-02 15:47:40.433494856 +0000 UTC m=+109.135958373" watchObservedRunningTime="2026-06-02 15:47:40.435859089 +0000 UTC m=+109.138322608"
Jun 02 15:47:43.344150 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:43.344123    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cdb7335b-4f53-4632-8403-db5825ff15c6","Type":"ContainerStarted","Data":"27faf57c3a5ab2bb3254d10313caf963077619085067b790b9f8cfba1e3f6e10"}
Jun 02 15:47:43.344408 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:43.344157    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cdb7335b-4f53-4632-8403-db5825ff15c6","Type":"ContainerStarted","Data":"84acf303f2f92fdddd7bab88f1f7a96c6777444ce07674be0246b55aed939565"}
Jun 02 15:47:43.344408 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:43.344181    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cdb7335b-4f53-4632-8403-db5825ff15c6","Type":"ContainerStarted","Data":"9152190d2416565669c7ddfea0af3e92730c00b8fa866f58254d4cfee5d1b1fa"}
Jun 02 15:47:43.344408 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:43.344189    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cdb7335b-4f53-4632-8403-db5825ff15c6","Type":"ContainerStarted","Data":"7e276afd9a1ebe93bc3794f575851d5226de8a625db210236abb35b7742b2ac0"}
Jun 02 15:47:44.349861 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:44.349823    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cdb7335b-4f53-4632-8403-db5825ff15c6","Type":"ContainerStarted","Data":"f202242dad83c260e190bd27fb5f475e4958156cb1837763c69e9b459e88c79c"}
Jun 02 15:47:44.349861 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:44.349863    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cdb7335b-4f53-4632-8403-db5825ff15c6","Type":"ContainerStarted","Data":"3fccfd78c96af091b2ece0a5661e49ff5648f9cc2e335b032527a6a59c6d1f6d"}
Jun 02 15:47:44.382365 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:44.382309    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.643690558 podStartE2EDuration="5.382293547s" podCreationTimestamp="2026-06-02 15:47:39 +0000 UTC" firstStartedPulling="2026-06-02 15:47:40.327880344 +0000 UTC m=+109.030343841" lastFinishedPulling="2026-06-02 15:47:43.06648333 +0000 UTC m=+111.768946830" observedRunningTime="2026-06-02 15:47:44.380323684 +0000 UTC m=+113.082787202" watchObservedRunningTime="2026-06-02 15:47:44.382293547 +0000 UTC m=+113.084757067"
Jun 02 15:47:44.447096 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:44.447070    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:46.342420 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:47:46.342393    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-86d46bcd55-5mltm"
Jun 02 15:48:39.447656 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:48:39.447625    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:48:39.465849 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:48:39.465825    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:48:39.527686 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:48:39.527660    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:40.578699 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:49:40.578645    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-rbwl6"]
Jun 02 15:49:40.581044 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:49:40.581023    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbwl6"
Jun 02 15:49:40.583253 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:49:40.583233    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jun 02 15:49:40.590071 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:49:40.590051    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rbwl6"]
Jun 02 15:49:40.753435 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:49:40.753388    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/44384f36-6a8f-47b3-b329-083afe044047-kubelet-config\") pod \"global-pull-secret-syncer-rbwl6\" (UID: \"44384f36-6a8f-47b3-b329-083afe044047\") " pod="kube-system/global-pull-secret-syncer-rbwl6"
Jun 02 15:49:40.753435 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:49:40.753438    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/44384f36-6a8f-47b3-b329-083afe044047-original-pull-secret\") pod \"global-pull-secret-syncer-rbwl6\" (UID: \"44384f36-6a8f-47b3-b329-083afe044047\") " pod="kube-system/global-pull-secret-syncer-rbwl6"
Jun 02 15:49:40.753649 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:49:40.753479    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/44384f36-6a8f-47b3-b329-083afe044047-dbus\") pod \"global-pull-secret-syncer-rbwl6\" (UID: \"44384f36-6a8f-47b3-b329-083afe044047\") " pod="kube-system/global-pull-secret-syncer-rbwl6"
Jun 02 15:49:40.854727 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:49:40.854628    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/44384f36-6a8f-47b3-b329-083afe044047-kubelet-config\") pod \"global-pull-secret-syncer-rbwl6\" (UID: \"44384f36-6a8f-47b3-b329-083afe044047\") " pod="kube-system/global-pull-secret-syncer-rbwl6"
Jun 02 15:49:40.854727 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:49:40.854692    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/44384f36-6a8f-47b3-b329-083afe044047-original-pull-secret\") pod \"global-pull-secret-syncer-rbwl6\" (UID: \"44384f36-6a8f-47b3-b329-083afe044047\") " pod="kube-system/global-pull-secret-syncer-rbwl6"
Jun 02 15:49:40.854889 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:49:40.854732    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/44384f36-6a8f-47b3-b329-083afe044047-dbus\") pod \"global-pull-secret-syncer-rbwl6\" (UID: \"44384f36-6a8f-47b3-b329-083afe044047\") " pod="kube-system/global-pull-secret-syncer-rbwl6"
Jun 02 15:49:40.854889 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:49:40.854759    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/44384f36-6a8f-47b3-b329-083afe044047-kubelet-config\") pod \"global-pull-secret-syncer-rbwl6\" (UID: \"44384f36-6a8f-47b3-b329-083afe044047\") " pod="kube-system/global-pull-secret-syncer-rbwl6"
Jun 02 15:49:40.854953 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:49:40.854890    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/44384f36-6a8f-47b3-b329-083afe044047-dbus\") pod \"global-pull-secret-syncer-rbwl6\" (UID: \"44384f36-6a8f-47b3-b329-083afe044047\") " pod="kube-system/global-pull-secret-syncer-rbwl6"
Jun 02 15:49:40.857024 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:49:40.856984    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/44384f36-6a8f-47b3-b329-083afe044047-original-pull-secret\") pod \"global-pull-secret-syncer-rbwl6\" (UID: \"44384f36-6a8f-47b3-b329-083afe044047\") " pod="kube-system/global-pull-secret-syncer-rbwl6"
Jun 02 15:49:40.889849 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:49:40.889813    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbwl6"
Jun 02 15:49:41.018383 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:49:41.018256    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rbwl6"]
Jun 02 15:49:41.021025 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:49:41.020975    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44384f36_6a8f_47b3_b329_083afe044047.slice/crio-b7091b7384d3fb1f7f27db9143f0b21700e6c382814f2ad8577b0aa766466f38 WatchSource:0}: Error finding container b7091b7384d3fb1f7f27db9143f0b21700e6c382814f2ad8577b0aa766466f38: Status 404 returned error can't find the container with id b7091b7384d3fb1f7f27db9143f0b21700e6c382814f2ad8577b0aa766466f38
Jun 02 15:49:41.691026 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:49:41.690972    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rbwl6" event={"ID":"44384f36-6a8f-47b3-b329-083afe044047","Type":"ContainerStarted","Data":"b7091b7384d3fb1f7f27db9143f0b21700e6c382814f2ad8577b0aa766466f38"}
Jun 02 15:49:45.706884 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:49:45.706841    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rbwl6" event={"ID":"44384f36-6a8f-47b3-b329-083afe044047","Type":"ContainerStarted","Data":"9eedabdaa00b643e0ec1d8be17ca9a5febc0b0ce44163707c809af69e9c9b593"}
Jun 02 15:49:45.731400 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:49:45.731344    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rbwl6" podStartSLOduration=1.236700077 podStartE2EDuration="5.731327456s" podCreationTimestamp="2026-06-02 15:49:40 +0000 UTC" firstStartedPulling="2026-06-02 15:49:41.022635595 +0000 UTC m=+229.725099093" lastFinishedPulling="2026-06-02 15:49:45.517262955 +0000 UTC m=+234.219726472" observedRunningTime="2026-06-02 15:49:45.729643868 +0000 UTC m=+234.432107398" watchObservedRunningTime="2026-06-02 15:49:45.731327456 +0000 UTC m=+234.433790975"
Jun 02 15:50:51.790132 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:50:51.790097    2578 kubelet.go:1628] "Image garbage collection succeeded"
Jun 02 15:53:53.682180 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:53:53.682138    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/s3-init-qb5bg"]
Jun 02 15:53:53.685072 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:53:53.685054    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-qb5bg"
Jun 02 15:53:53.689046 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:53:53.689023    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jun 02 15:53:53.689723 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:53:53.689707    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jun 02 15:53:53.689786 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:53:53.689714    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jun 02 15:53:53.689786 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:53:53.689763    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-8w2ng\""
Jun 02 15:53:53.735833 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:53:53.732398    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-qb5bg"]
Jun 02 15:53:53.743831 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:53:53.743793    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqchw\" (UniqueName: \"kubernetes.io/projected/0256bb0d-e0cf-483e-8e40-6e9ca793f7ec-kube-api-access-lqchw\") pod \"s3-init-qb5bg\" (UID: \"0256bb0d-e0cf-483e-8e40-6e9ca793f7ec\") " pod="kserve/s3-init-qb5bg"
Jun 02 15:53:53.845111 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:53:53.845068    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lqchw\" (UniqueName: \"kubernetes.io/projected/0256bb0d-e0cf-483e-8e40-6e9ca793f7ec-kube-api-access-lqchw\") pod \"s3-init-qb5bg\" (UID: \"0256bb0d-e0cf-483e-8e40-6e9ca793f7ec\") " pod="kserve/s3-init-qb5bg"
Jun 02 15:53:53.858840 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:53:53.858807    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqchw\" (UniqueName: \"kubernetes.io/projected/0256bb0d-e0cf-483e-8e40-6e9ca793f7ec-kube-api-access-lqchw\") pod \"s3-init-qb5bg\" (UID: \"0256bb0d-e0cf-483e-8e40-6e9ca793f7ec\") " pod="kserve/s3-init-qb5bg"
Jun 02 15:53:54.004971 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:53:54.004882    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-qb5bg"
Jun 02 15:53:54.129126 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:53:54.129098    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-qb5bg"]
Jun 02 15:53:54.131572 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:53:54.131543    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0256bb0d_e0cf_483e_8e40_6e9ca793f7ec.slice/crio-7153b64a15a47da3ec55233d7b0e79ffdbf8323fa54ab2ce47e99e28ee2f9dfb WatchSource:0}: Error finding container 7153b64a15a47da3ec55233d7b0e79ffdbf8323fa54ab2ce47e99e28ee2f9dfb: Status 404 returned error can't find the container with id 7153b64a15a47da3ec55233d7b0e79ffdbf8323fa54ab2ce47e99e28ee2f9dfb
Jun 02 15:53:54.133357 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:53:54.133337    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 15:53:54.406759 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:53:54.406720    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qb5bg" event={"ID":"0256bb0d-e0cf-483e-8e40-6e9ca793f7ec","Type":"ContainerStarted","Data":"7153b64a15a47da3ec55233d7b0e79ffdbf8323fa54ab2ce47e99e28ee2f9dfb"}
Jun 02 15:53:59.425114 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:53:59.425073    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qb5bg" event={"ID":"0256bb0d-e0cf-483e-8e40-6e9ca793f7ec","Type":"ContainerStarted","Data":"c0408dc5f28bb4377f70ad4002f11194add22b387eec0db7a2b08b3ee62c4f64"}
Jun 02 15:53:59.448656 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:53:59.448606    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/s3-init-qb5bg" podStartSLOduration=2.058163619 podStartE2EDuration="6.448590911s" podCreationTimestamp="2026-06-02 15:53:53 +0000 UTC" firstStartedPulling="2026-06-02 15:53:54.133496988 +0000 UTC m=+482.835960485" lastFinishedPulling="2026-06-02 15:53:58.523924263 +0000 UTC m=+487.226387777" observedRunningTime="2026-06-02 15:53:59.448096526 +0000 UTC m=+488.150560242" watchObservedRunningTime="2026-06-02 15:53:59.448590911 +0000 UTC m=+488.151054430"
Jun 02 15:54:02.435271 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:02.435234    2578 generic.go:358] "Generic (PLEG): container finished" podID="0256bb0d-e0cf-483e-8e40-6e9ca793f7ec" containerID="c0408dc5f28bb4377f70ad4002f11194add22b387eec0db7a2b08b3ee62c4f64" exitCode=0
Jun 02 15:54:02.435659 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:02.435278    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qb5bg" event={"ID":"0256bb0d-e0cf-483e-8e40-6e9ca793f7ec","Type":"ContainerDied","Data":"c0408dc5f28bb4377f70ad4002f11194add22b387eec0db7a2b08b3ee62c4f64"}
Jun 02 15:54:03.558507 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:03.558484    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-qb5bg"
Jun 02 15:54:03.632978 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:03.632943    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqchw\" (UniqueName: \"kubernetes.io/projected/0256bb0d-e0cf-483e-8e40-6e9ca793f7ec-kube-api-access-lqchw\") pod \"0256bb0d-e0cf-483e-8e40-6e9ca793f7ec\" (UID: \"0256bb0d-e0cf-483e-8e40-6e9ca793f7ec\") "
Jun 02 15:54:03.635252 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:03.635226    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0256bb0d-e0cf-483e-8e40-6e9ca793f7ec-kube-api-access-lqchw" (OuterVolumeSpecName: "kube-api-access-lqchw") pod "0256bb0d-e0cf-483e-8e40-6e9ca793f7ec" (UID: "0256bb0d-e0cf-483e-8e40-6e9ca793f7ec"). InnerVolumeSpecName "kube-api-access-lqchw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 15:54:03.734558 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:03.734470    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqchw\" (UniqueName: \"kubernetes.io/projected/0256bb0d-e0cf-483e-8e40-6e9ca793f7ec-kube-api-access-lqchw\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 15:54:04.442668 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:04.442637    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-qb5bg"
Jun 02 15:54:04.442668 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:04.442665    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qb5bg" event={"ID":"0256bb0d-e0cf-483e-8e40-6e9ca793f7ec","Type":"ContainerDied","Data":"7153b64a15a47da3ec55233d7b0e79ffdbf8323fa54ab2ce47e99e28ee2f9dfb"}
Jun 02 15:54:04.442889 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:04.442695    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7153b64a15a47da3ec55233d7b0e79ffdbf8323fa54ab2ce47e99e28ee2f9dfb"
Jun 02 15:54:14.148575 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:14.148525    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb"]
Jun 02 15:54:14.149026 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:14.148856    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0256bb0d-e0cf-483e-8e40-6e9ca793f7ec" containerName="s3-init"
Jun 02 15:54:14.149026 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:14.148869    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0256bb0d-e0cf-483e-8e40-6e9ca793f7ec" containerName="s3-init"
Jun 02 15:54:14.149026 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:14.148936    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="0256bb0d-e0cf-483e-8e40-6e9ca793f7ec" containerName="s3-init"
Jun 02 15:54:14.151112 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:14.151094    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb"
Jun 02 15:54:14.155550 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:14.155533    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-b5790-kube-rbac-proxy-sar-config\""
Jun 02 15:54:14.156396 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:14.156369    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jun 02 15:54:14.156507 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:14.156378    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-b5790-predictor-serving-cert\""
Jun 02 15:54:14.156824 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:14.156808    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-l65kw\""
Jun 02 15:54:14.157441 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:14.157134    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jun 02 15:54:14.185827 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:14.185798    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb"]
Jun 02 15:54:14.225211 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:14.225170    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-raw-sklearn-batcher-b5790-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75a18a34-f0af-43bd-bcf3-82f5c8623661-isvc-raw-sklearn-batcher-b5790-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb\" (UID: \"75a18a34-f0af-43bd-bcf3-82f5c8623661\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb"
Jun 02 15:54:14.225398 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:14.225223    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swlgt\" (UniqueName: \"kubernetes.io/projected/75a18a34-f0af-43bd-bcf3-82f5c8623661-kube-api-access-swlgt\") pod \"isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb\" (UID: \"75a18a34-f0af-43bd-bcf3-82f5c8623661\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb"
Jun 02 15:54:14.225398 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:14.225296    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75a18a34-f0af-43bd-bcf3-82f5c8623661-proxy-tls\") pod \"isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb\" (UID: \"75a18a34-f0af-43bd-bcf3-82f5c8623661\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb"
Jun 02 15:54:14.225398 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:14.225344    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75a18a34-f0af-43bd-bcf3-82f5c8623661-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb\" (UID: \"75a18a34-f0af-43bd-bcf3-82f5c8623661\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb"
Jun 02 15:54:14.326360 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:14.326320    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75a18a34-f0af-43bd-bcf3-82f5c8623661-proxy-tls\") pod \"isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb\" (UID: \"75a18a34-f0af-43bd-bcf3-82f5c8623661\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb"
Jun 02 15:54:14.326360 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:14.326372    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75a18a34-f0af-43bd-bcf3-82f5c8623661-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb\" (UID: \"75a18a34-f0af-43bd-bcf3-82f5c8623661\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb"
Jun 02 15:54:14.326619 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:14.326406    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-raw-sklearn-batcher-b5790-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75a18a34-f0af-43bd-bcf3-82f5c8623661-isvc-raw-sklearn-batcher-b5790-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb\" (UID: \"75a18a34-f0af-43bd-bcf3-82f5c8623661\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb"
Jun 02 15:54:14.326619 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:14.326439    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-swlgt\" (UniqueName: \"kubernetes.io/projected/75a18a34-f0af-43bd-bcf3-82f5c8623661-kube-api-access-swlgt\") pod \"isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb\" (UID: \"75a18a34-f0af-43bd-bcf3-82f5c8623661\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb"
Jun 02 15:54:14.326830 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:14.326806    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75a18a34-f0af-43bd-bcf3-82f5c8623661-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb\" (UID: \"75a18a34-f0af-43bd-bcf3-82f5c8623661\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb"
Jun 02 15:54:14.327164 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:14.327143    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-raw-sklearn-batcher-b5790-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75a18a34-f0af-43bd-bcf3-82f5c8623661-isvc-raw-sklearn-batcher-b5790-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb\" (UID: \"75a18a34-f0af-43bd-bcf3-82f5c8623661\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb"
Jun 02 15:54:14.328807 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:14.328788    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75a18a34-f0af-43bd-bcf3-82f5c8623661-proxy-tls\") pod \"isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb\" (UID: \"75a18a34-f0af-43bd-bcf3-82f5c8623661\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb"
Jun 02 15:54:14.340514 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:14.340482    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swlgt\" (UniqueName: \"kubernetes.io/projected/75a18a34-f0af-43bd-bcf3-82f5c8623661-kube-api-access-swlgt\") pod \"isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb\" (UID: \"75a18a34-f0af-43bd-bcf3-82f5c8623661\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb"
Jun 02 15:54:14.461245 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:14.461152    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb"
Jun 02 15:54:14.600701 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:14.600662    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb"]
Jun 02 15:54:14.603684 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:54:14.603651    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75a18a34_f0af_43bd_bcf3_82f5c8623661.slice/crio-6082bd6389fe6b165a7581a3df00374d621303c5abd5051adce77dd4d4405db5 WatchSource:0}: Error finding container 6082bd6389fe6b165a7581a3df00374d621303c5abd5051adce77dd4d4405db5: Status 404 returned error can't find the container with id 6082bd6389fe6b165a7581a3df00374d621303c5abd5051adce77dd4d4405db5
Jun 02 15:54:15.476293 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:15.476252    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb" event={"ID":"75a18a34-f0af-43bd-bcf3-82f5c8623661","Type":"ContainerStarted","Data":"6082bd6389fe6b165a7581a3df00374d621303c5abd5051adce77dd4d4405db5"}
Jun 02 15:54:19.490233 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:19.490202    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb" event={"ID":"75a18a34-f0af-43bd-bcf3-82f5c8623661","Type":"ContainerStarted","Data":"4c3758cd68e32fef1c9b6040ad2920ba55189ea307d0a0fcc5bbb08327abe36e"}
Jun 02 15:54:23.503271 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:23.503237    2578 generic.go:358] "Generic (PLEG): container finished" podID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerID="4c3758cd68e32fef1c9b6040ad2920ba55189ea307d0a0fcc5bbb08327abe36e" exitCode=0
Jun 02 15:54:23.503647 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:23.503277    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb" event={"ID":"75a18a34-f0af-43bd-bcf3-82f5c8623661","Type":"ContainerDied","Data":"4c3758cd68e32fef1c9b6040ad2920ba55189ea307d0a0fcc5bbb08327abe36e"}
Jun 02 15:54:38.562256 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:38.562217    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb" event={"ID":"75a18a34-f0af-43bd-bcf3-82f5c8623661","Type":"ContainerStarted","Data":"0c0a8e6bb6920b762cf1e03d705947c11e6d31c1395080595a7ab09036a32d0d"}
Jun 02 15:54:40.571769 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:40.571685    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb" event={"ID":"75a18a34-f0af-43bd-bcf3-82f5c8623661","Type":"ContainerStarted","Data":"b59de275b5ddcce5ca789876e86b5450cb9e7c7e14526b16a7f4526792910521"}
Jun 02 15:54:44.587127 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:44.587088    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb" event={"ID":"75a18a34-f0af-43bd-bcf3-82f5c8623661","Type":"ContainerStarted","Data":"1964d0adaec092dd8450ded2b8db3f28053d3d06b9cf069e64e3c38ac550256a"}
Jun 02 15:54:44.587570 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:44.587320    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb"
Jun 02 15:54:44.587570 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:44.587441    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb"
Jun 02 15:54:44.588691 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:44.588661    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jun 02 15:54:44.617065 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:44.616990    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb" podStartSLOduration=1.158469351 podStartE2EDuration="30.616973649s" podCreationTimestamp="2026-06-02 15:54:14 +0000 UTC" firstStartedPulling="2026-06-02 15:54:14.605633931 +0000 UTC m=+503.308097430" lastFinishedPulling="2026-06-02 15:54:44.064138227 +0000 UTC m=+532.766601728" observedRunningTime="2026-06-02 15:54:44.613925624 +0000 UTC m=+533.316389173" watchObservedRunningTime="2026-06-02 15:54:44.616973649 +0000 UTC m=+533.319437168"
Jun 02 15:54:45.590145 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:45.590109    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb"
Jun 02 15:54:45.590560 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:45.590210    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jun 02 15:54:45.591083 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:45.591058    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 15:54:45.593751 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:45.593735    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb"
Jun 02 15:54:46.593194 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:46.593154    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jun 02 15:54:46.593583 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:46.593402    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 15:54:47.596252 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:47.596210    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jun 02 15:54:47.596643 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:47.596563    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 15:54:57.596821 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:57.596772    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jun 02 15:54:57.597378 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:54:57.597145    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 15:55:07.597117 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:07.597067    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jun 02 15:55:07.597663 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:07.597523    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 15:55:17.596551 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:17.596507    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jun 02 15:55:17.597057 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:17.596884    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 15:55:27.596239 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:27.596194    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jun 02 15:55:27.596709 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:27.596688    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 15:55:37.597068 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:37.597017    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jun 02 15:55:37.597506 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:37.597476    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 15:55:47.596744 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:47.596710    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb"
Jun 02 15:55:47.597209 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:47.596863    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb"
Jun 02 15:55:58.919582 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:58.919542    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb"]
Jun 02 15:55:58.920146 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:58.920052    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerName="kserve-container" containerID="cri-o://0c0a8e6bb6920b762cf1e03d705947c11e6d31c1395080595a7ab09036a32d0d" gracePeriod=30
Jun 02 15:55:58.920146 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:58.920065    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerName="agent" containerID="cri-o://1964d0adaec092dd8450ded2b8db3f28053d3d06b9cf069e64e3c38ac550256a" gracePeriod=30
Jun 02 15:55:58.920146 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:58.920104    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerName="kube-rbac-proxy" containerID="cri-o://b59de275b5ddcce5ca789876e86b5450cb9e7c7e14526b16a7f4526792910521" gracePeriod=30
Jun 02 15:55:59.079622 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.079591    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm"]
Jun 02 15:55:59.082104 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.082085    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm"
Jun 02 15:55:59.084642 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.084614    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-a7b79-predictor-serving-cert\""
Jun 02 15:55:59.084775 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.084693    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-a7b79-kube-rbac-proxy-sar-config\""
Jun 02 15:55:59.095543 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.095515    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm"]
Jun 02 15:55:59.135440 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.135407    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qsk9\" (UniqueName: \"kubernetes.io/projected/7fa53da0-d6c8-4d23-bfb8-cd87a7980cde-kube-api-access-5qsk9\") pod \"isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm\" (UID: \"7fa53da0-d6c8-4d23-bfb8-cd87a7980cde\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm"
Jun 02 15:55:59.135605 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.135470    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fa53da0-d6c8-4d23-bfb8-cd87a7980cde-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm\" (UID: \"7fa53da0-d6c8-4d23-bfb8-cd87a7980cde\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm"
Jun 02 15:55:59.135605 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.135560    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fa53da0-d6c8-4d23-bfb8-cd87a7980cde-proxy-tls\") pod \"isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm\" (UID: \"7fa53da0-d6c8-4d23-bfb8-cd87a7980cde\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm"
Jun 02 15:55:59.135705 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.135640    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-graph-raw-a7b79-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7fa53da0-d6c8-4d23-bfb8-cd87a7980cde-isvc-sklearn-graph-raw-a7b79-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm\" (UID: \"7fa53da0-d6c8-4d23-bfb8-cd87a7980cde\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm"
Jun 02 15:55:59.188366 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.188291    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5"]
Jun 02 15:55:59.190828 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.190811    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5"
Jun 02 15:55:59.193185 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.193166    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-a7b79-kube-rbac-proxy-sar-config\""
Jun 02 15:55:59.193333 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.193313    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-a7b79-predictor-serving-cert\""
Jun 02 15:55:59.205041 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.205018    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5"]
Jun 02 15:55:59.236406 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.236378    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-graph-raw-a7b79-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7fa53da0-d6c8-4d23-bfb8-cd87a7980cde-isvc-sklearn-graph-raw-a7b79-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm\" (UID: \"7fa53da0-d6c8-4d23-bfb8-cd87a7980cde\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm"
Jun 02 15:55:59.236573 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.236421    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5qsk9\" (UniqueName: \"kubernetes.io/projected/7fa53da0-d6c8-4d23-bfb8-cd87a7980cde-kube-api-access-5qsk9\") pod \"isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm\" (UID: \"7fa53da0-d6c8-4d23-bfb8-cd87a7980cde\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm"
Jun 02 15:55:59.236573 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.236471    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fa53da0-d6c8-4d23-bfb8-cd87a7980cde-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm\" (UID: \"7fa53da0-d6c8-4d23-bfb8-cd87a7980cde\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm"
Jun 02 15:55:59.236573 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.236515    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fa53da0-d6c8-4d23-bfb8-cd87a7980cde-proxy-tls\") pod \"isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm\" (UID: \"7fa53da0-d6c8-4d23-bfb8-cd87a7980cde\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm"
Jun 02 15:55:59.236909 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.236883    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fa53da0-d6c8-4d23-bfb8-cd87a7980cde-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm\" (UID: \"7fa53da0-d6c8-4d23-bfb8-cd87a7980cde\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm"
Jun 02 15:55:59.237131 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.237110    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-graph-raw-a7b79-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7fa53da0-d6c8-4d23-bfb8-cd87a7980cde-isvc-sklearn-graph-raw-a7b79-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm\" (UID: \"7fa53da0-d6c8-4d23-bfb8-cd87a7980cde\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm"
Jun 02 15:55:59.239157 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.239133    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fa53da0-d6c8-4d23-bfb8-cd87a7980cde-proxy-tls\") pod \"isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm\" (UID: \"7fa53da0-d6c8-4d23-bfb8-cd87a7980cde\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm"
Jun 02 15:55:59.246552 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.246530    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qsk9\" (UniqueName: \"kubernetes.io/projected/7fa53da0-d6c8-4d23-bfb8-cd87a7980cde-kube-api-access-5qsk9\") pod \"isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm\" (UID: \"7fa53da0-d6c8-4d23-bfb8-cd87a7980cde\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm"
Jun 02 15:55:59.337245 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.337203    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5df37958-2286-4d27-898c-9628699e3ff7-proxy-tls\") pod \"isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5\" (UID: \"5df37958-2286-4d27-898c-9628699e3ff7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5"
Jun 02 15:55:59.337427 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.337257    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wccjl\" (UniqueName: \"kubernetes.io/projected/5df37958-2286-4d27-898c-9628699e3ff7-kube-api-access-wccjl\") pod \"isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5\" (UID: \"5df37958-2286-4d27-898c-9628699e3ff7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5"
Jun 02 15:55:59.337427 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.337355    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5df37958-2286-4d27-898c-9628699e3ff7-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5\" (UID: \"5df37958-2286-4d27-898c-9628699e3ff7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5"
Jun 02 15:55:59.337427 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.337401    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-graph-raw-a7b79-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5df37958-2286-4d27-898c-9628699e3ff7-isvc-xgboost-graph-raw-a7b79-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5\" (UID: \"5df37958-2286-4d27-898c-9628699e3ff7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5"
Jun 02 15:55:59.394285 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.394246    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm"
Jun 02 15:55:59.438561 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.438478    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wccjl\" (UniqueName: \"kubernetes.io/projected/5df37958-2286-4d27-898c-9628699e3ff7-kube-api-access-wccjl\") pod \"isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5\" (UID: \"5df37958-2286-4d27-898c-9628699e3ff7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5"
Jun 02 15:55:59.438695 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.438582    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5df37958-2286-4d27-898c-9628699e3ff7-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5\" (UID: \"5df37958-2286-4d27-898c-9628699e3ff7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5"
Jun 02 15:55:59.438695 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.438637    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-graph-raw-a7b79-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5df37958-2286-4d27-898c-9628699e3ff7-isvc-xgboost-graph-raw-a7b79-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5\" (UID: \"5df37958-2286-4d27-898c-9628699e3ff7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5"
Jun 02 15:55:59.438769 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.438697    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5df37958-2286-4d27-898c-9628699e3ff7-proxy-tls\") pod \"isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5\" (UID: \"5df37958-2286-4d27-898c-9628699e3ff7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5"
Jun 02 15:55:59.438946 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.438926    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5df37958-2286-4d27-898c-9628699e3ff7-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5\" (UID: \"5df37958-2286-4d27-898c-9628699e3ff7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5"
Jun 02 15:55:59.439301 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.439276    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-graph-raw-a7b79-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5df37958-2286-4d27-898c-9628699e3ff7-isvc-xgboost-graph-raw-a7b79-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5\" (UID: \"5df37958-2286-4d27-898c-9628699e3ff7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5"
Jun 02 15:55:59.441438 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.441248    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5df37958-2286-4d27-898c-9628699e3ff7-proxy-tls\") pod \"isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5\" (UID: \"5df37958-2286-4d27-898c-9628699e3ff7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5"
Jun 02 15:55:59.448370 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.448341    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wccjl\" (UniqueName: \"kubernetes.io/projected/5df37958-2286-4d27-898c-9628699e3ff7-kube-api-access-wccjl\") pod \"isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5\" (UID: \"5df37958-2286-4d27-898c-9628699e3ff7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5"
Jun 02 15:55:59.501861 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.501834    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5"
Jun 02 15:55:59.533119 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.532958    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm"]
Jun 02 15:55:59.540312 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:55:59.540282    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa53da0_d6c8_4d23_bfb8_cd87a7980cde.slice/crio-e14e76c06cdea8e46a0c0474c69f358bc057e1838333348945355a70246427a5 WatchSource:0}: Error finding container e14e76c06cdea8e46a0c0474c69f358bc057e1838333348945355a70246427a5: Status 404 returned error can't find the container with id e14e76c06cdea8e46a0c0474c69f358bc057e1838333348945355a70246427a5
Jun 02 15:55:59.633783 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.633752    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5"]
Jun 02 15:55:59.637186 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:55:59.637162    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5df37958_2286_4d27_898c_9628699e3ff7.slice/crio-7e26de8e71eea95d3029f7fde906b81be7cec18f24ca14aaf896b6d595f76d55 WatchSource:0}: Error finding container 7e26de8e71eea95d3029f7fde906b81be7cec18f24ca14aaf896b6d595f76d55: Status 404 returned error can't find the container with id 7e26de8e71eea95d3029f7fde906b81be7cec18f24ca14aaf896b6d595f76d55
Jun 02 15:55:59.810957 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.810880    2578 generic.go:358] "Generic (PLEG): container finished" podID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerID="b59de275b5ddcce5ca789876e86b5450cb9e7c7e14526b16a7f4526792910521" exitCode=2
Jun 02 15:55:59.811148 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.810963    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb" event={"ID":"75a18a34-f0af-43bd-bcf3-82f5c8623661","Type":"ContainerDied","Data":"b59de275b5ddcce5ca789876e86b5450cb9e7c7e14526b16a7f4526792910521"}
Jun 02 15:55:59.812325 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.812302    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5" event={"ID":"5df37958-2286-4d27-898c-9628699e3ff7","Type":"ContainerStarted","Data":"96574d12277239467cf0e853446c097977554afd46dc51c42351034ee81e9a96"}
Jun 02 15:55:59.812444 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.812332    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5" event={"ID":"5df37958-2286-4d27-898c-9628699e3ff7","Type":"ContainerStarted","Data":"7e26de8e71eea95d3029f7fde906b81be7cec18f24ca14aaf896b6d595f76d55"}
Jun 02 15:55:59.813700 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.813680    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm" event={"ID":"7fa53da0-d6c8-4d23-bfb8-cd87a7980cde","Type":"ContainerStarted","Data":"203b68dccca5b5b6bfd7d3ea58ce8677ce36680bfc711e824d2627993a6e2d99"}
Jun 02 15:55:59.813700 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:55:59.813705    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm" event={"ID":"7fa53da0-d6c8-4d23-bfb8-cd87a7980cde","Type":"ContainerStarted","Data":"e14e76c06cdea8e46a0c0474c69f358bc057e1838333348945355a70246427a5"}
Jun 02 15:56:00.590616 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:00.590574    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused"
Jun 02 15:56:03.827416 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:03.827325    2578 generic.go:358] "Generic (PLEG): container finished" podID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerID="0c0a8e6bb6920b762cf1e03d705947c11e6d31c1395080595a7ab09036a32d0d" exitCode=0
Jun 02 15:56:03.827416 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:03.827397    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb" event={"ID":"75a18a34-f0af-43bd-bcf3-82f5c8623661","Type":"ContainerDied","Data":"0c0a8e6bb6920b762cf1e03d705947c11e6d31c1395080595a7ab09036a32d0d"}
Jun 02 15:56:03.828678 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:03.828656    2578 generic.go:358] "Generic (PLEG): container finished" podID="5df37958-2286-4d27-898c-9628699e3ff7" containerID="96574d12277239467cf0e853446c097977554afd46dc51c42351034ee81e9a96" exitCode=0
Jun 02 15:56:03.828793 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:03.828713    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5" event={"ID":"5df37958-2286-4d27-898c-9628699e3ff7","Type":"ContainerDied","Data":"96574d12277239467cf0e853446c097977554afd46dc51c42351034ee81e9a96"}
Jun 02 15:56:03.829939 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:03.829917    2578 generic.go:358] "Generic (PLEG): container finished" podID="7fa53da0-d6c8-4d23-bfb8-cd87a7980cde" containerID="203b68dccca5b5b6bfd7d3ea58ce8677ce36680bfc711e824d2627993a6e2d99" exitCode=0
Jun 02 15:56:03.830026 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:03.830010    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm" event={"ID":"7fa53da0-d6c8-4d23-bfb8-cd87a7980cde","Type":"ContainerDied","Data":"203b68dccca5b5b6bfd7d3ea58ce8677ce36680bfc711e824d2627993a6e2d99"}
Jun 02 15:56:04.836084 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:04.836044    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm" event={"ID":"7fa53da0-d6c8-4d23-bfb8-cd87a7980cde","Type":"ContainerStarted","Data":"5a791f5ce7d4077608a4803221266482051d639167c894765f0417103b607f0f"}
Jun 02 15:56:04.836604 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:04.836103    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm" event={"ID":"7fa53da0-d6c8-4d23-bfb8-cd87a7980cde","Type":"ContainerStarted","Data":"512e455f5f0f20eb9c2063e5317154dcfd38980dbeb6b709d023d2f4bcc3e90a"}
Jun 02 15:56:04.836604 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:04.836356    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm"
Jun 02 15:56:04.860747 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:04.860680    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm" podStartSLOduration=5.86066066 podStartE2EDuration="5.86066066s" podCreationTimestamp="2026-06-02 15:55:59 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 15:56:04.85827329 +0000 UTC m=+613.560736810" watchObservedRunningTime="2026-06-02 15:56:04.86066066 +0000 UTC m=+613.563124180"
Jun 02 15:56:05.590792 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:05.590719    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused"
Jun 02 15:56:05.839918 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:05.839883    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm"
Jun 02 15:56:05.841279 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:05.841208    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm" podUID="7fa53da0-d6c8-4d23-bfb8-cd87a7980cde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jun 02 15:56:06.842989 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:06.842943    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm" podUID="7fa53da0-d6c8-4d23-bfb8-cd87a7980cde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jun 02 15:56:07.597279 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:07.597114    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jun 02 15:56:07.597494 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:07.597465    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 15:56:10.590949 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:10.590852    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused"
Jun 02 15:56:10.591461 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:10.591032    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb"
Jun 02 15:56:11.849274 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:11.849200    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm"
Jun 02 15:56:11.849787 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:11.849753    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm" podUID="7fa53da0-d6c8-4d23-bfb8-cd87a7980cde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jun 02 15:56:15.590769 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:15.590718    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused"
Jun 02 15:56:17.596984 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:17.596929    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jun 02 15:56:17.597467 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:17.597300    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 15:56:20.591071 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:20.590952    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused"
Jun 02 15:56:20.892317 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:20.892236    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5" event={"ID":"5df37958-2286-4d27-898c-9628699e3ff7","Type":"ContainerStarted","Data":"4640e288ae3931721bfcb7a181e534388c4447d0027631e5aad87722e3de319b"}
Jun 02 15:56:20.892317 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:20.892280    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5" event={"ID":"5df37958-2286-4d27-898c-9628699e3ff7","Type":"ContainerStarted","Data":"c0e511a58b95904c1d970a72a807eaa4e8338ee1dfaf97144ef811b59bfb8f8f"}
Jun 02 15:56:20.892516 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:20.892496    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5"
Jun 02 15:56:20.919406 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:20.919360    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5" podStartSLOduration=5.539580937 podStartE2EDuration="21.919347756s" podCreationTimestamp="2026-06-02 15:55:59 +0000 UTC" firstStartedPulling="2026-06-02 15:56:03.830113264 +0000 UTC m=+612.532576764" lastFinishedPulling="2026-06-02 15:56:20.209880082 +0000 UTC m=+628.912343583" observedRunningTime="2026-06-02 15:56:20.91750412 +0000 UTC m=+629.619967640" watchObservedRunningTime="2026-06-02 15:56:20.919347756 +0000 UTC m=+629.621811275"
Jun 02 15:56:21.849940 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:21.849905    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm" podUID="7fa53da0-d6c8-4d23-bfb8-cd87a7980cde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jun 02 15:56:21.895769 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:21.895743    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5"
Jun 02 15:56:21.896900 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:21.896872    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5" podUID="5df37958-2286-4d27-898c-9628699e3ff7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jun 02 15:56:22.899453 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:22.899417    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5" podUID="5df37958-2286-4d27-898c-9628699e3ff7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jun 02 15:56:25.591005 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:25.590949    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused"
Jun 02 15:56:27.596626 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:27.596582    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jun 02 15:56:27.597107 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:27.596734    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb"
Jun 02 15:56:27.597107 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:27.596922    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 15:56:27.597107 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:27.596990    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb"
Jun 02 15:56:27.908003 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:27.907896    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5"
Jun 02 15:56:28.491282 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:27.908396    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5" podUID="5df37958-2286-4d27-898c-9628699e3ff7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jun 02 15:56:29.622754 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:29.622733    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb"
Jun 02 15:56:29.719407 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:29.719353    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-raw-sklearn-batcher-b5790-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75a18a34-f0af-43bd-bcf3-82f5c8623661-isvc-raw-sklearn-batcher-b5790-kube-rbac-proxy-sar-config\") pod \"75a18a34-f0af-43bd-bcf3-82f5c8623661\" (UID: \"75a18a34-f0af-43bd-bcf3-82f5c8623661\") "
Jun 02 15:56:29.719607 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:29.719415    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75a18a34-f0af-43bd-bcf3-82f5c8623661-kserve-provision-location\") pod \"75a18a34-f0af-43bd-bcf3-82f5c8623661\" (UID: \"75a18a34-f0af-43bd-bcf3-82f5c8623661\") "
Jun 02 15:56:29.719607 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:29.719449    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75a18a34-f0af-43bd-bcf3-82f5c8623661-proxy-tls\") pod \"75a18a34-f0af-43bd-bcf3-82f5c8623661\" (UID: \"75a18a34-f0af-43bd-bcf3-82f5c8623661\") "
Jun 02 15:56:29.719607 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:29.719570    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swlgt\" (UniqueName: \"kubernetes.io/projected/75a18a34-f0af-43bd-bcf3-82f5c8623661-kube-api-access-swlgt\") pod \"75a18a34-f0af-43bd-bcf3-82f5c8623661\" (UID: \"75a18a34-f0af-43bd-bcf3-82f5c8623661\") "
Jun 02 15:56:29.719772 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:29.719721    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75a18a34-f0af-43bd-bcf3-82f5c8623661-isvc-raw-sklearn-batcher-b5790-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-raw-sklearn-batcher-b5790-kube-rbac-proxy-sar-config") pod "75a18a34-f0af-43bd-bcf3-82f5c8623661" (UID: "75a18a34-f0af-43bd-bcf3-82f5c8623661"). InnerVolumeSpecName "isvc-raw-sklearn-batcher-b5790-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:56:29.719841 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:29.719787    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a18a34-f0af-43bd-bcf3-82f5c8623661-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "75a18a34-f0af-43bd-bcf3-82f5c8623661" (UID: "75a18a34-f0af-43bd-bcf3-82f5c8623661"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 15:56:29.719841 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:29.719804    2578 reconciler_common.go:299] "Volume detached for volume \"isvc-raw-sklearn-batcher-b5790-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75a18a34-f0af-43bd-bcf3-82f5c8623661-isvc-raw-sklearn-batcher-b5790-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 15:56:29.721779 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:29.721757    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75a18a34-f0af-43bd-bcf3-82f5c8623661-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "75a18a34-f0af-43bd-bcf3-82f5c8623661" (UID: "75a18a34-f0af-43bd-bcf3-82f5c8623661"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 15:56:29.721871 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:29.721788    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a18a34-f0af-43bd-bcf3-82f5c8623661-kube-api-access-swlgt" (OuterVolumeSpecName: "kube-api-access-swlgt") pod "75a18a34-f0af-43bd-bcf3-82f5c8623661" (UID: "75a18a34-f0af-43bd-bcf3-82f5c8623661"). InnerVolumeSpecName "kube-api-access-swlgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 15:56:29.820672 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:29.820587    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75a18a34-f0af-43bd-bcf3-82f5c8623661-kserve-provision-location\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 15:56:29.820672 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:29.820621    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75a18a34-f0af-43bd-bcf3-82f5c8623661-proxy-tls\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 15:56:29.820672 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:29.820632    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swlgt\" (UniqueName: \"kubernetes.io/projected/75a18a34-f0af-43bd-bcf3-82f5c8623661-kube-api-access-swlgt\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 15:56:29.922347 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:29.922316    2578 generic.go:358] "Generic (PLEG): container finished" podID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerID="1964d0adaec092dd8450ded2b8db3f28053d3d06b9cf069e64e3c38ac550256a" exitCode=0
Jun 02 15:56:29.922484 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:29.922396    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb"
Jun 02 15:56:29.922484 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:29.922395    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb" event={"ID":"75a18a34-f0af-43bd-bcf3-82f5c8623661","Type":"ContainerDied","Data":"1964d0adaec092dd8450ded2b8db3f28053d3d06b9cf069e64e3c38ac550256a"}
Jun 02 15:56:29.922484 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:29.922430    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb" event={"ID":"75a18a34-f0af-43bd-bcf3-82f5c8623661","Type":"ContainerDied","Data":"6082bd6389fe6b165a7581a3df00374d621303c5abd5051adce77dd4d4405db5"}
Jun 02 15:56:29.922484 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:29.922445    2578 scope.go:117] "RemoveContainer" containerID="1964d0adaec092dd8450ded2b8db3f28053d3d06b9cf069e64e3c38ac550256a"
Jun 02 15:56:29.931386 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:29.931126    2578 scope.go:117] "RemoveContainer" containerID="b59de275b5ddcce5ca789876e86b5450cb9e7c7e14526b16a7f4526792910521"
Jun 02 15:56:29.938453 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:29.938432    2578 scope.go:117] "RemoveContainer" containerID="0c0a8e6bb6920b762cf1e03d705947c11e6d31c1395080595a7ab09036a32d0d"
Jun 02 15:56:29.945129 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:29.945107    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb"]
Jun 02 15:56:29.945815 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:29.945800    2578 scope.go:117] "RemoveContainer" containerID="4c3758cd68e32fef1c9b6040ad2920ba55189ea307d0a0fcc5bbb08327abe36e"
Jun 02 15:56:29.953516 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:29.953500    2578 scope.go:117] "RemoveContainer" containerID="1964d0adaec092dd8450ded2b8db3f28053d3d06b9cf069e64e3c38ac550256a"
Jun 02 15:56:29.953793 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:56:29.953764    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1964d0adaec092dd8450ded2b8db3f28053d3d06b9cf069e64e3c38ac550256a\": container with ID starting with 1964d0adaec092dd8450ded2b8db3f28053d3d06b9cf069e64e3c38ac550256a not found: ID does not exist" containerID="1964d0adaec092dd8450ded2b8db3f28053d3d06b9cf069e64e3c38ac550256a"
Jun 02 15:56:29.953873 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:29.953793    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1964d0adaec092dd8450ded2b8db3f28053d3d06b9cf069e64e3c38ac550256a"} err="failed to get container status \"1964d0adaec092dd8450ded2b8db3f28053d3d06b9cf069e64e3c38ac550256a\": rpc error: code = NotFound desc = could not find container \"1964d0adaec092dd8450ded2b8db3f28053d3d06b9cf069e64e3c38ac550256a\": container with ID starting with 1964d0adaec092dd8450ded2b8db3f28053d3d06b9cf069e64e3c38ac550256a not found: ID does not exist"
Jun 02 15:56:29.953873 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:29.953811    2578 scope.go:117] "RemoveContainer" containerID="b59de275b5ddcce5ca789876e86b5450cb9e7c7e14526b16a7f4526792910521"
Jun 02 15:56:29.954069 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:56:29.954039    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b59de275b5ddcce5ca789876e86b5450cb9e7c7e14526b16a7f4526792910521\": container with ID starting with b59de275b5ddcce5ca789876e86b5450cb9e7c7e14526b16a7f4526792910521 not found: ID does not exist" containerID="b59de275b5ddcce5ca789876e86b5450cb9e7c7e14526b16a7f4526792910521"
Jun 02 15:56:29.954176 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:29.954069    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b59de275b5ddcce5ca789876e86b5450cb9e7c7e14526b16a7f4526792910521"} err="failed to get container status \"b59de275b5ddcce5ca789876e86b5450cb9e7c7e14526b16a7f4526792910521\": rpc error: code = NotFound desc = could not find container \"b59de275b5ddcce5ca789876e86b5450cb9e7c7e14526b16a7f4526792910521\": container with ID starting with b59de275b5ddcce5ca789876e86b5450cb9e7c7e14526b16a7f4526792910521 not found: ID does not exist"
Jun 02 15:56:29.954176 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:29.954085    2578 scope.go:117] "RemoveContainer" containerID="0c0a8e6bb6920b762cf1e03d705947c11e6d31c1395080595a7ab09036a32d0d"
Jun 02 15:56:29.954335 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:56:29.954318    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c0a8e6bb6920b762cf1e03d705947c11e6d31c1395080595a7ab09036a32d0d\": container with ID starting with 0c0a8e6bb6920b762cf1e03d705947c11e6d31c1395080595a7ab09036a32d0d not found: ID does not exist" containerID="0c0a8e6bb6920b762cf1e03d705947c11e6d31c1395080595a7ab09036a32d0d"
Jun 02 15:56:29.954394 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:29.954342    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c0a8e6bb6920b762cf1e03d705947c11e6d31c1395080595a7ab09036a32d0d"} err="failed to get container status \"0c0a8e6bb6920b762cf1e03d705947c11e6d31c1395080595a7ab09036a32d0d\": rpc error: code = NotFound desc = could not find container \"0c0a8e6bb6920b762cf1e03d705947c11e6d31c1395080595a7ab09036a32d0d\": container with ID starting with 0c0a8e6bb6920b762cf1e03d705947c11e6d31c1395080595a7ab09036a32d0d not found: ID does not exist"
Jun 02 15:56:29.954394 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:29.954358    2578 scope.go:117] "RemoveContainer" containerID="4c3758cd68e32fef1c9b6040ad2920ba55189ea307d0a0fcc5bbb08327abe36e"
Jun 02 15:56:29.954604 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:56:29.954585    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c3758cd68e32fef1c9b6040ad2920ba55189ea307d0a0fcc5bbb08327abe36e\": container with ID starting with 4c3758cd68e32fef1c9b6040ad2920ba55189ea307d0a0fcc5bbb08327abe36e not found: ID does not exist" containerID="4c3758cd68e32fef1c9b6040ad2920ba55189ea307d0a0fcc5bbb08327abe36e"
Jun 02 15:56:29.954659 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:29.954613    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c3758cd68e32fef1c9b6040ad2920ba55189ea307d0a0fcc5bbb08327abe36e"} err="failed to get container status \"4c3758cd68e32fef1c9b6040ad2920ba55189ea307d0a0fcc5bbb08327abe36e\": rpc error: code = NotFound desc = could not find container \"4c3758cd68e32fef1c9b6040ad2920ba55189ea307d0a0fcc5bbb08327abe36e\": container with ID starting with 4c3758cd68e32fef1c9b6040ad2920ba55189ea307d0a0fcc5bbb08327abe36e not found: ID does not exist"
Jun 02 15:56:29.954659 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:29.954647    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b5790-predictor-58fd77887b-q6smb"]
Jun 02 15:56:31.850565 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:31.850526    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm" podUID="7fa53da0-d6c8-4d23-bfb8-cd87a7980cde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jun 02 15:56:31.907954 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:31.907925    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" path="/var/lib/kubelet/pods/75a18a34-f0af-43bd-bcf3-82f5c8623661/volumes"
Jun 02 15:56:37.909268 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:37.909223    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5" podUID="5df37958-2286-4d27-898c-9628699e3ff7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jun 02 15:56:41.850041 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:41.849976    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm" podUID="7fa53da0-d6c8-4d23-bfb8-cd87a7980cde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jun 02 15:56:47.908440 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:47.908403    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5" podUID="5df37958-2286-4d27-898c-9628699e3ff7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jun 02 15:56:51.850845 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:51.850794    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm" podUID="7fa53da0-d6c8-4d23-bfb8-cd87a7980cde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jun 02 15:56:57.908528 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:56:57.908488    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5" podUID="5df37958-2286-4d27-898c-9628699e3ff7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jun 02 15:57:01.849839 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:01.849792    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm" podUID="7fa53da0-d6c8-4d23-bfb8-cd87a7980cde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jun 02 15:57:07.908940 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:07.908902    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5" podUID="5df37958-2286-4d27-898c-9628699e3ff7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jun 02 15:57:11.850506 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:11.850459    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm" podUID="7fa53da0-d6c8-4d23-bfb8-cd87a7980cde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jun 02 15:57:12.904775 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:12.904734    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm" podUID="7fa53da0-d6c8-4d23-bfb8-cd87a7980cde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jun 02 15:57:17.908606 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:17.908569    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5" podUID="5df37958-2286-4d27-898c-9628699e3ff7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jun 02 15:57:22.906215 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:22.906182    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm"
Jun 02 15:57:27.909769 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:27.909734    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5"
Jun 02 15:57:49.774391 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:49.774298    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm"]
Jun 02 15:57:49.774905 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:49.774730    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm" podUID="7fa53da0-d6c8-4d23-bfb8-cd87a7980cde" containerName="kserve-container" containerID="cri-o://512e455f5f0f20eb9c2063e5317154dcfd38980dbeb6b709d023d2f4bcc3e90a" gracePeriod=30
Jun 02 15:57:49.774905 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:49.774787    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm" podUID="7fa53da0-d6c8-4d23-bfb8-cd87a7980cde" containerName="kube-rbac-proxy" containerID="cri-o://5a791f5ce7d4077608a4803221266482051d639167c894765f0417103b607f0f" gracePeriod=30
Jun 02 15:57:49.965361 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:49.965320    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx"]
Jun 02 15:57:49.965713 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:49.965700    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerName="kube-rbac-proxy"
Jun 02 15:57:49.965756 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:49.965716    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerName="kube-rbac-proxy"
Jun 02 15:57:49.965756 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:49.965740    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerName="agent"
Jun 02 15:57:49.965756 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:49.965746    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerName="agent"
Jun 02 15:57:49.965848 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:49.965756    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerName="storage-initializer"
Jun 02 15:57:49.965848 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:49.965762    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerName="storage-initializer"
Jun 02 15:57:49.965848 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:49.965769    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerName="kserve-container"
Jun 02 15:57:49.965848 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:49.965774    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerName="kserve-container"
Jun 02 15:57:49.971626 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:49.966410    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerName="agent"
Jun 02 15:57:49.971626 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:49.966441    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerName="kube-rbac-proxy"
Jun 02 15:57:49.971626 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:49.966464    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="75a18a34-f0af-43bd-bcf3-82f5c8623661" containerName="kserve-container"
Jun 02 15:57:49.971927 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:49.971904    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx"
Jun 02 15:57:49.984150 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:49.984121    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-ae9ae-kube-rbac-proxy-sar-config\""
Jun 02 15:57:49.985810 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:49.985790    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-ae9ae-predictor-serving-cert\""
Jun 02 15:57:50.013571 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:50.013538    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx"]
Jun 02 15:57:50.046180 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:50.046085    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a5dc465-78aa-4712-940a-e7c7a6f27fff-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx\" (UID: \"5a5dc465-78aa-4712-940a-e7c7a6f27fff\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx"
Jun 02 15:57:50.046180 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:50.046151    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a5dc465-78aa-4712-940a-e7c7a6f27fff-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx\" (UID: \"5a5dc465-78aa-4712-940a-e7c7a6f27fff\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx"
Jun 02 15:57:50.046375 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:50.046181    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-graph-raw-hpa-ae9ae-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a5dc465-78aa-4712-940a-e7c7a6f27fff-isvc-sklearn-graph-raw-hpa-ae9ae-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx\" (UID: \"5a5dc465-78aa-4712-940a-e7c7a6f27fff\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx"
Jun 02 15:57:50.046375 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:50.046208    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qx67\" (UniqueName: \"kubernetes.io/projected/5a5dc465-78aa-4712-940a-e7c7a6f27fff-kube-api-access-2qx67\") pod \"isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx\" (UID: \"5a5dc465-78aa-4712-940a-e7c7a6f27fff\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx"
Jun 02 15:57:50.099841 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:50.099808    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww"]
Jun 02 15:57:50.102878 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:50.102857    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww"
Jun 02 15:57:50.116701 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:50.116676    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-ae9ae-kube-rbac-proxy-sar-config\""
Jun 02 15:57:50.123816 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:50.123790    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-ae9ae-predictor-serving-cert\""
Jun 02 15:57:50.138707 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:50.138677    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww"]
Jun 02 15:57:50.147600 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:50.147567    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52548776-159b-4aa2-aa31-33accbfffe9f-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww\" (UID: \"52548776-159b-4aa2-aa31-33accbfffe9f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww"
Jun 02 15:57:50.147771 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:50.147619    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a5dc465-78aa-4712-940a-e7c7a6f27fff-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx\" (UID: \"5a5dc465-78aa-4712-940a-e7c7a6f27fff\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx"
Jun 02 15:57:50.147771 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:50.147723    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-graph-raw-hpa-ae9ae-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52548776-159b-4aa2-aa31-33accbfffe9f-isvc-xgboost-graph-raw-hpa-ae9ae-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww\" (UID: \"52548776-159b-4aa2-aa31-33accbfffe9f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww"
Jun 02 15:57:50.147898 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:50.147767    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52548776-159b-4aa2-aa31-33accbfffe9f-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww\" (UID: \"52548776-159b-4aa2-aa31-33accbfffe9f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww"
Jun 02 15:57:50.147898 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:50.147836    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a5dc465-78aa-4712-940a-e7c7a6f27fff-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx\" (UID: \"5a5dc465-78aa-4712-940a-e7c7a6f27fff\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx"
Jun 02 15:57:50.147898 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:50.147873    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-graph-raw-hpa-ae9ae-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a5dc465-78aa-4712-940a-e7c7a6f27fff-isvc-sklearn-graph-raw-hpa-ae9ae-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx\" (UID: \"5a5dc465-78aa-4712-940a-e7c7a6f27fff\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx"
Jun 02 15:57:50.148082 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:50.147902    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldrk2\" (UniqueName: \"kubernetes.io/projected/52548776-159b-4aa2-aa31-33accbfffe9f-kube-api-access-ldrk2\") pod \"isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww\" (UID: \"52548776-159b-4aa2-aa31-33accbfffe9f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww"
Jun 02 15:57:50.148082 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:50.147932    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2qx67\" (UniqueName: \"kubernetes.io/projected/5a5dc465-78aa-4712-940a-e7c7a6f27fff-kube-api-access-2qx67\") pod \"isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx\" (UID: \"5a5dc465-78aa-4712-940a-e7c7a6f27fff\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx"
Jun 02 15:57:50.148082 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:50.148026    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a5dc465-78aa-4712-940a-e7c7a6f27fff-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx\" (UID: \"5a5dc465-78aa-4712-940a-e7c7a6f27fff\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx"
Jun 02 15:57:50.148468 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:50.148445    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-graph-raw-hpa-ae9ae-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a5dc465-78aa-4712-940a-e7c7a6f27fff-isvc-sklearn-graph-raw-hpa-ae9ae-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx\" (UID: \"5a5dc465-78aa-4712-940a-e7c7a6f27fff\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx"
Jun 02 15:57:50.150402 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:50.150379    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a5dc465-78aa-4712-940a-e7c7a6f27fff-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx\" (UID: \"5a5dc465-78aa-4712-940a-e7c7a6f27fff\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx"
Jun 02 15:57:50.171193 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:50.171163    2578 generic.go:358] "Generic (PLEG): container finished" podID="7fa53da0-d6c8-4d23-bfb8-cd87a7980cde" containerID="5a791f5ce7d4077608a4803221266482051d639167c894765f0417103b607f0f" exitCode=2
Jun 02 15:57:50.171346 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:50.171244    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm" event={"ID":"7fa53da0-d6c8-4d23-bfb8-cd87a7980cde","Type":"ContainerDied","Data":"5a791f5ce7d4077608a4803221266482051d639167c894765f0417103b607f0f"}
Jun 02 15:57:50.176011 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:50.175964    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qx67\" (UniqueName: \"kubernetes.io/projected/5a5dc465-78aa-4712-940a-e7c7a6f27fff-kube-api-access-2qx67\") pod \"isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx\" (UID: \"5a5dc465-78aa-4712-940a-e7c7a6f27fff\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx"
Jun 02 15:57:50.249320 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:50.249279    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52548776-159b-4aa2-aa31-33accbfffe9f-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww\" (UID: \"52548776-159b-4aa2-aa31-33accbfffe9f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww"
Jun 02 15:57:50.249484 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:57:50.249444    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-serving-cert: secret "isvc-xgboost-graph-raw-hpa-ae9ae-predictor-serving-cert" not found
Jun 02 15:57:50.249536 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:50.249444    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-graph-raw-hpa-ae9ae-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52548776-159b-4aa2-aa31-33accbfffe9f-isvc-xgboost-graph-raw-hpa-ae9ae-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww\" (UID: \"52548776-159b-4aa2-aa31-33accbfffe9f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww"
Jun 02 15:57:50.249597 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:57:50.249539    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52548776-159b-4aa2-aa31-33accbfffe9f-proxy-tls podName:52548776-159b-4aa2-aa31-33accbfffe9f nodeName:}" failed. No retries permitted until 2026-06-02 15:57:50.74951662 +0000 UTC m=+719.451980121 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/52548776-159b-4aa2-aa31-33accbfffe9f-proxy-tls") pod "isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww" (UID: "52548776-159b-4aa2-aa31-33accbfffe9f") : secret "isvc-xgboost-graph-raw-hpa-ae9ae-predictor-serving-cert" not found
Jun 02 15:57:50.249597 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:50.249573    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52548776-159b-4aa2-aa31-33accbfffe9f-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww\" (UID: \"52548776-159b-4aa2-aa31-33accbfffe9f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww"
Jun 02 15:57:50.249699 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:50.249645    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ldrk2\" (UniqueName: \"kubernetes.io/projected/52548776-159b-4aa2-aa31-33accbfffe9f-kube-api-access-ldrk2\") pod \"isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww\" (UID: \"52548776-159b-4aa2-aa31-33accbfffe9f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww"
Jun 02 15:57:50.250059 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:50.250037    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52548776-159b-4aa2-aa31-33accbfffe9f-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww\" (UID: \"52548776-159b-4aa2-aa31-33accbfffe9f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww"
Jun 02 15:57:50.250182 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:50.250165    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-graph-raw-hpa-ae9ae-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52548776-159b-4aa2-aa31-33accbfffe9f-isvc-xgboost-graph-raw-hpa-ae9ae-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww\" (UID: \"52548776-159b-4aa2-aa31-33accbfffe9f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww"
Jun 02 15:57:50.270706 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:50.270664    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldrk2\" (UniqueName: \"kubernetes.io/projected/52548776-159b-4aa2-aa31-33accbfffe9f-kube-api-access-ldrk2\") pod \"isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww\" (UID: \"52548776-159b-4aa2-aa31-33accbfffe9f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww"
Jun 02 15:57:50.282788 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:50.282748    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx"
Jun 02 15:57:50.316587 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:50.316499    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5"]
Jun 02 15:57:50.316959 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:50.316928    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5" podUID="5df37958-2286-4d27-898c-9628699e3ff7" containerName="kserve-container" containerID="cri-o://c0e511a58b95904c1d970a72a807eaa4e8338ee1dfaf97144ef811b59bfb8f8f" gracePeriod=30
Jun 02 15:57:50.317069 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:50.316979    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5" podUID="5df37958-2286-4d27-898c-9628699e3ff7" containerName="kube-rbac-proxy" containerID="cri-o://4640e288ae3931721bfcb7a181e534388c4447d0027631e5aad87722e3de319b" gracePeriod=30
Jun 02 15:57:50.444748 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:50.444712    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx"]
Jun 02 15:57:50.448835 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:57:50.448806    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a5dc465_78aa_4712_940a_e7c7a6f27fff.slice/crio-358f32d09d8420747c16e1b4d92527505db665e122c080d11f2431892cd3d05a WatchSource:0}: Error finding container 358f32d09d8420747c16e1b4d92527505db665e122c080d11f2431892cd3d05a: Status 404 returned error can't find the container with id 358f32d09d8420747c16e1b4d92527505db665e122c080d11f2431892cd3d05a
Jun 02 15:57:50.754380 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:50.754346    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52548776-159b-4aa2-aa31-33accbfffe9f-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww\" (UID: \"52548776-159b-4aa2-aa31-33accbfffe9f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww"
Jun 02 15:57:50.756687 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:50.756665    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52548776-159b-4aa2-aa31-33accbfffe9f-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww\" (UID: \"52548776-159b-4aa2-aa31-33accbfffe9f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww"
Jun 02 15:57:51.013629 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:51.013536    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww"
Jun 02 15:57:51.169499 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:51.169466    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww"]
Jun 02 15:57:51.173990 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:57:51.173958    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52548776_159b_4aa2_aa31_33accbfffe9f.slice/crio-604cfa1225ac5e529af1173fb1f993f17d8148f97cbf1d057162d7d5bc8142b4 WatchSource:0}: Error finding container 604cfa1225ac5e529af1173fb1f993f17d8148f97cbf1d057162d7d5bc8142b4: Status 404 returned error can't find the container with id 604cfa1225ac5e529af1173fb1f993f17d8148f97cbf1d057162d7d5bc8142b4
Jun 02 15:57:51.176444 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:51.176418    2578 generic.go:358] "Generic (PLEG): container finished" podID="5df37958-2286-4d27-898c-9628699e3ff7" containerID="4640e288ae3931721bfcb7a181e534388c4447d0027631e5aad87722e3de319b" exitCode=2
Jun 02 15:57:51.176551 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:51.176497    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5" event={"ID":"5df37958-2286-4d27-898c-9628699e3ff7","Type":"ContainerDied","Data":"4640e288ae3931721bfcb7a181e534388c4447d0027631e5aad87722e3de319b"}
Jun 02 15:57:51.177906 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:51.177883    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx" event={"ID":"5a5dc465-78aa-4712-940a-e7c7a6f27fff","Type":"ContainerStarted","Data":"0971e2e45190f3ae1314b9cdd4a90f0eb0a93eb4e9ce1402d1f1ac0b63823f69"}
Jun 02 15:57:51.177988 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:51.177917    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx" event={"ID":"5a5dc465-78aa-4712-940a-e7c7a6f27fff","Type":"ContainerStarted","Data":"358f32d09d8420747c16e1b4d92527505db665e122c080d11f2431892cd3d05a"}
Jun 02 15:57:51.844115 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:51.844071    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm" podUID="7fa53da0-d6c8-4d23-bfb8-cd87a7980cde" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
Jun 02 15:57:52.182072 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:52.182039    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww" event={"ID":"52548776-159b-4aa2-aa31-33accbfffe9f","Type":"ContainerStarted","Data":"53d2cf147a8f2c6d6f59d62f10627ae3fdae30c1373deeb5be25568fa4e499c8"}
Jun 02 15:57:52.182072 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:52.182081    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww" event={"ID":"52548776-159b-4aa2-aa31-33accbfffe9f","Type":"ContainerStarted","Data":"604cfa1225ac5e529af1173fb1f993f17d8148f97cbf1d057162d7d5bc8142b4"}
Jun 02 15:57:52.900222 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:52.900183    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5" podUID="5df37958-2286-4d27-898c-9628699e3ff7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jun 02 15:57:52.904912 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:52.904876    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm" podUID="7fa53da0-d6c8-4d23-bfb8-cd87a7980cde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jun 02 15:57:54.450209 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:54.450177    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5"
Jun 02 15:57:54.489961 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:54.489919    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wccjl\" (UniqueName: \"kubernetes.io/projected/5df37958-2286-4d27-898c-9628699e3ff7-kube-api-access-wccjl\") pod \"5df37958-2286-4d27-898c-9628699e3ff7\" (UID: \"5df37958-2286-4d27-898c-9628699e3ff7\") "
Jun 02 15:57:54.490158 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:54.489982    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5df37958-2286-4d27-898c-9628699e3ff7-proxy-tls\") pod \"5df37958-2286-4d27-898c-9628699e3ff7\" (UID: \"5df37958-2286-4d27-898c-9628699e3ff7\") "
Jun 02 15:57:54.490158 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:54.490061    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-graph-raw-a7b79-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5df37958-2286-4d27-898c-9628699e3ff7-isvc-xgboost-graph-raw-a7b79-kube-rbac-proxy-sar-config\") pod \"5df37958-2286-4d27-898c-9628699e3ff7\" (UID: \"5df37958-2286-4d27-898c-9628699e3ff7\") "
Jun 02 15:57:54.490158 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:54.490150    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5df37958-2286-4d27-898c-9628699e3ff7-kserve-provision-location\") pod \"5df37958-2286-4d27-898c-9628699e3ff7\" (UID: \"5df37958-2286-4d27-898c-9628699e3ff7\") "
Jun 02 15:57:54.490442 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:54.490412    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5df37958-2286-4d27-898c-9628699e3ff7-isvc-xgboost-graph-raw-a7b79-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-graph-raw-a7b79-kube-rbac-proxy-sar-config") pod "5df37958-2286-4d27-898c-9628699e3ff7" (UID: "5df37958-2286-4d27-898c-9628699e3ff7"). InnerVolumeSpecName "isvc-xgboost-graph-raw-a7b79-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:57:54.490570 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:54.490490    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5df37958-2286-4d27-898c-9628699e3ff7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5df37958-2286-4d27-898c-9628699e3ff7" (UID: "5df37958-2286-4d27-898c-9628699e3ff7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 15:57:54.492693 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:54.492668    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5df37958-2286-4d27-898c-9628699e3ff7-kube-api-access-wccjl" (OuterVolumeSpecName: "kube-api-access-wccjl") pod "5df37958-2286-4d27-898c-9628699e3ff7" (UID: "5df37958-2286-4d27-898c-9628699e3ff7"). InnerVolumeSpecName "kube-api-access-wccjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 15:57:54.492788 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:54.492752    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5df37958-2286-4d27-898c-9628699e3ff7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5df37958-2286-4d27-898c-9628699e3ff7" (UID: "5df37958-2286-4d27-898c-9628699e3ff7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 15:57:54.518759 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:54.518734    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm"
Jun 02 15:57:54.591417 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:54.591377    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fa53da0-d6c8-4d23-bfb8-cd87a7980cde-kserve-provision-location\") pod \"7fa53da0-d6c8-4d23-bfb8-cd87a7980cde\" (UID: \"7fa53da0-d6c8-4d23-bfb8-cd87a7980cde\") "
Jun 02 15:57:54.591614 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:54.591445    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-graph-raw-a7b79-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7fa53da0-d6c8-4d23-bfb8-cd87a7980cde-isvc-sklearn-graph-raw-a7b79-kube-rbac-proxy-sar-config\") pod \"7fa53da0-d6c8-4d23-bfb8-cd87a7980cde\" (UID: \"7fa53da0-d6c8-4d23-bfb8-cd87a7980cde\") "
Jun 02 15:57:54.591614 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:54.591467    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qsk9\" (UniqueName: \"kubernetes.io/projected/7fa53da0-d6c8-4d23-bfb8-cd87a7980cde-kube-api-access-5qsk9\") pod \"7fa53da0-d6c8-4d23-bfb8-cd87a7980cde\" (UID: \"7fa53da0-d6c8-4d23-bfb8-cd87a7980cde\") "
Jun 02 15:57:54.591614 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:54.591490    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fa53da0-d6c8-4d23-bfb8-cd87a7980cde-proxy-tls\") pod \"7fa53da0-d6c8-4d23-bfb8-cd87a7980cde\" (UID: \"7fa53da0-d6c8-4d23-bfb8-cd87a7980cde\") "
Jun 02 15:57:54.591788 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:54.591740    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5df37958-2286-4d27-898c-9628699e3ff7-kserve-provision-location\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 15:57:54.591788 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:54.591760    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wccjl\" (UniqueName: \"kubernetes.io/projected/5df37958-2286-4d27-898c-9628699e3ff7-kube-api-access-wccjl\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 15:57:54.591788 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:54.591776    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5df37958-2286-4d27-898c-9628699e3ff7-proxy-tls\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 15:57:54.591905 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:54.591783    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fa53da0-d6c8-4d23-bfb8-cd87a7980cde-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7fa53da0-d6c8-4d23-bfb8-cd87a7980cde" (UID: "7fa53da0-d6c8-4d23-bfb8-cd87a7980cde"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 15:57:54.591905 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:54.591790    2578 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-graph-raw-a7b79-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5df37958-2286-4d27-898c-9628699e3ff7-isvc-xgboost-graph-raw-a7b79-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 15:57:54.591905 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:54.591819    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fa53da0-d6c8-4d23-bfb8-cd87a7980cde-isvc-sklearn-graph-raw-a7b79-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-graph-raw-a7b79-kube-rbac-proxy-sar-config") pod "7fa53da0-d6c8-4d23-bfb8-cd87a7980cde" (UID: "7fa53da0-d6c8-4d23-bfb8-cd87a7980cde"). InnerVolumeSpecName "isvc-sklearn-graph-raw-a7b79-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:57:54.593713 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:54.593689    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fa53da0-d6c8-4d23-bfb8-cd87a7980cde-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7fa53da0-d6c8-4d23-bfb8-cd87a7980cde" (UID: "7fa53da0-d6c8-4d23-bfb8-cd87a7980cde"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 15:57:54.593790 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:54.593689    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fa53da0-d6c8-4d23-bfb8-cd87a7980cde-kube-api-access-5qsk9" (OuterVolumeSpecName: "kube-api-access-5qsk9") pod "7fa53da0-d6c8-4d23-bfb8-cd87a7980cde" (UID: "7fa53da0-d6c8-4d23-bfb8-cd87a7980cde"). InnerVolumeSpecName "kube-api-access-5qsk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 15:57:54.692910 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:54.692855    2578 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-graph-raw-a7b79-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7fa53da0-d6c8-4d23-bfb8-cd87a7980cde-isvc-sklearn-graph-raw-a7b79-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 15:57:54.692910 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:54.692906    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qsk9\" (UniqueName: \"kubernetes.io/projected/7fa53da0-d6c8-4d23-bfb8-cd87a7980cde-kube-api-access-5qsk9\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 15:57:54.692910 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:54.692918    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fa53da0-d6c8-4d23-bfb8-cd87a7980cde-proxy-tls\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 15:57:54.692910 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:54.692928    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fa53da0-d6c8-4d23-bfb8-cd87a7980cde-kserve-provision-location\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 15:57:55.192494 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:55.192459    2578 generic.go:358] "Generic (PLEG): container finished" podID="52548776-159b-4aa2-aa31-33accbfffe9f" containerID="53d2cf147a8f2c6d6f59d62f10627ae3fdae30c1373deeb5be25568fa4e499c8" exitCode=0
Jun 02 15:57:55.192650 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:55.192538    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww" event={"ID":"52548776-159b-4aa2-aa31-33accbfffe9f","Type":"ContainerDied","Data":"53d2cf147a8f2c6d6f59d62f10627ae3fdae30c1373deeb5be25568fa4e499c8"}
Jun 02 15:57:55.194424 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:55.194395    2578 generic.go:358] "Generic (PLEG): container finished" podID="5df37958-2286-4d27-898c-9628699e3ff7" containerID="c0e511a58b95904c1d970a72a807eaa4e8338ee1dfaf97144ef811b59bfb8f8f" exitCode=0
Jun 02 15:57:55.194525 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:55.194466    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5"
Jun 02 15:57:55.194525 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:55.194473    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5" event={"ID":"5df37958-2286-4d27-898c-9628699e3ff7","Type":"ContainerDied","Data":"c0e511a58b95904c1d970a72a807eaa4e8338ee1dfaf97144ef811b59bfb8f8f"}
Jun 02 15:57:55.194525 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:55.194514    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5" event={"ID":"5df37958-2286-4d27-898c-9628699e3ff7","Type":"ContainerDied","Data":"7e26de8e71eea95d3029f7fde906b81be7cec18f24ca14aaf896b6d595f76d55"}
Jun 02 15:57:55.194680 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:55.194537    2578 scope.go:117] "RemoveContainer" containerID="4640e288ae3931721bfcb7a181e534388c4447d0027631e5aad87722e3de319b"
Jun 02 15:57:55.196336 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:55.196307    2578 generic.go:358] "Generic (PLEG): container finished" podID="7fa53da0-d6c8-4d23-bfb8-cd87a7980cde" containerID="512e455f5f0f20eb9c2063e5317154dcfd38980dbeb6b709d023d2f4bcc3e90a" exitCode=0
Jun 02 15:57:55.196468 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:55.196387    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm"
Jun 02 15:57:55.196468 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:55.196394    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm" event={"ID":"7fa53da0-d6c8-4d23-bfb8-cd87a7980cde","Type":"ContainerDied","Data":"512e455f5f0f20eb9c2063e5317154dcfd38980dbeb6b709d023d2f4bcc3e90a"}
Jun 02 15:57:55.196468 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:55.196431    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm" event={"ID":"7fa53da0-d6c8-4d23-bfb8-cd87a7980cde","Type":"ContainerDied","Data":"e14e76c06cdea8e46a0c0474c69f358bc057e1838333348945355a70246427a5"}
Jun 02 15:57:55.198055 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:55.198032    2578 generic.go:358] "Generic (PLEG): container finished" podID="5a5dc465-78aa-4712-940a-e7c7a6f27fff" containerID="0971e2e45190f3ae1314b9cdd4a90f0eb0a93eb4e9ce1402d1f1ac0b63823f69" exitCode=0
Jun 02 15:57:55.198120 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:55.198068    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx" event={"ID":"5a5dc465-78aa-4712-940a-e7c7a6f27fff","Type":"ContainerDied","Data":"0971e2e45190f3ae1314b9cdd4a90f0eb0a93eb4e9ce1402d1f1ac0b63823f69"}
Jun 02 15:57:55.207062 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:55.207042    2578 scope.go:117] "RemoveContainer" containerID="c0e511a58b95904c1d970a72a807eaa4e8338ee1dfaf97144ef811b59bfb8f8f"
Jun 02 15:57:55.226062 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:55.226035    2578 scope.go:117] "RemoveContainer" containerID="96574d12277239467cf0e853446c097977554afd46dc51c42351034ee81e9a96"
Jun 02 15:57:55.247906 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:55.247743    2578 scope.go:117] "RemoveContainer" containerID="4640e288ae3931721bfcb7a181e534388c4447d0027631e5aad87722e3de319b"
Jun 02 15:57:55.248216 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:57:55.248182    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4640e288ae3931721bfcb7a181e534388c4447d0027631e5aad87722e3de319b\": container with ID starting with 4640e288ae3931721bfcb7a181e534388c4447d0027631e5aad87722e3de319b not found: ID does not exist" containerID="4640e288ae3931721bfcb7a181e534388c4447d0027631e5aad87722e3de319b"
Jun 02 15:57:55.248326 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:55.248216    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4640e288ae3931721bfcb7a181e534388c4447d0027631e5aad87722e3de319b"} err="failed to get container status \"4640e288ae3931721bfcb7a181e534388c4447d0027631e5aad87722e3de319b\": rpc error: code = NotFound desc = could not find container \"4640e288ae3931721bfcb7a181e534388c4447d0027631e5aad87722e3de319b\": container with ID starting with 4640e288ae3931721bfcb7a181e534388c4447d0027631e5aad87722e3de319b not found: ID does not exist"
Jun 02 15:57:55.248326 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:55.248236    2578 scope.go:117] "RemoveContainer" containerID="c0e511a58b95904c1d970a72a807eaa4e8338ee1dfaf97144ef811b59bfb8f8f"
Jun 02 15:57:55.248493 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:57:55.248476    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0e511a58b95904c1d970a72a807eaa4e8338ee1dfaf97144ef811b59bfb8f8f\": container with ID starting with c0e511a58b95904c1d970a72a807eaa4e8338ee1dfaf97144ef811b59bfb8f8f not found: ID does not exist" containerID="c0e511a58b95904c1d970a72a807eaa4e8338ee1dfaf97144ef811b59bfb8f8f"
Jun 02 15:57:55.248550 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:55.248495    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0e511a58b95904c1d970a72a807eaa4e8338ee1dfaf97144ef811b59bfb8f8f"} err="failed to get container status \"c0e511a58b95904c1d970a72a807eaa4e8338ee1dfaf97144ef811b59bfb8f8f\": rpc error: code = NotFound desc = could not find container \"c0e511a58b95904c1d970a72a807eaa4e8338ee1dfaf97144ef811b59bfb8f8f\": container with ID starting with c0e511a58b95904c1d970a72a807eaa4e8338ee1dfaf97144ef811b59bfb8f8f not found: ID does not exist"
Jun 02 15:57:55.248550 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:55.248508    2578 scope.go:117] "RemoveContainer" containerID="96574d12277239467cf0e853446c097977554afd46dc51c42351034ee81e9a96"
Jun 02 15:57:55.248761 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:57:55.248739    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96574d12277239467cf0e853446c097977554afd46dc51c42351034ee81e9a96\": container with ID starting with 96574d12277239467cf0e853446c097977554afd46dc51c42351034ee81e9a96 not found: ID does not exist" containerID="96574d12277239467cf0e853446c097977554afd46dc51c42351034ee81e9a96"
Jun 02 15:57:55.248862 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:55.248764    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96574d12277239467cf0e853446c097977554afd46dc51c42351034ee81e9a96"} err="failed to get container status \"96574d12277239467cf0e853446c097977554afd46dc51c42351034ee81e9a96\": rpc error: code = NotFound desc = could not find container \"96574d12277239467cf0e853446c097977554afd46dc51c42351034ee81e9a96\": container with ID starting with 96574d12277239467cf0e853446c097977554afd46dc51c42351034ee81e9a96 not found: ID does not exist"
Jun 02 15:57:55.248862 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:55.248780    2578 scope.go:117] "RemoveContainer" containerID="5a791f5ce7d4077608a4803221266482051d639167c894765f0417103b607f0f"
Jun 02 15:57:55.257649 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:55.257631    2578 scope.go:117] "RemoveContainer" containerID="512e455f5f0f20eb9c2063e5317154dcfd38980dbeb6b709d023d2f4bcc3e90a"
Jun 02 15:57:55.263384 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:55.263357    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5"]
Jun 02 15:57:55.267231 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:55.267107    2578 scope.go:117] "RemoveContainer" containerID="203b68dccca5b5b6bfd7d3ea58ce8677ce36680bfc711e824d2627993a6e2d99"
Jun 02 15:57:55.274099 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:55.274068    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-a7b79-predictor-79b5ccd85-mmkz5"]
Jun 02 15:57:55.277061 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:55.277036    2578 scope.go:117] "RemoveContainer" containerID="5a791f5ce7d4077608a4803221266482051d639167c894765f0417103b607f0f"
Jun 02 15:57:55.277343 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:57:55.277321    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a791f5ce7d4077608a4803221266482051d639167c894765f0417103b607f0f\": container with ID starting with 5a791f5ce7d4077608a4803221266482051d639167c894765f0417103b607f0f not found: ID does not exist" containerID="5a791f5ce7d4077608a4803221266482051d639167c894765f0417103b607f0f"
Jun 02 15:57:55.277424 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:55.277353    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a791f5ce7d4077608a4803221266482051d639167c894765f0417103b607f0f"} err="failed to get container status \"5a791f5ce7d4077608a4803221266482051d639167c894765f0417103b607f0f\": rpc error: code = NotFound desc = could not find container \"5a791f5ce7d4077608a4803221266482051d639167c894765f0417103b607f0f\": container with ID starting with 5a791f5ce7d4077608a4803221266482051d639167c894765f0417103b607f0f not found: ID does not exist"
Jun 02 15:57:55.277424 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:55.277379    2578 scope.go:117] "RemoveContainer" containerID="512e455f5f0f20eb9c2063e5317154dcfd38980dbeb6b709d023d2f4bcc3e90a"
Jun 02 15:57:55.277603 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:57:55.277587    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"512e455f5f0f20eb9c2063e5317154dcfd38980dbeb6b709d023d2f4bcc3e90a\": container with ID starting with 512e455f5f0f20eb9c2063e5317154dcfd38980dbeb6b709d023d2f4bcc3e90a not found: ID does not exist" containerID="512e455f5f0f20eb9c2063e5317154dcfd38980dbeb6b709d023d2f4bcc3e90a"
Jun 02 15:57:55.277674 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:55.277611    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"512e455f5f0f20eb9c2063e5317154dcfd38980dbeb6b709d023d2f4bcc3e90a"} err="failed to get container status \"512e455f5f0f20eb9c2063e5317154dcfd38980dbeb6b709d023d2f4bcc3e90a\": rpc error: code = NotFound desc = could not find container \"512e455f5f0f20eb9c2063e5317154dcfd38980dbeb6b709d023d2f4bcc3e90a\": container with ID starting with 512e455f5f0f20eb9c2063e5317154dcfd38980dbeb6b709d023d2f4bcc3e90a not found: ID does not exist"
Jun 02 15:57:55.277674 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:55.277630    2578 scope.go:117] "RemoveContainer" containerID="203b68dccca5b5b6bfd7d3ea58ce8677ce36680bfc711e824d2627993a6e2d99"
Jun 02 15:57:55.277934 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:57:55.277846    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"203b68dccca5b5b6bfd7d3ea58ce8677ce36680bfc711e824d2627993a6e2d99\": container with ID starting with 203b68dccca5b5b6bfd7d3ea58ce8677ce36680bfc711e824d2627993a6e2d99 not found: ID does not exist" containerID="203b68dccca5b5b6bfd7d3ea58ce8677ce36680bfc711e824d2627993a6e2d99"
Jun 02 15:57:55.277934 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:55.277866    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"203b68dccca5b5b6bfd7d3ea58ce8677ce36680bfc711e824d2627993a6e2d99"} err="failed to get container status \"203b68dccca5b5b6bfd7d3ea58ce8677ce36680bfc711e824d2627993a6e2d99\": rpc error: code = NotFound desc = could not find container \"203b68dccca5b5b6bfd7d3ea58ce8677ce36680bfc711e824d2627993a6e2d99\": container with ID starting with 203b68dccca5b5b6bfd7d3ea58ce8677ce36680bfc711e824d2627993a6e2d99 not found: ID does not exist"
Jun 02 15:57:55.289060 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:55.289033    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm"]
Jun 02 15:57:55.309662 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:55.309632    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-a7b79-predictor-7666ccb64d-qbtzm"]
Jun 02 15:57:55.908367 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:55.908332    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5df37958-2286-4d27-898c-9628699e3ff7" path="/var/lib/kubelet/pods/5df37958-2286-4d27-898c-9628699e3ff7/volumes"
Jun 02 15:57:55.908817 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:55.908804    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fa53da0-d6c8-4d23-bfb8-cd87a7980cde" path="/var/lib/kubelet/pods/7fa53da0-d6c8-4d23-bfb8-cd87a7980cde/volumes"
Jun 02 15:57:56.207464 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:56.207367    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx" event={"ID":"5a5dc465-78aa-4712-940a-e7c7a6f27fff","Type":"ContainerStarted","Data":"242ee0b6ffeacb0f2e771294c66b58b670f5ef27f10d8babaafaaf0bb0228e92"}
Jun 02 15:57:56.207464 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:56.207406    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx" event={"ID":"5a5dc465-78aa-4712-940a-e7c7a6f27fff","Type":"ContainerStarted","Data":"00850e18a07cb320b062fbea019476f6fff2d2c3096799fce294ce6f4fb61233"}
Jun 02 15:57:56.207783 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:56.207711    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx"
Jun 02 15:57:56.207860 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:56.207844    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx"
Jun 02 15:57:56.209219 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:56.209192    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx" podUID="5a5dc465-78aa-4712-940a-e7c7a6f27fff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jun 02 15:57:56.209360 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:56.209340    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww" event={"ID":"52548776-159b-4aa2-aa31-33accbfffe9f","Type":"ContainerStarted","Data":"69a760093ff72f3d05961028cc37b5aab1d11de7d502b52f2a9aed8070c971d8"}
Jun 02 15:57:56.209404 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:56.209370    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww" event={"ID":"52548776-159b-4aa2-aa31-33accbfffe9f","Type":"ContainerStarted","Data":"fbc7c364ba396d1b69164a6adaed4628887e2def9d3aea3d19915e8b0e527733"}
Jun 02 15:57:56.209582 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:56.209567    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww"
Jun 02 15:57:56.239860 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:56.239808    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx" podStartSLOduration=7.239793503 podStartE2EDuration="7.239793503s" podCreationTimestamp="2026-06-02 15:57:49 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 15:57:56.239617261 +0000 UTC m=+724.942080779" watchObservedRunningTime="2026-06-02 15:57:56.239793503 +0000 UTC m=+724.942257022"
Jun 02 15:57:57.212960 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:57.212916    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx" podUID="5a5dc465-78aa-4712-940a-e7c7a6f27fff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jun 02 15:57:57.212960 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:57.212952    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww"
Jun 02 15:57:57.214089 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:57.214065    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww" podUID="52548776-159b-4aa2-aa31-33accbfffe9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jun 02 15:57:58.215433 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:57:58.215393    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww" podUID="52548776-159b-4aa2-aa31-33accbfffe9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jun 02 15:58:02.217468 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:58:02.217438    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx"
Jun 02 15:58:02.217986 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:58:02.217956    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx" podUID="5a5dc465-78aa-4712-940a-e7c7a6f27fff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jun 02 15:58:02.247923 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:58:02.247854    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww" podStartSLOduration=12.247838686 podStartE2EDuration="12.247838686s" podCreationTimestamp="2026-06-02 15:57:50 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 15:57:56.274814472 +0000 UTC m=+724.977277988" watchObservedRunningTime="2026-06-02 15:58:02.247838686 +0000 UTC m=+730.950302204"
Jun 02 15:58:03.219634 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:58:03.219601    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww"
Jun 02 15:58:03.220122 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:58:03.220096    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww" podUID="52548776-159b-4aa2-aa31-33accbfffe9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jun 02 15:58:12.218835 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:58:12.218795    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx" podUID="5a5dc465-78aa-4712-940a-e7c7a6f27fff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jun 02 15:58:13.220694 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:58:13.220654    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww" podUID="52548776-159b-4aa2-aa31-33accbfffe9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jun 02 15:58:22.217931 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:58:22.217892    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx" podUID="5a5dc465-78aa-4712-940a-e7c7a6f27fff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jun 02 15:58:23.220746 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:58:23.220702    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww" podUID="52548776-159b-4aa2-aa31-33accbfffe9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jun 02 15:58:32.218774 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:58:32.218726    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx" podUID="5a5dc465-78aa-4712-940a-e7c7a6f27fff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jun 02 15:58:33.220382 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:58:33.220342    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww" podUID="52548776-159b-4aa2-aa31-33accbfffe9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jun 02 15:58:42.218532 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:58:42.218494    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx" podUID="5a5dc465-78aa-4712-940a-e7c7a6f27fff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jun 02 15:58:43.220397 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:58:43.220346    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww" podUID="52548776-159b-4aa2-aa31-33accbfffe9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jun 02 15:58:52.217838 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:58:52.217800    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx" podUID="5a5dc465-78aa-4712-940a-e7c7a6f27fff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jun 02 15:58:53.220798 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:58:53.220753    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww" podUID="52548776-159b-4aa2-aa31-33accbfffe9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jun 02 15:59:02.218954 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:02.218928    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx"
Jun 02 15:59:03.221155 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:03.221126    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww"
Jun 02 15:59:29.969403 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:29.969363    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx"]
Jun 02 15:59:29.969823 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:29.969772    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx" podUID="5a5dc465-78aa-4712-940a-e7c7a6f27fff" containerName="kserve-container" containerID="cri-o://00850e18a07cb320b062fbea019476f6fff2d2c3096799fce294ce6f4fb61233" gracePeriod=30
Jun 02 15:59:29.969876 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:29.969809    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx" podUID="5a5dc465-78aa-4712-940a-e7c7a6f27fff" containerName="kube-rbac-proxy" containerID="cri-o://242ee0b6ffeacb0f2e771294c66b58b670f5ef27f10d8babaafaaf0bb0228e92" gracePeriod=30
Jun 02 15:59:30.014025 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:30.013986    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-0d859-predictor-58bd984d44-v47p2"]
Jun 02 15:59:30.014346 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:30.014334    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fa53da0-d6c8-4d23-bfb8-cd87a7980cde" containerName="kube-rbac-proxy"
Jun 02 15:59:30.014392 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:30.014348    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fa53da0-d6c8-4d23-bfb8-cd87a7980cde" containerName="kube-rbac-proxy"
Jun 02 15:59:30.014392 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:30.014362    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5df37958-2286-4d27-898c-9628699e3ff7" containerName="kserve-container"
Jun 02 15:59:30.014392 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:30.014368    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5df37958-2286-4d27-898c-9628699e3ff7" containerName="kserve-container"
Jun 02 15:59:30.014392 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:30.014377    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5df37958-2286-4d27-898c-9628699e3ff7" containerName="storage-initializer"
Jun 02 15:59:30.014392 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:30.014383    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5df37958-2286-4d27-898c-9628699e3ff7" containerName="storage-initializer"
Jun 02 15:59:30.014392 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:30.014392    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fa53da0-d6c8-4d23-bfb8-cd87a7980cde" containerName="storage-initializer"
Jun 02 15:59:30.014561 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:30.014397    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fa53da0-d6c8-4d23-bfb8-cd87a7980cde" containerName="storage-initializer"
Jun 02 15:59:30.014561 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:30.014408    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5df37958-2286-4d27-898c-9628699e3ff7" containerName="kube-rbac-proxy"
Jun 02 15:59:30.014561 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:30.014413    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5df37958-2286-4d27-898c-9628699e3ff7" containerName="kube-rbac-proxy"
Jun 02 15:59:30.014561 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:30.014425    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fa53da0-d6c8-4d23-bfb8-cd87a7980cde" containerName="kserve-container"
Jun 02 15:59:30.014561 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:30.014430    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fa53da0-d6c8-4d23-bfb8-cd87a7980cde" containerName="kserve-container"
Jun 02 15:59:30.014561 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:30.014489    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="7fa53da0-d6c8-4d23-bfb8-cd87a7980cde" containerName="kserve-container"
Jun 02 15:59:30.014561 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:30.014497    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="7fa53da0-d6c8-4d23-bfb8-cd87a7980cde" containerName="kube-rbac-proxy"
Jun 02 15:59:30.014561 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:30.014507    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="5df37958-2286-4d27-898c-9628699e3ff7" containerName="kube-rbac-proxy"
Jun 02 15:59:30.014561 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:30.014514    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="5df37958-2286-4d27-898c-9628699e3ff7" containerName="kserve-container"
Jun 02 15:59:30.016470 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:30.016456    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-0d859-predictor-58bd984d44-v47p2"
Jun 02 15:59:30.018308 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:30.018287    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-0d859-kube-rbac-proxy-sar-config\""
Jun 02 15:59:30.018422 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:30.018353    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-0d859-predictor-serving-cert\""
Jun 02 15:59:30.028607 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:30.028586    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-0d859-predictor-58bd984d44-v47p2"]
Jun 02 15:59:30.111444 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:30.111415    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"message-dumper-raw-0d859-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14a65876-5613-49f5-89ad-6988b8d0aee7-message-dumper-raw-0d859-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-0d859-predictor-58bd984d44-v47p2\" (UID: \"14a65876-5613-49f5-89ad-6988b8d0aee7\") " pod="kserve-ci-e2e-test/message-dumper-raw-0d859-predictor-58bd984d44-v47p2"
Jun 02 15:59:30.111610 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:30.111547    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5td4z\" (UniqueName: \"kubernetes.io/projected/14a65876-5613-49f5-89ad-6988b8d0aee7-kube-api-access-5td4z\") pod \"message-dumper-raw-0d859-predictor-58bd984d44-v47p2\" (UID: \"14a65876-5613-49f5-89ad-6988b8d0aee7\") " pod="kserve-ci-e2e-test/message-dumper-raw-0d859-predictor-58bd984d44-v47p2"
Jun 02 15:59:30.111610 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:30.111600    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14a65876-5613-49f5-89ad-6988b8d0aee7-proxy-tls\") pod \"message-dumper-raw-0d859-predictor-58bd984d44-v47p2\" (UID: \"14a65876-5613-49f5-89ad-6988b8d0aee7\") " pod="kserve-ci-e2e-test/message-dumper-raw-0d859-predictor-58bd984d44-v47p2"
Jun 02 15:59:30.114611 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:30.114588    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww"]
Jun 02 15:59:30.114933 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:30.114889    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww" podUID="52548776-159b-4aa2-aa31-33accbfffe9f" containerName="kserve-container" containerID="cri-o://fbc7c364ba396d1b69164a6adaed4628887e2def9d3aea3d19915e8b0e527733" gracePeriod=30
Jun 02 15:59:30.115010 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:30.114924    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww" podUID="52548776-159b-4aa2-aa31-33accbfffe9f" containerName="kube-rbac-proxy" containerID="cri-o://69a760093ff72f3d05961028cc37b5aab1d11de7d502b52f2a9aed8070c971d8" gracePeriod=30
Jun 02 15:59:30.212019 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:30.211961    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5td4z\" (UniqueName: \"kubernetes.io/projected/14a65876-5613-49f5-89ad-6988b8d0aee7-kube-api-access-5td4z\") pod \"message-dumper-raw-0d859-predictor-58bd984d44-v47p2\" (UID: \"14a65876-5613-49f5-89ad-6988b8d0aee7\") " pod="kserve-ci-e2e-test/message-dumper-raw-0d859-predictor-58bd984d44-v47p2"
Jun 02 15:59:30.212195 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:30.212105    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14a65876-5613-49f5-89ad-6988b8d0aee7-proxy-tls\") pod \"message-dumper-raw-0d859-predictor-58bd984d44-v47p2\" (UID: \"14a65876-5613-49f5-89ad-6988b8d0aee7\") " pod="kserve-ci-e2e-test/message-dumper-raw-0d859-predictor-58bd984d44-v47p2"
Jun 02 15:59:30.212195 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:30.212146    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"message-dumper-raw-0d859-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14a65876-5613-49f5-89ad-6988b8d0aee7-message-dumper-raw-0d859-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-0d859-predictor-58bd984d44-v47p2\" (UID: \"14a65876-5613-49f5-89ad-6988b8d0aee7\") " pod="kserve-ci-e2e-test/message-dumper-raw-0d859-predictor-58bd984d44-v47p2"
Jun 02 15:59:30.212787 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:30.212766    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"message-dumper-raw-0d859-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14a65876-5613-49f5-89ad-6988b8d0aee7-message-dumper-raw-0d859-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-0d859-predictor-58bd984d44-v47p2\" (UID: \"14a65876-5613-49f5-89ad-6988b8d0aee7\") " pod="kserve-ci-e2e-test/message-dumper-raw-0d859-predictor-58bd984d44-v47p2"
Jun 02 15:59:30.214382 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:30.214363    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14a65876-5613-49f5-89ad-6988b8d0aee7-proxy-tls\") pod \"message-dumper-raw-0d859-predictor-58bd984d44-v47p2\" (UID: \"14a65876-5613-49f5-89ad-6988b8d0aee7\") " pod="kserve-ci-e2e-test/message-dumper-raw-0d859-predictor-58bd984d44-v47p2"
Jun 02 15:59:30.220932 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:30.220875    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5td4z\" (UniqueName: \"kubernetes.io/projected/14a65876-5613-49f5-89ad-6988b8d0aee7-kube-api-access-5td4z\") pod \"message-dumper-raw-0d859-predictor-58bd984d44-v47p2\" (UID: \"14a65876-5613-49f5-89ad-6988b8d0aee7\") " pod="kserve-ci-e2e-test/message-dumper-raw-0d859-predictor-58bd984d44-v47p2"
Jun 02 15:59:30.326949 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:30.326920    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-0d859-predictor-58bd984d44-v47p2"
Jun 02 15:59:30.461513 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:30.461482    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-0d859-predictor-58bd984d44-v47p2"]
Jun 02 15:59:30.462485 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:59:30.462462    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14a65876_5613_49f5_89ad_6988b8d0aee7.slice/crio-f9280f6784146049e81a6b491c67a375599e088a447db638f290eaabe450f335 WatchSource:0}: Error finding container f9280f6784146049e81a6b491c67a375599e088a447db638f290eaabe450f335: Status 404 returned error can't find the container with id f9280f6784146049e81a6b491c67a375599e088a447db638f290eaabe450f335
Jun 02 15:59:30.464267 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:30.464253    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 15:59:30.486558 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:30.486503    2578 generic.go:358] "Generic (PLEG): container finished" podID="5a5dc465-78aa-4712-940a-e7c7a6f27fff" containerID="242ee0b6ffeacb0f2e771294c66b58b670f5ef27f10d8babaafaaf0bb0228e92" exitCode=2
Jun 02 15:59:30.486644 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:30.486578    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx" event={"ID":"5a5dc465-78aa-4712-940a-e7c7a6f27fff","Type":"ContainerDied","Data":"242ee0b6ffeacb0f2e771294c66b58b670f5ef27f10d8babaafaaf0bb0228e92"}
Jun 02 15:59:30.488262 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:30.488239    2578 generic.go:358] "Generic (PLEG): container finished" podID="52548776-159b-4aa2-aa31-33accbfffe9f" containerID="69a760093ff72f3d05961028cc37b5aab1d11de7d502b52f2a9aed8070c971d8" exitCode=2
Jun 02 15:59:30.488332 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:30.488263    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww" event={"ID":"52548776-159b-4aa2-aa31-33accbfffe9f","Type":"ContainerDied","Data":"69a760093ff72f3d05961028cc37b5aab1d11de7d502b52f2a9aed8070c971d8"}
Jun 02 15:59:30.489184 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:30.489165    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-0d859-predictor-58bd984d44-v47p2" event={"ID":"14a65876-5613-49f5-89ad-6988b8d0aee7","Type":"ContainerStarted","Data":"f9280f6784146049e81a6b491c67a375599e088a447db638f290eaabe450f335"}
Jun 02 15:59:32.213887 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:32.213844    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx" podUID="5a5dc465-78aa-4712-940a-e7c7a6f27fff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jun 02 15:59:32.218257 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:32.218234    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx" podUID="5a5dc465-78aa-4712-940a-e7c7a6f27fff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jun 02 15:59:32.497307 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:32.497222    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-0d859-predictor-58bd984d44-v47p2" event={"ID":"14a65876-5613-49f5-89ad-6988b8d0aee7","Type":"ContainerStarted","Data":"64d15930efcc7ca70524a3f33654540d28ef6e6b583181331e0f8fd0f4e13fa7"}
Jun 02 15:59:32.497307 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:32.497258    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-0d859-predictor-58bd984d44-v47p2" event={"ID":"14a65876-5613-49f5-89ad-6988b8d0aee7","Type":"ContainerStarted","Data":"5e80cf55127bcc46bcdf2ddbf3f594e08dd16a740356b1827dd9e0a706c4b668"}
Jun 02 15:59:32.497480 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:32.497319    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-0d859-predictor-58bd984d44-v47p2"
Jun 02 15:59:32.521276 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:32.521219    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-0d859-predictor-58bd984d44-v47p2" podStartSLOduration=2.458154599 podStartE2EDuration="3.521203709s" podCreationTimestamp="2026-06-02 15:59:29 +0000 UTC" firstStartedPulling="2026-06-02 15:59:30.464373985 +0000 UTC m=+819.166837482" lastFinishedPulling="2026-06-02 15:59:31.527423083 +0000 UTC m=+820.229886592" observedRunningTime="2026-06-02 15:59:32.515182007 +0000 UTC m=+821.217645539" watchObservedRunningTime="2026-06-02 15:59:32.521203709 +0000 UTC m=+821.223667225"
Jun 02 15:59:33.215704 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:33.215669    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww" podUID="52548776-159b-4aa2-aa31-33accbfffe9f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Jun 02 15:59:33.219987 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:33.219965    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww" podUID="52548776-159b-4aa2-aa31-33accbfffe9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jun 02 15:59:33.500873 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:33.500795    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-0d859-predictor-58bd984d44-v47p2"
Jun 02 15:59:33.502408 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:33.502387    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-0d859-predictor-58bd984d44-v47p2"
Jun 02 15:59:33.854506 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:33.854486    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww"
Jun 02 15:59:33.945284 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:33.945258    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52548776-159b-4aa2-aa31-33accbfffe9f-proxy-tls\") pod \"52548776-159b-4aa2-aa31-33accbfffe9f\" (UID: \"52548776-159b-4aa2-aa31-33accbfffe9f\") "
Jun 02 15:59:33.945434 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:33.945292    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldrk2\" (UniqueName: \"kubernetes.io/projected/52548776-159b-4aa2-aa31-33accbfffe9f-kube-api-access-ldrk2\") pod \"52548776-159b-4aa2-aa31-33accbfffe9f\" (UID: \"52548776-159b-4aa2-aa31-33accbfffe9f\") "
Jun 02 15:59:33.945434 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:33.945337    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-graph-raw-hpa-ae9ae-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52548776-159b-4aa2-aa31-33accbfffe9f-isvc-xgboost-graph-raw-hpa-ae9ae-kube-rbac-proxy-sar-config\") pod \"52548776-159b-4aa2-aa31-33accbfffe9f\" (UID: \"52548776-159b-4aa2-aa31-33accbfffe9f\") "
Jun 02 15:59:33.945434 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:33.945383    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52548776-159b-4aa2-aa31-33accbfffe9f-kserve-provision-location\") pod \"52548776-159b-4aa2-aa31-33accbfffe9f\" (UID: \"52548776-159b-4aa2-aa31-33accbfffe9f\") "
Jun 02 15:59:33.945687 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:33.945657    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52548776-159b-4aa2-aa31-33accbfffe9f-isvc-xgboost-graph-raw-hpa-ae9ae-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-graph-raw-hpa-ae9ae-kube-rbac-proxy-sar-config") pod "52548776-159b-4aa2-aa31-33accbfffe9f" (UID: "52548776-159b-4aa2-aa31-33accbfffe9f"). InnerVolumeSpecName "isvc-xgboost-graph-raw-hpa-ae9ae-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:59:33.945808 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:33.945703    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52548776-159b-4aa2-aa31-33accbfffe9f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "52548776-159b-4aa2-aa31-33accbfffe9f" (UID: "52548776-159b-4aa2-aa31-33accbfffe9f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 15:59:33.947318 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:33.947294    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52548776-159b-4aa2-aa31-33accbfffe9f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "52548776-159b-4aa2-aa31-33accbfffe9f" (UID: "52548776-159b-4aa2-aa31-33accbfffe9f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 15:59:33.947394 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:33.947356    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52548776-159b-4aa2-aa31-33accbfffe9f-kube-api-access-ldrk2" (OuterVolumeSpecName: "kube-api-access-ldrk2") pod "52548776-159b-4aa2-aa31-33accbfffe9f" (UID: "52548776-159b-4aa2-aa31-33accbfffe9f"). InnerVolumeSpecName "kube-api-access-ldrk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 15:59:34.045891 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.045831    2578 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-graph-raw-hpa-ae9ae-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52548776-159b-4aa2-aa31-33accbfffe9f-isvc-xgboost-graph-raw-hpa-ae9ae-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 15:59:34.045891 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.045855    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52548776-159b-4aa2-aa31-33accbfffe9f-kserve-provision-location\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 15:59:34.045891 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.045865    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52548776-159b-4aa2-aa31-33accbfffe9f-proxy-tls\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 15:59:34.045891 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.045874    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldrk2\" (UniqueName: \"kubernetes.io/projected/52548776-159b-4aa2-aa31-33accbfffe9f-kube-api-access-ldrk2\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 15:59:34.297867 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.297812    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx"
Jun 02 15:59:34.449298 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.449272    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qx67\" (UniqueName: \"kubernetes.io/projected/5a5dc465-78aa-4712-940a-e7c7a6f27fff-kube-api-access-2qx67\") pod \"5a5dc465-78aa-4712-940a-e7c7a6f27fff\" (UID: \"5a5dc465-78aa-4712-940a-e7c7a6f27fff\") "
Jun 02 15:59:34.449474 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.449356    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a5dc465-78aa-4712-940a-e7c7a6f27fff-proxy-tls\") pod \"5a5dc465-78aa-4712-940a-e7c7a6f27fff\" (UID: \"5a5dc465-78aa-4712-940a-e7c7a6f27fff\") "
Jun 02 15:59:34.449474 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.449380    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-graph-raw-hpa-ae9ae-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a5dc465-78aa-4712-940a-e7c7a6f27fff-isvc-sklearn-graph-raw-hpa-ae9ae-kube-rbac-proxy-sar-config\") pod \"5a5dc465-78aa-4712-940a-e7c7a6f27fff\" (UID: \"5a5dc465-78aa-4712-940a-e7c7a6f27fff\") "
Jun 02 15:59:34.449474 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.449425    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a5dc465-78aa-4712-940a-e7c7a6f27fff-kserve-provision-location\") pod \"5a5dc465-78aa-4712-940a-e7c7a6f27fff\" (UID: \"5a5dc465-78aa-4712-940a-e7c7a6f27fff\") "
Jun 02 15:59:34.449724 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.449699    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a5dc465-78aa-4712-940a-e7c7a6f27fff-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5a5dc465-78aa-4712-940a-e7c7a6f27fff" (UID: "5a5dc465-78aa-4712-940a-e7c7a6f27fff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 15:59:34.449853 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.449753    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a5dc465-78aa-4712-940a-e7c7a6f27fff-isvc-sklearn-graph-raw-hpa-ae9ae-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-graph-raw-hpa-ae9ae-kube-rbac-proxy-sar-config") pod "5a5dc465-78aa-4712-940a-e7c7a6f27fff" (UID: "5a5dc465-78aa-4712-940a-e7c7a6f27fff"). InnerVolumeSpecName "isvc-sklearn-graph-raw-hpa-ae9ae-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:59:34.451355 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.451329    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a5dc465-78aa-4712-940a-e7c7a6f27fff-kube-api-access-2qx67" (OuterVolumeSpecName: "kube-api-access-2qx67") pod "5a5dc465-78aa-4712-940a-e7c7a6f27fff" (UID: "5a5dc465-78aa-4712-940a-e7c7a6f27fff"). InnerVolumeSpecName "kube-api-access-2qx67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 15:59:34.451576 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.451558    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a5dc465-78aa-4712-940a-e7c7a6f27fff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5a5dc465-78aa-4712-940a-e7c7a6f27fff" (UID: "5a5dc465-78aa-4712-940a-e7c7a6f27fff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 15:59:34.505110 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.505080    2578 generic.go:358] "Generic (PLEG): container finished" podID="5a5dc465-78aa-4712-940a-e7c7a6f27fff" containerID="00850e18a07cb320b062fbea019476f6fff2d2c3096799fce294ce6f4fb61233" exitCode=0
Jun 02 15:59:34.505232 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.505157    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx"
Jun 02 15:59:34.505232 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.505167    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx" event={"ID":"5a5dc465-78aa-4712-940a-e7c7a6f27fff","Type":"ContainerDied","Data":"00850e18a07cb320b062fbea019476f6fff2d2c3096799fce294ce6f4fb61233"}
Jun 02 15:59:34.505232 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.505208    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx" event={"ID":"5a5dc465-78aa-4712-940a-e7c7a6f27fff","Type":"ContainerDied","Data":"358f32d09d8420747c16e1b4d92527505db665e122c080d11f2431892cd3d05a"}
Jun 02 15:59:34.505368 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.505231    2578 scope.go:117] "RemoveContainer" containerID="242ee0b6ffeacb0f2e771294c66b58b670f5ef27f10d8babaafaaf0bb0228e92"
Jun 02 15:59:34.506794 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.506761    2578 generic.go:358] "Generic (PLEG): container finished" podID="52548776-159b-4aa2-aa31-33accbfffe9f" containerID="fbc7c364ba396d1b69164a6adaed4628887e2def9d3aea3d19915e8b0e527733" exitCode=0
Jun 02 15:59:34.506899 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.506841    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww"
Jun 02 15:59:34.506899 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.506852    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww" event={"ID":"52548776-159b-4aa2-aa31-33accbfffe9f","Type":"ContainerDied","Data":"fbc7c364ba396d1b69164a6adaed4628887e2def9d3aea3d19915e8b0e527733"}
Jun 02 15:59:34.506899 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.506885    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww" event={"ID":"52548776-159b-4aa2-aa31-33accbfffe9f","Type":"ContainerDied","Data":"604cfa1225ac5e529af1173fb1f993f17d8148f97cbf1d057162d7d5bc8142b4"}
Jun 02 15:59:34.513283 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.513235    2578 scope.go:117] "RemoveContainer" containerID="00850e18a07cb320b062fbea019476f6fff2d2c3096799fce294ce6f4fb61233"
Jun 02 15:59:34.521379 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.521355    2578 scope.go:117] "RemoveContainer" containerID="0971e2e45190f3ae1314b9cdd4a90f0eb0a93eb4e9ce1402d1f1ac0b63823f69"
Jun 02 15:59:34.527672 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.527657    2578 scope.go:117] "RemoveContainer" containerID="242ee0b6ffeacb0f2e771294c66b58b670f5ef27f10d8babaafaaf0bb0228e92"
Jun 02 15:59:34.527927 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:59:34.527906    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"242ee0b6ffeacb0f2e771294c66b58b670f5ef27f10d8babaafaaf0bb0228e92\": container with ID starting with 242ee0b6ffeacb0f2e771294c66b58b670f5ef27f10d8babaafaaf0bb0228e92 not found: ID does not exist" containerID="242ee0b6ffeacb0f2e771294c66b58b670f5ef27f10d8babaafaaf0bb0228e92"
Jun 02 15:59:34.528026 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.527934    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"242ee0b6ffeacb0f2e771294c66b58b670f5ef27f10d8babaafaaf0bb0228e92"} err="failed to get container status \"242ee0b6ffeacb0f2e771294c66b58b670f5ef27f10d8babaafaaf0bb0228e92\": rpc error: code = NotFound desc = could not find container \"242ee0b6ffeacb0f2e771294c66b58b670f5ef27f10d8babaafaaf0bb0228e92\": container with ID starting with 242ee0b6ffeacb0f2e771294c66b58b670f5ef27f10d8babaafaaf0bb0228e92 not found: ID does not exist"
Jun 02 15:59:34.528026 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.527948    2578 scope.go:117] "RemoveContainer" containerID="00850e18a07cb320b062fbea019476f6fff2d2c3096799fce294ce6f4fb61233"
Jun 02 15:59:34.528234 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:59:34.528220    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00850e18a07cb320b062fbea019476f6fff2d2c3096799fce294ce6f4fb61233\": container with ID starting with 00850e18a07cb320b062fbea019476f6fff2d2c3096799fce294ce6f4fb61233 not found: ID does not exist" containerID="00850e18a07cb320b062fbea019476f6fff2d2c3096799fce294ce6f4fb61233"
Jun 02 15:59:34.528287 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.528238    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00850e18a07cb320b062fbea019476f6fff2d2c3096799fce294ce6f4fb61233"} err="failed to get container status \"00850e18a07cb320b062fbea019476f6fff2d2c3096799fce294ce6f4fb61233\": rpc error: code = NotFound desc = could not find container \"00850e18a07cb320b062fbea019476f6fff2d2c3096799fce294ce6f4fb61233\": container with ID starting with 00850e18a07cb320b062fbea019476f6fff2d2c3096799fce294ce6f4fb61233 not found: ID does not exist"
Jun 02 15:59:34.528287 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.528250    2578 scope.go:117] "RemoveContainer" containerID="0971e2e45190f3ae1314b9cdd4a90f0eb0a93eb4e9ce1402d1f1ac0b63823f69"
Jun 02 15:59:34.528458 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:59:34.528443    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0971e2e45190f3ae1314b9cdd4a90f0eb0a93eb4e9ce1402d1f1ac0b63823f69\": container with ID starting with 0971e2e45190f3ae1314b9cdd4a90f0eb0a93eb4e9ce1402d1f1ac0b63823f69 not found: ID does not exist" containerID="0971e2e45190f3ae1314b9cdd4a90f0eb0a93eb4e9ce1402d1f1ac0b63823f69"
Jun 02 15:59:34.528492 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.528466    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0971e2e45190f3ae1314b9cdd4a90f0eb0a93eb4e9ce1402d1f1ac0b63823f69"} err="failed to get container status \"0971e2e45190f3ae1314b9cdd4a90f0eb0a93eb4e9ce1402d1f1ac0b63823f69\": rpc error: code = NotFound desc = could not find container \"0971e2e45190f3ae1314b9cdd4a90f0eb0a93eb4e9ce1402d1f1ac0b63823f69\": container with ID starting with 0971e2e45190f3ae1314b9cdd4a90f0eb0a93eb4e9ce1402d1f1ac0b63823f69 not found: ID does not exist"
Jun 02 15:59:34.528492 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.528479    2578 scope.go:117] "RemoveContainer" containerID="69a760093ff72f3d05961028cc37b5aab1d11de7d502b52f2a9aed8070c971d8"
Jun 02 15:59:34.532247 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.532226    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww"]
Jun 02 15:59:34.535008 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.534976    2578 scope.go:117] "RemoveContainer" containerID="fbc7c364ba396d1b69164a6adaed4628887e2def9d3aea3d19915e8b0e527733"
Jun 02 15:59:34.538211 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.538188    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ae9ae-predictor-5bf88f9fc7-fk8ww"]
Jun 02 15:59:34.541614 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.541599    2578 scope.go:117] "RemoveContainer" containerID="53d2cf147a8f2c6d6f59d62f10627ae3fdae30c1373deeb5be25568fa4e499c8"
Jun 02 15:59:34.547585 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.547572    2578 scope.go:117] "RemoveContainer" containerID="69a760093ff72f3d05961028cc37b5aab1d11de7d502b52f2a9aed8070c971d8"
Jun 02 15:59:34.547806 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:59:34.547790    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69a760093ff72f3d05961028cc37b5aab1d11de7d502b52f2a9aed8070c971d8\": container with ID starting with 69a760093ff72f3d05961028cc37b5aab1d11de7d502b52f2a9aed8070c971d8 not found: ID does not exist" containerID="69a760093ff72f3d05961028cc37b5aab1d11de7d502b52f2a9aed8070c971d8"
Jun 02 15:59:34.547847 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.547811    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69a760093ff72f3d05961028cc37b5aab1d11de7d502b52f2a9aed8070c971d8"} err="failed to get container status \"69a760093ff72f3d05961028cc37b5aab1d11de7d502b52f2a9aed8070c971d8\": rpc error: code = NotFound desc = could not find container \"69a760093ff72f3d05961028cc37b5aab1d11de7d502b52f2a9aed8070c971d8\": container with ID starting with 69a760093ff72f3d05961028cc37b5aab1d11de7d502b52f2a9aed8070c971d8 not found: ID does not exist"
Jun 02 15:59:34.547847 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.547825    2578 scope.go:117] "RemoveContainer" containerID="fbc7c364ba396d1b69164a6adaed4628887e2def9d3aea3d19915e8b0e527733"
Jun 02 15:59:34.548082 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:59:34.548042    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbc7c364ba396d1b69164a6adaed4628887e2def9d3aea3d19915e8b0e527733\": container with ID starting with fbc7c364ba396d1b69164a6adaed4628887e2def9d3aea3d19915e8b0e527733 not found: ID does not exist" containerID="fbc7c364ba396d1b69164a6adaed4628887e2def9d3aea3d19915e8b0e527733"
Jun 02 15:59:34.548082 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.548065    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbc7c364ba396d1b69164a6adaed4628887e2def9d3aea3d19915e8b0e527733"} err="failed to get container status \"fbc7c364ba396d1b69164a6adaed4628887e2def9d3aea3d19915e8b0e527733\": rpc error: code = NotFound desc = could not find container \"fbc7c364ba396d1b69164a6adaed4628887e2def9d3aea3d19915e8b0e527733\": container with ID starting with fbc7c364ba396d1b69164a6adaed4628887e2def9d3aea3d19915e8b0e527733 not found: ID does not exist"
Jun 02 15:59:34.548197 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.548085    2578 scope.go:117] "RemoveContainer" containerID="53d2cf147a8f2c6d6f59d62f10627ae3fdae30c1373deeb5be25568fa4e499c8"
Jun 02 15:59:34.548370 ip-10-0-140-246 kubenswrapper[2578]: E0602 15:59:34.548349    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53d2cf147a8f2c6d6f59d62f10627ae3fdae30c1373deeb5be25568fa4e499c8\": container with ID starting with 53d2cf147a8f2c6d6f59d62f10627ae3fdae30c1373deeb5be25568fa4e499c8 not found: ID does not exist" containerID="53d2cf147a8f2c6d6f59d62f10627ae3fdae30c1373deeb5be25568fa4e499c8"
Jun 02 15:59:34.548475 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.548383    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53d2cf147a8f2c6d6f59d62f10627ae3fdae30c1373deeb5be25568fa4e499c8"} err="failed to get container status \"53d2cf147a8f2c6d6f59d62f10627ae3fdae30c1373deeb5be25568fa4e499c8\": rpc error: code = NotFound desc = could not find container \"53d2cf147a8f2c6d6f59d62f10627ae3fdae30c1373deeb5be25568fa4e499c8\": container with ID starting with 53d2cf147a8f2c6d6f59d62f10627ae3fdae30c1373deeb5be25568fa4e499c8 not found: ID does not exist"
Jun 02 15:59:34.550170 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.550151    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx"]
Jun 02 15:59:34.550440 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.550423    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a5dc465-78aa-4712-940a-e7c7a6f27fff-kserve-provision-location\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 15:59:34.550483 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.550444    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qx67\" (UniqueName: \"kubernetes.io/projected/5a5dc465-78aa-4712-940a-e7c7a6f27fff-kube-api-access-2qx67\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 15:59:34.550483 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.550454    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a5dc465-78aa-4712-940a-e7c7a6f27fff-proxy-tls\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 15:59:34.550483 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.550463    2578 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-graph-raw-hpa-ae9ae-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a5dc465-78aa-4712-940a-e7c7a6f27fff-isvc-sklearn-graph-raw-hpa-ae9ae-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 15:59:34.554273 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:34.554257    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ae9ae-predictor-6c79649c9b-g9wrx"]
Jun 02 15:59:35.907525 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:35.907494    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52548776-159b-4aa2-aa31-33accbfffe9f" path="/var/lib/kubelet/pods/52548776-159b-4aa2-aa31-33accbfffe9f/volumes"
Jun 02 15:59:35.908042 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:35.908027    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a5dc465-78aa-4712-940a-e7c7a6f27fff" path="/var/lib/kubelet/pods/5a5dc465-78aa-4712-940a-e7c7a6f27fff/volumes"
Jun 02 15:59:40.515673 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:40.515645    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-0d859-predictor-58bd984d44-v47p2"
Jun 02 15:59:50.124274 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:50.124239    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc"]
Jun 02 15:59:50.124718 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:50.124697    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a5dc465-78aa-4712-940a-e7c7a6f27fff" containerName="kube-rbac-proxy"
Jun 02 15:59:50.124785 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:50.124722    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a5dc465-78aa-4712-940a-e7c7a6f27fff" containerName="kube-rbac-proxy"
Jun 02 15:59:50.124785 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:50.124742    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a5dc465-78aa-4712-940a-e7c7a6f27fff" containerName="storage-initializer"
Jun 02 15:59:50.124785 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:50.124751    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a5dc465-78aa-4712-940a-e7c7a6f27fff" containerName="storage-initializer"
Jun 02 15:59:50.124785 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:50.124762    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52548776-159b-4aa2-aa31-33accbfffe9f" containerName="kube-rbac-proxy"
Jun 02 15:59:50.124785 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:50.124770    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="52548776-159b-4aa2-aa31-33accbfffe9f" containerName="kube-rbac-proxy"
Jun 02 15:59:50.125039 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:50.124797    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52548776-159b-4aa2-aa31-33accbfffe9f" containerName="kserve-container"
Jun 02 15:59:50.125039 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:50.124805    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="52548776-159b-4aa2-aa31-33accbfffe9f" containerName="kserve-container"
Jun 02 15:59:50.125039 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:50.124818    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a5dc465-78aa-4712-940a-e7c7a6f27fff" containerName="kserve-container"
Jun 02 15:59:50.125039 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:50.124827    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a5dc465-78aa-4712-940a-e7c7a6f27fff" containerName="kserve-container"
Jun 02 15:59:50.125039 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:50.124841    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52548776-159b-4aa2-aa31-33accbfffe9f" containerName="storage-initializer"
Jun 02 15:59:50.125039 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:50.124849    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="52548776-159b-4aa2-aa31-33accbfffe9f" containerName="storage-initializer"
Jun 02 15:59:50.125039 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:50.124915    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="52548776-159b-4aa2-aa31-33accbfffe9f" containerName="kserve-container"
Jun 02 15:59:50.125039 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:50.124926    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="5a5dc465-78aa-4712-940a-e7c7a6f27fff" containerName="kube-rbac-proxy"
Jun 02 15:59:50.125039 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:50.124940    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="5a5dc465-78aa-4712-940a-e7c7a6f27fff" containerName="kserve-container"
Jun 02 15:59:50.125039 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:50.124953    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="52548776-159b-4aa2-aa31-33accbfffe9f" containerName="kube-rbac-proxy"
Jun 02 15:59:50.127484 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:50.127464    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc"
Jun 02 15:59:50.130635 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:50.130611    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-0d859-predictor-serving-cert\""
Jun 02 15:59:50.130748 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:50.130731    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-0d859-kube-rbac-proxy-sar-config\""
Jun 02 15:59:50.143499 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:50.143475    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc"]
Jun 02 15:59:50.281513 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:50.281480    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84534d3c-6555-42a5-b308-9d8a82670acd-kserve-provision-location\") pod \"isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc\" (UID: \"84534d3c-6555-42a5-b308-9d8a82670acd\") " pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc"
Jun 02 15:59:50.281665 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:50.281522    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84534d3c-6555-42a5-b308-9d8a82670acd-proxy-tls\") pod \"isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc\" (UID: \"84534d3c-6555-42a5-b308-9d8a82670acd\") " pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc"
Jun 02 15:59:50.281665 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:50.281580    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5j7l\" (UniqueName: \"kubernetes.io/projected/84534d3c-6555-42a5-b308-9d8a82670acd-kube-api-access-c5j7l\") pod \"isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc\" (UID: \"84534d3c-6555-42a5-b308-9d8a82670acd\") " pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc"
Jun 02 15:59:50.281665 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:50.281631    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-logger-raw-0d859-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84534d3c-6555-42a5-b308-9d8a82670acd-isvc-logger-raw-0d859-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc\" (UID: \"84534d3c-6555-42a5-b308-9d8a82670acd\") " pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc"
Jun 02 15:59:50.382824 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:50.382745    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c5j7l\" (UniqueName: \"kubernetes.io/projected/84534d3c-6555-42a5-b308-9d8a82670acd-kube-api-access-c5j7l\") pod \"isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc\" (UID: \"84534d3c-6555-42a5-b308-9d8a82670acd\") " pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc"
Jun 02 15:59:50.382824 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:50.382803    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-logger-raw-0d859-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84534d3c-6555-42a5-b308-9d8a82670acd-isvc-logger-raw-0d859-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc\" (UID: \"84534d3c-6555-42a5-b308-9d8a82670acd\") " pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc"
Jun 02 15:59:50.383050 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:50.382864    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84534d3c-6555-42a5-b308-9d8a82670acd-kserve-provision-location\") pod \"isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc\" (UID: \"84534d3c-6555-42a5-b308-9d8a82670acd\") " pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc"
Jun 02 15:59:50.383050 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:50.382911    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84534d3c-6555-42a5-b308-9d8a82670acd-proxy-tls\") pod \"isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc\" (UID: \"84534d3c-6555-42a5-b308-9d8a82670acd\") " pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc"
Jun 02 15:59:50.383211 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:50.383192    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84534d3c-6555-42a5-b308-9d8a82670acd-kserve-provision-location\") pod \"isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc\" (UID: \"84534d3c-6555-42a5-b308-9d8a82670acd\") " pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc"
Jun 02 15:59:50.383413 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:50.383384    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-logger-raw-0d859-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84534d3c-6555-42a5-b308-9d8a82670acd-isvc-logger-raw-0d859-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc\" (UID: \"84534d3c-6555-42a5-b308-9d8a82670acd\") " pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc"
Jun 02 15:59:50.385189 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:50.385173    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84534d3c-6555-42a5-b308-9d8a82670acd-proxy-tls\") pod \"isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc\" (UID: \"84534d3c-6555-42a5-b308-9d8a82670acd\") " pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc"
Jun 02 15:59:50.395252 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:50.395231    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5j7l\" (UniqueName: \"kubernetes.io/projected/84534d3c-6555-42a5-b308-9d8a82670acd-kube-api-access-c5j7l\") pod \"isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc\" (UID: \"84534d3c-6555-42a5-b308-9d8a82670acd\") " pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc"
Jun 02 15:59:50.438066 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:50.438032    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc"
Jun 02 15:59:50.579506 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:50.579465    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc"]
Jun 02 15:59:50.582130 ip-10-0-140-246 kubenswrapper[2578]: W0602 15:59:50.582087    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84534d3c_6555_42a5_b308_9d8a82670acd.slice/crio-b78c60bf5e7f5fbc6e49d04b66144ddfb3f5cea95fe14a7bcfb98d644880f484 WatchSource:0}: Error finding container b78c60bf5e7f5fbc6e49d04b66144ddfb3f5cea95fe14a7bcfb98d644880f484: Status 404 returned error can't find the container with id b78c60bf5e7f5fbc6e49d04b66144ddfb3f5cea95fe14a7bcfb98d644880f484
Jun 02 15:59:51.560774 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:51.560722    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc" event={"ID":"84534d3c-6555-42a5-b308-9d8a82670acd","Type":"ContainerStarted","Data":"251330dba3445b98c348e20487a59a51aaf42e4a36e3117d3cae19610aa420b3"}
Jun 02 15:59:51.561196 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:51.560787    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc" event={"ID":"84534d3c-6555-42a5-b308-9d8a82670acd","Type":"ContainerStarted","Data":"b78c60bf5e7f5fbc6e49d04b66144ddfb3f5cea95fe14a7bcfb98d644880f484"}
Jun 02 15:59:54.569855 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:54.569822    2578 generic.go:358] "Generic (PLEG): container finished" podID="84534d3c-6555-42a5-b308-9d8a82670acd" containerID="251330dba3445b98c348e20487a59a51aaf42e4a36e3117d3cae19610aa420b3" exitCode=0
Jun 02 15:59:54.570199 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:54.569897    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc" event={"ID":"84534d3c-6555-42a5-b308-9d8a82670acd","Type":"ContainerDied","Data":"251330dba3445b98c348e20487a59a51aaf42e4a36e3117d3cae19610aa420b3"}
Jun 02 15:59:55.574468 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:55.574435    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc" event={"ID":"84534d3c-6555-42a5-b308-9d8a82670acd","Type":"ContainerStarted","Data":"472ece8f666d0b4b1381b656e1d32ba5aa3959cbb86760d63dedda2230909031"}
Jun 02 15:59:55.574860 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:55.574479    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc" event={"ID":"84534d3c-6555-42a5-b308-9d8a82670acd","Type":"ContainerStarted","Data":"7f157be2aba8328b69f4499634a8a49428370c8f417f3a471faf2cd8fff04fc1"}
Jun 02 15:59:55.574860 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:55.574490    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc" event={"ID":"84534d3c-6555-42a5-b308-9d8a82670acd","Type":"ContainerStarted","Data":"fc1c082aafcc5e95f4f4da8575bfbbe43be331a864b75d07abdd575246d44db2"}
Jun 02 15:59:55.574860 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:55.574804    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc"
Jun 02 15:59:55.575044 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:55.574935    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc"
Jun 02 15:59:55.576029 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:55.575985    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jun 02 15:59:55.599043 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:55.598990    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc" podStartSLOduration=5.598977803 podStartE2EDuration="5.598977803s" podCreationTimestamp="2026-06-02 15:59:50 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 15:59:55.596909045 +0000 UTC m=+844.299372563" watchObservedRunningTime="2026-06-02 15:59:55.598977803 +0000 UTC m=+844.301441322"
Jun 02 15:59:56.579987 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:56.579954    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc"
Jun 02 15:59:56.580459 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:56.580081    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jun 02 15:59:56.580935 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:56.580907    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 15:59:57.583264 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:57.583221    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jun 02 15:59:57.583691 ip-10-0-140-246 kubenswrapper[2578]: I0602 15:59:57.583658    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:00:02.587381 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:00:02.587351    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc"
Jun 02 16:00:02.587983 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:00:02.587949    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jun 02 16:00:02.588218 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:00:02.588195    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:00:12.588346 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:00:12.588241    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jun 02 16:00:12.588765 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:00:12.588649    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:00:22.587869 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:00:22.587827    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jun 02 16:00:22.588278 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:00:22.588180    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:00:32.588494 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:00:32.588446    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jun 02 16:00:32.588917 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:00:32.588869    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:00:42.588608 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:00:42.588520    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jun 02 16:00:42.589017 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:00:42.588976    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:00:52.588680 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:00:52.588642    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jun 02 16:00:52.589234 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:00:52.589149    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:01:02.589060 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:02.589028    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc"
Jun 02 16:01:02.589538 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:02.589162    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc"
Jun 02 16:01:15.058292 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.058263    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-0d859-predictor-58bd984d44-v47p2_14a65876-5613-49f5-89ad-6988b8d0aee7/kserve-container/0.log"
Jun 02 16:01:15.248529 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.248496    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc"]
Jun 02 16:01:15.248964 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.248933    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" containerName="kserve-container" containerID="cri-o://fc1c082aafcc5e95f4f4da8575bfbbe43be331a864b75d07abdd575246d44db2" gracePeriod=30
Jun 02 16:01:15.249209 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.249012    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" containerName="agent" containerID="cri-o://472ece8f666d0b4b1381b656e1d32ba5aa3959cbb86760d63dedda2230909031" gracePeriod=30
Jun 02 16:01:15.249300 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.249035    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" containerName="kube-rbac-proxy" containerID="cri-o://7f157be2aba8328b69f4499634a8a49428370c8f417f3a471faf2cd8fff04fc1" gracePeriod=30
Jun 02 16:01:15.310208 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.310147    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6"]
Jun 02 16:01:15.313194 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.313175    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6"
Jun 02 16:01:15.315241 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.315220    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-261e5-predictor-serving-cert\""
Jun 02 16:01:15.315241 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.315237    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-261e5-kube-rbac-proxy-sar-config\""
Jun 02 16:01:15.323826 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.323806    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6"]
Jun 02 16:01:15.366161 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.366133    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-0d859-predictor-58bd984d44-v47p2"]
Jun 02 16:01:15.366428 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.366401    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-0d859-predictor-58bd984d44-v47p2" podUID="14a65876-5613-49f5-89ad-6988b8d0aee7" containerName="kserve-container" containerID="cri-o://5e80cf55127bcc46bcdf2ddbf3f594e08dd16a740356b1827dd9e0a706c4b668" gracePeriod=30
Jun 02 16:01:15.366529 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.366434    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-0d859-predictor-58bd984d44-v47p2" podUID="14a65876-5613-49f5-89ad-6988b8d0aee7" containerName="kube-rbac-proxy" containerID="cri-o://64d15930efcc7ca70524a3f33654540d28ef6e6b583181331e0f8fd0f4e13fa7" gracePeriod=30
Jun 02 16:01:15.387164 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.387141    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-scale-raw-261e5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0061fcda-676e-40ef-a3af-7c90f1ea2a65-isvc-sklearn-scale-raw-261e5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6\" (UID: \"0061fcda-676e-40ef-a3af-7c90f1ea2a65\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6"
Jun 02 16:01:15.387239 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.387180    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0061fcda-676e-40ef-a3af-7c90f1ea2a65-proxy-tls\") pod \"isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6\" (UID: \"0061fcda-676e-40ef-a3af-7c90f1ea2a65\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6"
Jun 02 16:01:15.387239 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.387200    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km6sl\" (UniqueName: \"kubernetes.io/projected/0061fcda-676e-40ef-a3af-7c90f1ea2a65-kube-api-access-km6sl\") pod \"isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6\" (UID: \"0061fcda-676e-40ef-a3af-7c90f1ea2a65\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6"
Jun 02 16:01:15.387336 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.387284    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0061fcda-676e-40ef-a3af-7c90f1ea2a65-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6\" (UID: \"0061fcda-676e-40ef-a3af-7c90f1ea2a65\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6"
Jun 02 16:01:15.487706 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.487679    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-scale-raw-261e5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0061fcda-676e-40ef-a3af-7c90f1ea2a65-isvc-sklearn-scale-raw-261e5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6\" (UID: \"0061fcda-676e-40ef-a3af-7c90f1ea2a65\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6"
Jun 02 16:01:15.487814 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.487717    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0061fcda-676e-40ef-a3af-7c90f1ea2a65-proxy-tls\") pod \"isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6\" (UID: \"0061fcda-676e-40ef-a3af-7c90f1ea2a65\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6"
Jun 02 16:01:15.487814 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.487739    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-km6sl\" (UniqueName: \"kubernetes.io/projected/0061fcda-676e-40ef-a3af-7c90f1ea2a65-kube-api-access-km6sl\") pod \"isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6\" (UID: \"0061fcda-676e-40ef-a3af-7c90f1ea2a65\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6"
Jun 02 16:01:15.487814 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.487762    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0061fcda-676e-40ef-a3af-7c90f1ea2a65-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6\" (UID: \"0061fcda-676e-40ef-a3af-7c90f1ea2a65\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6"
Jun 02 16:01:15.487981 ip-10-0-140-246 kubenswrapper[2578]: E0602 16:01:15.487863    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-serving-cert: secret "isvc-sklearn-scale-raw-261e5-predictor-serving-cert" not found
Jun 02 16:01:15.487981 ip-10-0-140-246 kubenswrapper[2578]: E0602 16:01:15.487958    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0061fcda-676e-40ef-a3af-7c90f1ea2a65-proxy-tls podName:0061fcda-676e-40ef-a3af-7c90f1ea2a65 nodeName:}" failed. No retries permitted until 2026-06-02 16:01:15.987936062 +0000 UTC m=+924.690399576 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0061fcda-676e-40ef-a3af-7c90f1ea2a65-proxy-tls") pod "isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6" (UID: "0061fcda-676e-40ef-a3af-7c90f1ea2a65") : secret "isvc-sklearn-scale-raw-261e5-predictor-serving-cert" not found
Jun 02 16:01:15.488304 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.488279    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0061fcda-676e-40ef-a3af-7c90f1ea2a65-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6\" (UID: \"0061fcda-676e-40ef-a3af-7c90f1ea2a65\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6"
Jun 02 16:01:15.488757 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.488735    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-scale-raw-261e5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0061fcda-676e-40ef-a3af-7c90f1ea2a65-isvc-sklearn-scale-raw-261e5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6\" (UID: \"0061fcda-676e-40ef-a3af-7c90f1ea2a65\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6"
Jun 02 16:01:15.499591 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.499572    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km6sl\" (UniqueName: \"kubernetes.io/projected/0061fcda-676e-40ef-a3af-7c90f1ea2a65-kube-api-access-km6sl\") pod \"isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6\" (UID: \"0061fcda-676e-40ef-a3af-7c90f1ea2a65\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6"
Jun 02 16:01:15.512679 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.512650    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/message-dumper-raw-0d859-predictor-58bd984d44-v47p2" podUID="14a65876-5613-49f5-89ad-6988b8d0aee7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jun 02 16:01:15.605520 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.605497    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-0d859-predictor-58bd984d44-v47p2"
Jun 02 16:01:15.689689 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.689659    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14a65876-5613-49f5-89ad-6988b8d0aee7-proxy-tls\") pod \"14a65876-5613-49f5-89ad-6988b8d0aee7\" (UID: \"14a65876-5613-49f5-89ad-6988b8d0aee7\") "
Jun 02 16:01:15.689828 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.689705    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"message-dumper-raw-0d859-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14a65876-5613-49f5-89ad-6988b8d0aee7-message-dumper-raw-0d859-kube-rbac-proxy-sar-config\") pod \"14a65876-5613-49f5-89ad-6988b8d0aee7\" (UID: \"14a65876-5613-49f5-89ad-6988b8d0aee7\") "
Jun 02 16:01:15.689828 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.689760    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5td4z\" (UniqueName: \"kubernetes.io/projected/14a65876-5613-49f5-89ad-6988b8d0aee7-kube-api-access-5td4z\") pod \"14a65876-5613-49f5-89ad-6988b8d0aee7\" (UID: \"14a65876-5613-49f5-89ad-6988b8d0aee7\") "
Jun 02 16:01:15.690148 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.690125    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14a65876-5613-49f5-89ad-6988b8d0aee7-message-dumper-raw-0d859-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "message-dumper-raw-0d859-kube-rbac-proxy-sar-config") pod "14a65876-5613-49f5-89ad-6988b8d0aee7" (UID: "14a65876-5613-49f5-89ad-6988b8d0aee7"). InnerVolumeSpecName "message-dumper-raw-0d859-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:01:15.691729 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.691708    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14a65876-5613-49f5-89ad-6988b8d0aee7-kube-api-access-5td4z" (OuterVolumeSpecName: "kube-api-access-5td4z") pod "14a65876-5613-49f5-89ad-6988b8d0aee7" (UID: "14a65876-5613-49f5-89ad-6988b8d0aee7"). InnerVolumeSpecName "kube-api-access-5td4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:01:15.691796 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.691750    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14a65876-5613-49f5-89ad-6988b8d0aee7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "14a65876-5613-49f5-89ad-6988b8d0aee7" (UID: "14a65876-5613-49f5-89ad-6988b8d0aee7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:01:15.790620 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.790586    2578 reconciler_common.go:299] "Volume detached for volume \"message-dumper-raw-0d859-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14a65876-5613-49f5-89ad-6988b8d0aee7-message-dumper-raw-0d859-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 16:01:15.790620 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.790616    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5td4z\" (UniqueName: \"kubernetes.io/projected/14a65876-5613-49f5-89ad-6988b8d0aee7-kube-api-access-5td4z\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 16:01:15.790778 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.790632    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14a65876-5613-49f5-89ad-6988b8d0aee7-proxy-tls\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 16:01:15.814884 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.814848    2578 generic.go:358] "Generic (PLEG): container finished" podID="14a65876-5613-49f5-89ad-6988b8d0aee7" containerID="64d15930efcc7ca70524a3f33654540d28ef6e6b583181331e0f8fd0f4e13fa7" exitCode=2
Jun 02 16:01:15.814884 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.814876    2578 generic.go:358] "Generic (PLEG): container finished" podID="14a65876-5613-49f5-89ad-6988b8d0aee7" containerID="5e80cf55127bcc46bcdf2ddbf3f594e08dd16a740356b1827dd9e0a706c4b668" exitCode=2
Jun 02 16:01:15.815099 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.814928    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-0d859-predictor-58bd984d44-v47p2"
Jun 02 16:01:15.815099 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.814931    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-0d859-predictor-58bd984d44-v47p2" event={"ID":"14a65876-5613-49f5-89ad-6988b8d0aee7","Type":"ContainerDied","Data":"64d15930efcc7ca70524a3f33654540d28ef6e6b583181331e0f8fd0f4e13fa7"}
Jun 02 16:01:15.815099 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.814968    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-0d859-predictor-58bd984d44-v47p2" event={"ID":"14a65876-5613-49f5-89ad-6988b8d0aee7","Type":"ContainerDied","Data":"5e80cf55127bcc46bcdf2ddbf3f594e08dd16a740356b1827dd9e0a706c4b668"}
Jun 02 16:01:15.815099 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.814979    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-0d859-predictor-58bd984d44-v47p2" event={"ID":"14a65876-5613-49f5-89ad-6988b8d0aee7","Type":"ContainerDied","Data":"f9280f6784146049e81a6b491c67a375599e088a447db638f290eaabe450f335"}
Jun 02 16:01:15.815099 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.815010    2578 scope.go:117] "RemoveContainer" containerID="64d15930efcc7ca70524a3f33654540d28ef6e6b583181331e0f8fd0f4e13fa7"
Jun 02 16:01:15.817239 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.817218    2578 generic.go:358] "Generic (PLEG): container finished" podID="84534d3c-6555-42a5-b308-9d8a82670acd" containerID="7f157be2aba8328b69f4499634a8a49428370c8f417f3a471faf2cd8fff04fc1" exitCode=2
Jun 02 16:01:15.817347 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.817279    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc" event={"ID":"84534d3c-6555-42a5-b308-9d8a82670acd","Type":"ContainerDied","Data":"7f157be2aba8328b69f4499634a8a49428370c8f417f3a471faf2cd8fff04fc1"}
Jun 02 16:01:15.823585 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.823563    2578 scope.go:117] "RemoveContainer" containerID="5e80cf55127bcc46bcdf2ddbf3f594e08dd16a740356b1827dd9e0a706c4b668"
Jun 02 16:01:15.830691 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.830674    2578 scope.go:117] "RemoveContainer" containerID="64d15930efcc7ca70524a3f33654540d28ef6e6b583181331e0f8fd0f4e13fa7"
Jun 02 16:01:15.830960 ip-10-0-140-246 kubenswrapper[2578]: E0602 16:01:15.830943    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64d15930efcc7ca70524a3f33654540d28ef6e6b583181331e0f8fd0f4e13fa7\": container with ID starting with 64d15930efcc7ca70524a3f33654540d28ef6e6b583181331e0f8fd0f4e13fa7 not found: ID does not exist" containerID="64d15930efcc7ca70524a3f33654540d28ef6e6b583181331e0f8fd0f4e13fa7"
Jun 02 16:01:15.831027 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.830966    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64d15930efcc7ca70524a3f33654540d28ef6e6b583181331e0f8fd0f4e13fa7"} err="failed to get container status \"64d15930efcc7ca70524a3f33654540d28ef6e6b583181331e0f8fd0f4e13fa7\": rpc error: code = NotFound desc = could not find container \"64d15930efcc7ca70524a3f33654540d28ef6e6b583181331e0f8fd0f4e13fa7\": container with ID starting with 64d15930efcc7ca70524a3f33654540d28ef6e6b583181331e0f8fd0f4e13fa7 not found: ID does not exist"
Jun 02 16:01:15.831027 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.830982    2578 scope.go:117] "RemoveContainer" containerID="5e80cf55127bcc46bcdf2ddbf3f594e08dd16a740356b1827dd9e0a706c4b668"
Jun 02 16:01:15.831248 ip-10-0-140-246 kubenswrapper[2578]: E0602 16:01:15.831229    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e80cf55127bcc46bcdf2ddbf3f594e08dd16a740356b1827dd9e0a706c4b668\": container with ID starting with 5e80cf55127bcc46bcdf2ddbf3f594e08dd16a740356b1827dd9e0a706c4b668 not found: ID does not exist" containerID="5e80cf55127bcc46bcdf2ddbf3f594e08dd16a740356b1827dd9e0a706c4b668"
Jun 02 16:01:15.831291 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.831254    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e80cf55127bcc46bcdf2ddbf3f594e08dd16a740356b1827dd9e0a706c4b668"} err="failed to get container status \"5e80cf55127bcc46bcdf2ddbf3f594e08dd16a740356b1827dd9e0a706c4b668\": rpc error: code = NotFound desc = could not find container \"5e80cf55127bcc46bcdf2ddbf3f594e08dd16a740356b1827dd9e0a706c4b668\": container with ID starting with 5e80cf55127bcc46bcdf2ddbf3f594e08dd16a740356b1827dd9e0a706c4b668 not found: ID does not exist"
Jun 02 16:01:15.831291 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.831274    2578 scope.go:117] "RemoveContainer" containerID="64d15930efcc7ca70524a3f33654540d28ef6e6b583181331e0f8fd0f4e13fa7"
Jun 02 16:01:15.831506 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.831490    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64d15930efcc7ca70524a3f33654540d28ef6e6b583181331e0f8fd0f4e13fa7"} err="failed to get container status \"64d15930efcc7ca70524a3f33654540d28ef6e6b583181331e0f8fd0f4e13fa7\": rpc error: code = NotFound desc = could not find container \"64d15930efcc7ca70524a3f33654540d28ef6e6b583181331e0f8fd0f4e13fa7\": container with ID starting with 64d15930efcc7ca70524a3f33654540d28ef6e6b583181331e0f8fd0f4e13fa7 not found: ID does not exist"
Jun 02 16:01:15.831564 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.831507    2578 scope.go:117] "RemoveContainer" containerID="5e80cf55127bcc46bcdf2ddbf3f594e08dd16a740356b1827dd9e0a706c4b668"
Jun 02 16:01:15.831714 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.831699    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e80cf55127bcc46bcdf2ddbf3f594e08dd16a740356b1827dd9e0a706c4b668"} err="failed to get container status \"5e80cf55127bcc46bcdf2ddbf3f594e08dd16a740356b1827dd9e0a706c4b668\": rpc error: code = NotFound desc = could not find container \"5e80cf55127bcc46bcdf2ddbf3f594e08dd16a740356b1827dd9e0a706c4b668\": container with ID starting with 5e80cf55127bcc46bcdf2ddbf3f594e08dd16a740356b1827dd9e0a706c4b668 not found: ID does not exist"
Jun 02 16:01:15.836754 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.836727    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-0d859-predictor-58bd984d44-v47p2"]
Jun 02 16:01:15.838397 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.838374    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-0d859-predictor-58bd984d44-v47p2"]
Jun 02 16:01:15.907525 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.907494    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14a65876-5613-49f5-89ad-6988b8d0aee7" path="/var/lib/kubelet/pods/14a65876-5613-49f5-89ad-6988b8d0aee7/volumes"
Jun 02 16:01:15.992331 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.992300    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0061fcda-676e-40ef-a3af-7c90f1ea2a65-proxy-tls\") pod \"isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6\" (UID: \"0061fcda-676e-40ef-a3af-7c90f1ea2a65\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6"
Jun 02 16:01:15.994586 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:15.994567    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0061fcda-676e-40ef-a3af-7c90f1ea2a65-proxy-tls\") pod \"isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6\" (UID: \"0061fcda-676e-40ef-a3af-7c90f1ea2a65\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6"
Jun 02 16:01:16.224366 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:16.224280    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6"
Jun 02 16:01:16.352369 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:16.351983    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6"]
Jun 02 16:01:16.355213 ip-10-0-140-246 kubenswrapper[2578]: W0602 16:01:16.355035    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0061fcda_676e_40ef_a3af_7c90f1ea2a65.slice/crio-1b0778d41ae14d9337597f7fa27bc45dbb7471f8fb95acbba08a08dfbaacd949 WatchSource:0}: Error finding container 1b0778d41ae14d9337597f7fa27bc45dbb7471f8fb95acbba08a08dfbaacd949: Status 404 returned error can't find the container with id 1b0778d41ae14d9337597f7fa27bc45dbb7471f8fb95acbba08a08dfbaacd949
Jun 02 16:01:16.821808 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:16.821769    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6" event={"ID":"0061fcda-676e-40ef-a3af-7c90f1ea2a65","Type":"ContainerStarted","Data":"bb79270af56a3eb8acbf8735ed18b64f54c00449ccacd0170a8f5ff1b1102314"}
Jun 02 16:01:16.821808 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:16.821814    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6" event={"ID":"0061fcda-676e-40ef-a3af-7c90f1ea2a65","Type":"ContainerStarted","Data":"1b0778d41ae14d9337597f7fa27bc45dbb7471f8fb95acbba08a08dfbaacd949"}
Jun 02 16:01:17.583889 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:17.583853    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jun 02 16:01:19.838966 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:19.838931    2578 generic.go:358] "Generic (PLEG): container finished" podID="84534d3c-6555-42a5-b308-9d8a82670acd" containerID="fc1c082aafcc5e95f4f4da8575bfbbe43be331a864b75d07abdd575246d44db2" exitCode=0
Jun 02 16:01:19.839336 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:19.839008    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc" event={"ID":"84534d3c-6555-42a5-b308-9d8a82670acd","Type":"ContainerDied","Data":"fc1c082aafcc5e95f4f4da8575bfbbe43be331a864b75d07abdd575246d44db2"}
Jun 02 16:01:20.842920 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:20.842887    2578 generic.go:358] "Generic (PLEG): container finished" podID="0061fcda-676e-40ef-a3af-7c90f1ea2a65" containerID="bb79270af56a3eb8acbf8735ed18b64f54c00449ccacd0170a8f5ff1b1102314" exitCode=0
Jun 02 16:01:20.843297 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:20.842960    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6" event={"ID":"0061fcda-676e-40ef-a3af-7c90f1ea2a65","Type":"ContainerDied","Data":"bb79270af56a3eb8acbf8735ed18b64f54c00449ccacd0170a8f5ff1b1102314"}
Jun 02 16:01:21.848223 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:21.848190    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6" event={"ID":"0061fcda-676e-40ef-a3af-7c90f1ea2a65","Type":"ContainerStarted","Data":"c624a9b3b3d8c1137a2fef63ae982483222e8c7f2d11c3c95e99058fc913d6bb"}
Jun 02 16:01:21.848223 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:21.848227    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6" event={"ID":"0061fcda-676e-40ef-a3af-7c90f1ea2a65","Type":"ContainerStarted","Data":"02ea97f297c4c7347f0336e42dba55f94fbc1baa284f8546f3c269f9b9217210"}
Jun 02 16:01:21.848617 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:21.848425    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6"
Jun 02 16:01:21.870895 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:21.870855    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6" podStartSLOduration=6.870842134 podStartE2EDuration="6.870842134s" podCreationTimestamp="2026-06-02 16:01:15 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:01:21.869056283 +0000 UTC m=+930.571519799" watchObservedRunningTime="2026-06-02 16:01:21.870842134 +0000 UTC m=+930.573305652"
Jun 02 16:01:22.584284 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:22.584242    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jun 02 16:01:22.588527 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:22.588492    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jun 02 16:01:22.588843 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:22.588820    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:01:22.851035 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:22.850933    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6"
Jun 02 16:01:22.852008 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:22.851968    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6" podUID="0061fcda-676e-40ef-a3af-7c90f1ea2a65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jun 02 16:01:23.856635 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:23.856595    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6" podUID="0061fcda-676e-40ef-a3af-7c90f1ea2a65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jun 02 16:01:27.584408 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:27.584364    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jun 02 16:01:27.584759 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:27.584529    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc"
Jun 02 16:01:28.861209 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:28.861184    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6"
Jun 02 16:01:28.863504 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:28.861756    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6" podUID="0061fcda-676e-40ef-a3af-7c90f1ea2a65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jun 02 16:01:32.583736 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:32.583692    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jun 02 16:01:32.587920 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:32.587888    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jun 02 16:01:32.588324 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:32.588304    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:01:37.583486 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:37.583440    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jun 02 16:01:38.862563 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:38.862525    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6" podUID="0061fcda-676e-40ef-a3af-7c90f1ea2a65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jun 02 16:01:42.584235 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:42.584192    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jun 02 16:01:42.588552 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:42.588525    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jun 02 16:01:42.588680 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:42.588633    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc"
Jun 02 16:01:42.588920 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:42.588900    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:01:42.589044 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:42.589032    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc"
Jun 02 16:01:45.437383 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:45.437360    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc"
Jun 02 16:01:45.547013 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:45.546917    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84534d3c-6555-42a5-b308-9d8a82670acd-proxy-tls\") pod \"84534d3c-6555-42a5-b308-9d8a82670acd\" (UID: \"84534d3c-6555-42a5-b308-9d8a82670acd\") "
Jun 02 16:01:45.547013 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:45.546975    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5j7l\" (UniqueName: \"kubernetes.io/projected/84534d3c-6555-42a5-b308-9d8a82670acd-kube-api-access-c5j7l\") pod \"84534d3c-6555-42a5-b308-9d8a82670acd\" (UID: \"84534d3c-6555-42a5-b308-9d8a82670acd\") "
Jun 02 16:01:45.547226 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:45.547021    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-logger-raw-0d859-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84534d3c-6555-42a5-b308-9d8a82670acd-isvc-logger-raw-0d859-kube-rbac-proxy-sar-config\") pod \"84534d3c-6555-42a5-b308-9d8a82670acd\" (UID: \"84534d3c-6555-42a5-b308-9d8a82670acd\") "
Jun 02 16:01:45.547226 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:45.547057    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84534d3c-6555-42a5-b308-9d8a82670acd-kserve-provision-location\") pod \"84534d3c-6555-42a5-b308-9d8a82670acd\" (UID: \"84534d3c-6555-42a5-b308-9d8a82670acd\") "
Jun 02 16:01:45.547410 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:45.547376    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84534d3c-6555-42a5-b308-9d8a82670acd-isvc-logger-raw-0d859-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-logger-raw-0d859-kube-rbac-proxy-sar-config") pod "84534d3c-6555-42a5-b308-9d8a82670acd" (UID: "84534d3c-6555-42a5-b308-9d8a82670acd"). InnerVolumeSpecName "isvc-logger-raw-0d859-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:01:45.547410 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:45.547395    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84534d3c-6555-42a5-b308-9d8a82670acd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "84534d3c-6555-42a5-b308-9d8a82670acd" (UID: "84534d3c-6555-42a5-b308-9d8a82670acd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:01:45.548945 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:45.548924    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84534d3c-6555-42a5-b308-9d8a82670acd-kube-api-access-c5j7l" (OuterVolumeSpecName: "kube-api-access-c5j7l") pod "84534d3c-6555-42a5-b308-9d8a82670acd" (UID: "84534d3c-6555-42a5-b308-9d8a82670acd"). InnerVolumeSpecName "kube-api-access-c5j7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:01:45.549042 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:45.549022    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84534d3c-6555-42a5-b308-9d8a82670acd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "84534d3c-6555-42a5-b308-9d8a82670acd" (UID: "84534d3c-6555-42a5-b308-9d8a82670acd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:01:45.648630 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:45.648601    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84534d3c-6555-42a5-b308-9d8a82670acd-proxy-tls\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 16:01:45.648630 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:45.648626    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5j7l\" (UniqueName: \"kubernetes.io/projected/84534d3c-6555-42a5-b308-9d8a82670acd-kube-api-access-c5j7l\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 16:01:45.648630 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:45.648635    2578 reconciler_common.go:299] "Volume detached for volume \"isvc-logger-raw-0d859-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84534d3c-6555-42a5-b308-9d8a82670acd-isvc-logger-raw-0d859-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 16:01:45.648827 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:45.648646    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84534d3c-6555-42a5-b308-9d8a82670acd-kserve-provision-location\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 16:01:45.921058 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:45.921026    2578 generic.go:358] "Generic (PLEG): container finished" podID="84534d3c-6555-42a5-b308-9d8a82670acd" containerID="472ece8f666d0b4b1381b656e1d32ba5aa3959cbb86760d63dedda2230909031" exitCode=0
Jun 02 16:01:45.921183 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:45.921066    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc" event={"ID":"84534d3c-6555-42a5-b308-9d8a82670acd","Type":"ContainerDied","Data":"472ece8f666d0b4b1381b656e1d32ba5aa3959cbb86760d63dedda2230909031"}
Jun 02 16:01:45.921183 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:45.921089    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc" event={"ID":"84534d3c-6555-42a5-b308-9d8a82670acd","Type":"ContainerDied","Data":"b78c60bf5e7f5fbc6e49d04b66144ddfb3f5cea95fe14a7bcfb98d644880f484"}
Jun 02 16:01:45.921183 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:45.921103    2578 scope.go:117] "RemoveContainer" containerID="472ece8f666d0b4b1381b656e1d32ba5aa3959cbb86760d63dedda2230909031"
Jun 02 16:01:45.921183 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:45.921121    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc"
Jun 02 16:01:45.928796 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:45.928776    2578 scope.go:117] "RemoveContainer" containerID="7f157be2aba8328b69f4499634a8a49428370c8f417f3a471faf2cd8fff04fc1"
Jun 02 16:01:45.937489 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:45.937456    2578 scope.go:117] "RemoveContainer" containerID="fc1c082aafcc5e95f4f4da8575bfbbe43be331a864b75d07abdd575246d44db2"
Jun 02 16:01:45.940771 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:45.940745    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc"]
Jun 02 16:01:45.944986 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:45.944969    2578 scope.go:117] "RemoveContainer" containerID="251330dba3445b98c348e20487a59a51aaf42e4a36e3117d3cae19610aa420b3"
Jun 02 16:01:45.945873 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:45.945841    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-0d859-predictor-7cf485cd8c-8kvfc"]
Jun 02 16:01:45.951536 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:45.951522    2578 scope.go:117] "RemoveContainer" containerID="472ece8f666d0b4b1381b656e1d32ba5aa3959cbb86760d63dedda2230909031"
Jun 02 16:01:45.951771 ip-10-0-140-246 kubenswrapper[2578]: E0602 16:01:45.951751    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"472ece8f666d0b4b1381b656e1d32ba5aa3959cbb86760d63dedda2230909031\": container with ID starting with 472ece8f666d0b4b1381b656e1d32ba5aa3959cbb86760d63dedda2230909031 not found: ID does not exist" containerID="472ece8f666d0b4b1381b656e1d32ba5aa3959cbb86760d63dedda2230909031"
Jun 02 16:01:45.951843 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:45.951791    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"472ece8f666d0b4b1381b656e1d32ba5aa3959cbb86760d63dedda2230909031"} err="failed to get container status \"472ece8f666d0b4b1381b656e1d32ba5aa3959cbb86760d63dedda2230909031\": rpc error: code = NotFound desc = could not find container \"472ece8f666d0b4b1381b656e1d32ba5aa3959cbb86760d63dedda2230909031\": container with ID starting with 472ece8f666d0b4b1381b656e1d32ba5aa3959cbb86760d63dedda2230909031 not found: ID does not exist"
Jun 02 16:01:45.951843 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:45.951815    2578 scope.go:117] "RemoveContainer" containerID="7f157be2aba8328b69f4499634a8a49428370c8f417f3a471faf2cd8fff04fc1"
Jun 02 16:01:45.952036 ip-10-0-140-246 kubenswrapper[2578]: E0602 16:01:45.952018    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f157be2aba8328b69f4499634a8a49428370c8f417f3a471faf2cd8fff04fc1\": container with ID starting with 7f157be2aba8328b69f4499634a8a49428370c8f417f3a471faf2cd8fff04fc1 not found: ID does not exist" containerID="7f157be2aba8328b69f4499634a8a49428370c8f417f3a471faf2cd8fff04fc1"
Jun 02 16:01:45.952081 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:45.952042    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f157be2aba8328b69f4499634a8a49428370c8f417f3a471faf2cd8fff04fc1"} err="failed to get container status \"7f157be2aba8328b69f4499634a8a49428370c8f417f3a471faf2cd8fff04fc1\": rpc error: code = NotFound desc = could not find container \"7f157be2aba8328b69f4499634a8a49428370c8f417f3a471faf2cd8fff04fc1\": container with ID starting with 7f157be2aba8328b69f4499634a8a49428370c8f417f3a471faf2cd8fff04fc1 not found: ID does not exist"
Jun 02 16:01:45.952081 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:45.952057    2578 scope.go:117] "RemoveContainer" containerID="fc1c082aafcc5e95f4f4da8575bfbbe43be331a864b75d07abdd575246d44db2"
Jun 02 16:01:45.952268 ip-10-0-140-246 kubenswrapper[2578]: E0602 16:01:45.952250    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc1c082aafcc5e95f4f4da8575bfbbe43be331a864b75d07abdd575246d44db2\": container with ID starting with fc1c082aafcc5e95f4f4da8575bfbbe43be331a864b75d07abdd575246d44db2 not found: ID does not exist" containerID="fc1c082aafcc5e95f4f4da8575bfbbe43be331a864b75d07abdd575246d44db2"
Jun 02 16:01:45.952306 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:45.952275    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc1c082aafcc5e95f4f4da8575bfbbe43be331a864b75d07abdd575246d44db2"} err="failed to get container status \"fc1c082aafcc5e95f4f4da8575bfbbe43be331a864b75d07abdd575246d44db2\": rpc error: code = NotFound desc = could not find container \"fc1c082aafcc5e95f4f4da8575bfbbe43be331a864b75d07abdd575246d44db2\": container with ID starting with fc1c082aafcc5e95f4f4da8575bfbbe43be331a864b75d07abdd575246d44db2 not found: ID does not exist"
Jun 02 16:01:45.952306 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:45.952290    2578 scope.go:117] "RemoveContainer" containerID="251330dba3445b98c348e20487a59a51aaf42e4a36e3117d3cae19610aa420b3"
Jun 02 16:01:45.952512 ip-10-0-140-246 kubenswrapper[2578]: E0602 16:01:45.952493    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"251330dba3445b98c348e20487a59a51aaf42e4a36e3117d3cae19610aa420b3\": container with ID starting with 251330dba3445b98c348e20487a59a51aaf42e4a36e3117d3cae19610aa420b3 not found: ID does not exist" containerID="251330dba3445b98c348e20487a59a51aaf42e4a36e3117d3cae19610aa420b3"
Jun 02 16:01:45.952599 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:45.952514    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"251330dba3445b98c348e20487a59a51aaf42e4a36e3117d3cae19610aa420b3"} err="failed to get container status \"251330dba3445b98c348e20487a59a51aaf42e4a36e3117d3cae19610aa420b3\": rpc error: code = NotFound desc = could not find container \"251330dba3445b98c348e20487a59a51aaf42e4a36e3117d3cae19610aa420b3\": container with ID starting with 251330dba3445b98c348e20487a59a51aaf42e4a36e3117d3cae19610aa420b3 not found: ID does not exist"
Jun 02 16:01:47.907731 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:47.907701    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" path="/var/lib/kubelet/pods/84534d3c-6555-42a5-b308-9d8a82670acd/volumes"
Jun 02 16:01:48.862060 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:48.862023    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6" podUID="0061fcda-676e-40ef-a3af-7c90f1ea2a65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jun 02 16:01:58.861977 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:01:58.861936    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6" podUID="0061fcda-676e-40ef-a3af-7c90f1ea2a65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jun 02 16:02:08.862677 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:02:08.862634    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6" podUID="0061fcda-676e-40ef-a3af-7c90f1ea2a65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jun 02 16:02:18.861934 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:02:18.861893    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6" podUID="0061fcda-676e-40ef-a3af-7c90f1ea2a65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jun 02 16:02:28.862602 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:02:28.862557    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6" podUID="0061fcda-676e-40ef-a3af-7c90f1ea2a65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jun 02 16:02:38.861975 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:02:38.861936    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6" podUID="0061fcda-676e-40ef-a3af-7c90f1ea2a65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jun 02 16:02:44.903894 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:02:44.903856    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6" podUID="0061fcda-676e-40ef-a3af-7c90f1ea2a65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jun 02 16:02:54.904452 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:02:54.904410    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6" podUID="0061fcda-676e-40ef-a3af-7c90f1ea2a65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jun 02 16:03:04.904770 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:04.904728    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6" podUID="0061fcda-676e-40ef-a3af-7c90f1ea2a65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jun 02 16:03:14.904523 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:14.904485    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6" podUID="0061fcda-676e-40ef-a3af-7c90f1ea2a65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jun 02 16:03:24.903901 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:24.903859    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6" podUID="0061fcda-676e-40ef-a3af-7c90f1ea2a65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jun 02 16:03:34.904182 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:34.904151    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6"
Jun 02 16:03:35.679071 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:35.679039    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6"]
Jun 02 16:03:35.679399 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:35.679353    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6" podUID="0061fcda-676e-40ef-a3af-7c90f1ea2a65" containerName="kserve-container" containerID="cri-o://02ea97f297c4c7347f0336e42dba55f94fbc1baa284f8546f3c269f9b9217210" gracePeriod=30
Jun 02 16:03:35.679518 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:35.679390    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6" podUID="0061fcda-676e-40ef-a3af-7c90f1ea2a65" containerName="kube-rbac-proxy" containerID="cri-o://c624a9b3b3d8c1137a2fef63ae982483222e8c7f2d11c3c95e99058fc913d6bb" gracePeriod=30
Jun 02 16:03:35.840226 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:35.840190    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44"]
Jun 02 16:03:35.840595 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:35.840577    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" containerName="agent"
Jun 02 16:03:35.840701 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:35.840597    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" containerName="agent"
Jun 02 16:03:35.840701 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:35.840621    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14a65876-5613-49f5-89ad-6988b8d0aee7" containerName="kube-rbac-proxy"
Jun 02 16:03:35.840701 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:35.840629    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="14a65876-5613-49f5-89ad-6988b8d0aee7" containerName="kube-rbac-proxy"
Jun 02 16:03:35.840701 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:35.840639    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14a65876-5613-49f5-89ad-6988b8d0aee7" containerName="kserve-container"
Jun 02 16:03:35.840701 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:35.840647    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="14a65876-5613-49f5-89ad-6988b8d0aee7" containerName="kserve-container"
Jun 02 16:03:35.840701 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:35.840666    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" containerName="kserve-container"
Jun 02 16:03:35.840701 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:35.840674    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" containerName="kserve-container"
Jun 02 16:03:35.840701 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:35.840686    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" containerName="storage-initializer"
Jun 02 16:03:35.840701 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:35.840693    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" containerName="storage-initializer"
Jun 02 16:03:35.841142 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:35.840707    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" containerName="kube-rbac-proxy"
Jun 02 16:03:35.841142 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:35.840715    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" containerName="kube-rbac-proxy"
Jun 02 16:03:35.841142 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:35.840781    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" containerName="kserve-container"
Jun 02 16:03:35.841142 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:35.840793    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" containerName="kube-rbac-proxy"
Jun 02 16:03:35.841142 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:35.840805    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="14a65876-5613-49f5-89ad-6988b8d0aee7" containerName="kube-rbac-proxy"
Jun 02 16:03:35.841142 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:35.840817    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="14a65876-5613-49f5-89ad-6988b8d0aee7" containerName="kserve-container"
Jun 02 16:03:35.841142 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:35.840829    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="84534d3c-6555-42a5-b308-9d8a82670acd" containerName="agent"
Jun 02 16:03:35.843983 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:35.843965    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44"
Jun 02 16:03:35.846294 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:35.846274    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-cffc11-predictor-serving-cert\""
Jun 02 16:03:35.847672 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:35.847655    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-cffc11-kube-rbac-proxy-sar-config\""
Jun 02 16:03:35.860944 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:35.860922    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44"]
Jun 02 16:03:35.942146 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:35.942075    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-primary-cffc11-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7b6b9d3-a7bf-4793-8879-4dc2e14973a3-isvc-primary-cffc11-kube-rbac-proxy-sar-config\") pod \"isvc-primary-cffc11-predictor-6846fdc6ff-k4x44\" (UID: \"d7b6b9d3-a7bf-4793-8879-4dc2e14973a3\") " pod="kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44"
Jun 02 16:03:35.942146 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:35.942117    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7b6b9d3-a7bf-4793-8879-4dc2e14973a3-kserve-provision-location\") pod \"isvc-primary-cffc11-predictor-6846fdc6ff-k4x44\" (UID: \"d7b6b9d3-a7bf-4793-8879-4dc2e14973a3\") " pod="kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44"
Jun 02 16:03:35.942525 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:35.942156    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvw9b\" (UniqueName: \"kubernetes.io/projected/d7b6b9d3-a7bf-4793-8879-4dc2e14973a3-kube-api-access-kvw9b\") pod \"isvc-primary-cffc11-predictor-6846fdc6ff-k4x44\" (UID: \"d7b6b9d3-a7bf-4793-8879-4dc2e14973a3\") " pod="kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44"
Jun 02 16:03:35.942525 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:35.942217    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7b6b9d3-a7bf-4793-8879-4dc2e14973a3-proxy-tls\") pod \"isvc-primary-cffc11-predictor-6846fdc6ff-k4x44\" (UID: \"d7b6b9d3-a7bf-4793-8879-4dc2e14973a3\") " pod="kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44"
Jun 02 16:03:36.043274 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:36.043251    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7b6b9d3-a7bf-4793-8879-4dc2e14973a3-proxy-tls\") pod \"isvc-primary-cffc11-predictor-6846fdc6ff-k4x44\" (UID: \"d7b6b9d3-a7bf-4793-8879-4dc2e14973a3\") " pod="kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44"
Jun 02 16:03:36.043399 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:36.043336    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-primary-cffc11-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7b6b9d3-a7bf-4793-8879-4dc2e14973a3-isvc-primary-cffc11-kube-rbac-proxy-sar-config\") pod \"isvc-primary-cffc11-predictor-6846fdc6ff-k4x44\" (UID: \"d7b6b9d3-a7bf-4793-8879-4dc2e14973a3\") " pod="kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44"
Jun 02 16:03:36.043399 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:36.043370    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7b6b9d3-a7bf-4793-8879-4dc2e14973a3-kserve-provision-location\") pod \"isvc-primary-cffc11-predictor-6846fdc6ff-k4x44\" (UID: \"d7b6b9d3-a7bf-4793-8879-4dc2e14973a3\") " pod="kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44"
Jun 02 16:03:36.043512 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:36.043403    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kvw9b\" (UniqueName: \"kubernetes.io/projected/d7b6b9d3-a7bf-4793-8879-4dc2e14973a3-kube-api-access-kvw9b\") pod \"isvc-primary-cffc11-predictor-6846fdc6ff-k4x44\" (UID: \"d7b6b9d3-a7bf-4793-8879-4dc2e14973a3\") " pod="kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44"
Jun 02 16:03:36.043825 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:36.043802    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7b6b9d3-a7bf-4793-8879-4dc2e14973a3-kserve-provision-location\") pod \"isvc-primary-cffc11-predictor-6846fdc6ff-k4x44\" (UID: \"d7b6b9d3-a7bf-4793-8879-4dc2e14973a3\") " pod="kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44"
Jun 02 16:03:36.044060 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:36.044041    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-primary-cffc11-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7b6b9d3-a7bf-4793-8879-4dc2e14973a3-isvc-primary-cffc11-kube-rbac-proxy-sar-config\") pod \"isvc-primary-cffc11-predictor-6846fdc6ff-k4x44\" (UID: \"d7b6b9d3-a7bf-4793-8879-4dc2e14973a3\") " pod="kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44"
Jun 02 16:03:36.045649 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:36.045627    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7b6b9d3-a7bf-4793-8879-4dc2e14973a3-proxy-tls\") pod \"isvc-primary-cffc11-predictor-6846fdc6ff-k4x44\" (UID: \"d7b6b9d3-a7bf-4793-8879-4dc2e14973a3\") " pod="kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44"
Jun 02 16:03:36.059181 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:36.059160    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvw9b\" (UniqueName: \"kubernetes.io/projected/d7b6b9d3-a7bf-4793-8879-4dc2e14973a3-kube-api-access-kvw9b\") pod \"isvc-primary-cffc11-predictor-6846fdc6ff-k4x44\" (UID: \"d7b6b9d3-a7bf-4793-8879-4dc2e14973a3\") " pod="kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44"
Jun 02 16:03:36.156204 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:36.156174    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44"
Jun 02 16:03:36.245044 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:36.245012    2578 generic.go:358] "Generic (PLEG): container finished" podID="0061fcda-676e-40ef-a3af-7c90f1ea2a65" containerID="c624a9b3b3d8c1137a2fef63ae982483222e8c7f2d11c3c95e99058fc913d6bb" exitCode=2
Jun 02 16:03:36.245183 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:36.245022    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6" event={"ID":"0061fcda-676e-40ef-a3af-7c90f1ea2a65","Type":"ContainerDied","Data":"c624a9b3b3d8c1137a2fef63ae982483222e8c7f2d11c3c95e99058fc913d6bb"}
Jun 02 16:03:36.306570 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:36.306549    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44"]
Jun 02 16:03:36.308608 ip-10-0-140-246 kubenswrapper[2578]: W0602 16:03:36.308583    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7b6b9d3_a7bf_4793_8879_4dc2e14973a3.slice/crio-5b0f66b94ac03898742de89b8225be121ab860cbafab0fb6f38b2e776f291071 WatchSource:0}: Error finding container 5b0f66b94ac03898742de89b8225be121ab860cbafab0fb6f38b2e776f291071: Status 404 returned error can't find the container with id 5b0f66b94ac03898742de89b8225be121ab860cbafab0fb6f38b2e776f291071
Jun 02 16:03:37.249844 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:37.249810    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44" event={"ID":"d7b6b9d3-a7bf-4793-8879-4dc2e14973a3","Type":"ContainerStarted","Data":"ec458f66f9af54c5dd9aaedf7d5f87173a6828ad947f1fdb522c91c5c39d3b69"}
Jun 02 16:03:37.249844 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:37.249846    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44" event={"ID":"d7b6b9d3-a7bf-4793-8879-4dc2e14973a3","Type":"ContainerStarted","Data":"5b0f66b94ac03898742de89b8225be121ab860cbafab0fb6f38b2e776f291071"}
Jun 02 16:03:38.856816 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:38.856777    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6" podUID="0061fcda-676e-40ef-a3af-7c90f1ea2a65" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jun 02 16:03:40.260768 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:40.260735    2578 generic.go:358] "Generic (PLEG): container finished" podID="d7b6b9d3-a7bf-4793-8879-4dc2e14973a3" containerID="ec458f66f9af54c5dd9aaedf7d5f87173a6828ad947f1fdb522c91c5c39d3b69" exitCode=0
Jun 02 16:03:40.261254 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:40.260815    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44" event={"ID":"d7b6b9d3-a7bf-4793-8879-4dc2e14973a3","Type":"ContainerDied","Data":"ec458f66f9af54c5dd9aaedf7d5f87173a6828ad947f1fdb522c91c5c39d3b69"}
Jun 02 16:03:41.266082 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:41.266036    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44" event={"ID":"d7b6b9d3-a7bf-4793-8879-4dc2e14973a3","Type":"ContainerStarted","Data":"c8c649d4ad651f4694fb54cc3a32ba8835c7016d21a7bc4a8100ab80d3d78a88"}
Jun 02 16:03:41.266539 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:41.266093    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44" event={"ID":"d7b6b9d3-a7bf-4793-8879-4dc2e14973a3","Type":"ContainerStarted","Data":"91f3d848bc66e015dc2db927ff8718634db43c195b20fab5a27c96ee8757e122"}
Jun 02 16:03:41.266539 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:41.266348    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44"
Jun 02 16:03:41.295126 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:41.295067    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44" podStartSLOduration=6.29504739 podStartE2EDuration="6.29504739s" podCreationTimestamp="2026-06-02 16:03:35 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:03:41.29435532 +0000 UTC m=+1069.996818835" watchObservedRunningTime="2026-06-02 16:03:41.29504739 +0000 UTC m=+1069.997510910"
Jun 02 16:03:42.271858 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:42.271827    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44"
Jun 02 16:03:42.272990 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:42.272957    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44" podUID="d7b6b9d3-a7bf-4793-8879-4dc2e14973a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jun 02 16:03:43.275221 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:43.275178    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44" podUID="d7b6b9d3-a7bf-4793-8879-4dc2e14973a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jun 02 16:03:43.857125 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:43.857083    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6" podUID="0061fcda-676e-40ef-a3af-7c90f1ea2a65" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jun 02 16:03:44.727249 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:44.727227    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6"
Jun 02 16:03:44.803902 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:44.803824    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0061fcda-676e-40ef-a3af-7c90f1ea2a65-kserve-provision-location\") pod \"0061fcda-676e-40ef-a3af-7c90f1ea2a65\" (UID: \"0061fcda-676e-40ef-a3af-7c90f1ea2a65\") "
Jun 02 16:03:44.803902 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:44.803863    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0061fcda-676e-40ef-a3af-7c90f1ea2a65-proxy-tls\") pod \"0061fcda-676e-40ef-a3af-7c90f1ea2a65\" (UID: \"0061fcda-676e-40ef-a3af-7c90f1ea2a65\") "
Jun 02 16:03:44.804185 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:44.803908    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-km6sl\" (UniqueName: \"kubernetes.io/projected/0061fcda-676e-40ef-a3af-7c90f1ea2a65-kube-api-access-km6sl\") pod \"0061fcda-676e-40ef-a3af-7c90f1ea2a65\" (UID: \"0061fcda-676e-40ef-a3af-7c90f1ea2a65\") "
Jun 02 16:03:44.804185 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:44.803954    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-scale-raw-261e5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0061fcda-676e-40ef-a3af-7c90f1ea2a65-isvc-sklearn-scale-raw-261e5-kube-rbac-proxy-sar-config\") pod \"0061fcda-676e-40ef-a3af-7c90f1ea2a65\" (UID: \"0061fcda-676e-40ef-a3af-7c90f1ea2a65\") "
Jun 02 16:03:44.804283 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:44.804216    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0061fcda-676e-40ef-a3af-7c90f1ea2a65-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0061fcda-676e-40ef-a3af-7c90f1ea2a65" (UID: "0061fcda-676e-40ef-a3af-7c90f1ea2a65"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:03:44.804318 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:44.804302    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0061fcda-676e-40ef-a3af-7c90f1ea2a65-isvc-sklearn-scale-raw-261e5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-scale-raw-261e5-kube-rbac-proxy-sar-config") pod "0061fcda-676e-40ef-a3af-7c90f1ea2a65" (UID: "0061fcda-676e-40ef-a3af-7c90f1ea2a65"). InnerVolumeSpecName "isvc-sklearn-scale-raw-261e5-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:03:44.806083 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:44.806060    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0061fcda-676e-40ef-a3af-7c90f1ea2a65-kube-api-access-km6sl" (OuterVolumeSpecName: "kube-api-access-km6sl") pod "0061fcda-676e-40ef-a3af-7c90f1ea2a65" (UID: "0061fcda-676e-40ef-a3af-7c90f1ea2a65"). InnerVolumeSpecName "kube-api-access-km6sl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:03:44.806167 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:44.806089    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0061fcda-676e-40ef-a3af-7c90f1ea2a65-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0061fcda-676e-40ef-a3af-7c90f1ea2a65" (UID: "0061fcda-676e-40ef-a3af-7c90f1ea2a65"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:03:44.904698 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:44.904660    2578 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-scale-raw-261e5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0061fcda-676e-40ef-a3af-7c90f1ea2a65-isvc-sklearn-scale-raw-261e5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 16:03:44.904698 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:44.904693    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0061fcda-676e-40ef-a3af-7c90f1ea2a65-kserve-provision-location\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 16:03:44.904698 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:44.904703    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0061fcda-676e-40ef-a3af-7c90f1ea2a65-proxy-tls\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 16:03:44.904961 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:44.904712    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km6sl\" (UniqueName: \"kubernetes.io/projected/0061fcda-676e-40ef-a3af-7c90f1ea2a65-kube-api-access-km6sl\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 16:03:45.282107 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:45.282075    2578 generic.go:358] "Generic (PLEG): container finished" podID="0061fcda-676e-40ef-a3af-7c90f1ea2a65" containerID="02ea97f297c4c7347f0336e42dba55f94fbc1baa284f8546f3c269f9b9217210" exitCode=0
Jun 02 16:03:45.282253 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:45.282154    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6"
Jun 02 16:03:45.282253 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:45.282153    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6" event={"ID":"0061fcda-676e-40ef-a3af-7c90f1ea2a65","Type":"ContainerDied","Data":"02ea97f297c4c7347f0336e42dba55f94fbc1baa284f8546f3c269f9b9217210"}
Jun 02 16:03:45.282253 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:45.282195    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6" event={"ID":"0061fcda-676e-40ef-a3af-7c90f1ea2a65","Type":"ContainerDied","Data":"1b0778d41ae14d9337597f7fa27bc45dbb7471f8fb95acbba08a08dfbaacd949"}
Jun 02 16:03:45.282253 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:45.282216    2578 scope.go:117] "RemoveContainer" containerID="c624a9b3b3d8c1137a2fef63ae982483222e8c7f2d11c3c95e99058fc913d6bb"
Jun 02 16:03:45.290410 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:45.290393    2578 scope.go:117] "RemoveContainer" containerID="02ea97f297c4c7347f0336e42dba55f94fbc1baa284f8546f3c269f9b9217210"
Jun 02 16:03:45.297361 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:45.297345    2578 scope.go:117] "RemoveContainer" containerID="bb79270af56a3eb8acbf8735ed18b64f54c00449ccacd0170a8f5ff1b1102314"
Jun 02 16:03:45.303857 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:45.303840    2578 scope.go:117] "RemoveContainer" containerID="c624a9b3b3d8c1137a2fef63ae982483222e8c7f2d11c3c95e99058fc913d6bb"
Jun 02 16:03:45.304097 ip-10-0-140-246 kubenswrapper[2578]: E0602 16:03:45.304080    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c624a9b3b3d8c1137a2fef63ae982483222e8c7f2d11c3c95e99058fc913d6bb\": container with ID starting with c624a9b3b3d8c1137a2fef63ae982483222e8c7f2d11c3c95e99058fc913d6bb not found: ID does not exist" containerID="c624a9b3b3d8c1137a2fef63ae982483222e8c7f2d11c3c95e99058fc913d6bb"
Jun 02 16:03:45.304144 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:45.304105    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c624a9b3b3d8c1137a2fef63ae982483222e8c7f2d11c3c95e99058fc913d6bb"} err="failed to get container status \"c624a9b3b3d8c1137a2fef63ae982483222e8c7f2d11c3c95e99058fc913d6bb\": rpc error: code = NotFound desc = could not find container \"c624a9b3b3d8c1137a2fef63ae982483222e8c7f2d11c3c95e99058fc913d6bb\": container with ID starting with c624a9b3b3d8c1137a2fef63ae982483222e8c7f2d11c3c95e99058fc913d6bb not found: ID does not exist"
Jun 02 16:03:45.304144 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:45.304122    2578 scope.go:117] "RemoveContainer" containerID="02ea97f297c4c7347f0336e42dba55f94fbc1baa284f8546f3c269f9b9217210"
Jun 02 16:03:45.304356 ip-10-0-140-246 kubenswrapper[2578]: E0602 16:03:45.304338    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02ea97f297c4c7347f0336e42dba55f94fbc1baa284f8546f3c269f9b9217210\": container with ID starting with 02ea97f297c4c7347f0336e42dba55f94fbc1baa284f8546f3c269f9b9217210 not found: ID does not exist" containerID="02ea97f297c4c7347f0336e42dba55f94fbc1baa284f8546f3c269f9b9217210"
Jun 02 16:03:45.304405 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:45.304361    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02ea97f297c4c7347f0336e42dba55f94fbc1baa284f8546f3c269f9b9217210"} err="failed to get container status \"02ea97f297c4c7347f0336e42dba55f94fbc1baa284f8546f3c269f9b9217210\": rpc error: code = NotFound desc = could not find container \"02ea97f297c4c7347f0336e42dba55f94fbc1baa284f8546f3c269f9b9217210\": container with ID starting with 02ea97f297c4c7347f0336e42dba55f94fbc1baa284f8546f3c269f9b9217210 not found: ID does not exist"
Jun 02 16:03:45.304405 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:45.304377    2578 scope.go:117] "RemoveContainer" containerID="bb79270af56a3eb8acbf8735ed18b64f54c00449ccacd0170a8f5ff1b1102314"
Jun 02 16:03:45.304580 ip-10-0-140-246 kubenswrapper[2578]: E0602 16:03:45.304563    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb79270af56a3eb8acbf8735ed18b64f54c00449ccacd0170a8f5ff1b1102314\": container with ID starting with bb79270af56a3eb8acbf8735ed18b64f54c00449ccacd0170a8f5ff1b1102314 not found: ID does not exist" containerID="bb79270af56a3eb8acbf8735ed18b64f54c00449ccacd0170a8f5ff1b1102314"
Jun 02 16:03:45.304621 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:45.304583    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb79270af56a3eb8acbf8735ed18b64f54c00449ccacd0170a8f5ff1b1102314"} err="failed to get container status \"bb79270af56a3eb8acbf8735ed18b64f54c00449ccacd0170a8f5ff1b1102314\": rpc error: code = NotFound desc = could not find container \"bb79270af56a3eb8acbf8735ed18b64f54c00449ccacd0170a8f5ff1b1102314\": container with ID starting with bb79270af56a3eb8acbf8735ed18b64f54c00449ccacd0170a8f5ff1b1102314 not found: ID does not exist"
Jun 02 16:03:45.314743 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:45.314721    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6"]
Jun 02 16:03:45.318183 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:45.318157    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-261e5-predictor-666fd5f598-b9gc6"]
Jun 02 16:03:45.909223 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:45.909195    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0061fcda-676e-40ef-a3af-7c90f1ea2a65" path="/var/lib/kubelet/pods/0061fcda-676e-40ef-a3af-7c90f1ea2a65/volumes"
Jun 02 16:03:48.279863 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:48.279829    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44"
Jun 02 16:03:48.280478 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:48.280452    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44" podUID="d7b6b9d3-a7bf-4793-8879-4dc2e14973a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jun 02 16:03:58.280471 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:03:58.280433    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44" podUID="d7b6b9d3-a7bf-4793-8879-4dc2e14973a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jun 02 16:04:08.280890 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:04:08.280852    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44" podUID="d7b6b9d3-a7bf-4793-8879-4dc2e14973a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jun 02 16:04:18.281169 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:04:18.281127    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44" podUID="d7b6b9d3-a7bf-4793-8879-4dc2e14973a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jun 02 16:04:28.281246 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:04:28.281199    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44" podUID="d7b6b9d3-a7bf-4793-8879-4dc2e14973a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jun 02 16:04:38.281199 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:04:38.281161    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44" podUID="d7b6b9d3-a7bf-4793-8879-4dc2e14973a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jun 02 16:04:48.281144 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:04:48.281115    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44"
Jun 02 16:04:55.807836 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:04:55.807791    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-cffc11-predictor-85fd598f88-gbsnp"]
Jun 02 16:04:55.808370 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:04:55.808190    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0061fcda-676e-40ef-a3af-7c90f1ea2a65" containerName="kube-rbac-proxy"
Jun 02 16:04:55.808370 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:04:55.808203    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0061fcda-676e-40ef-a3af-7c90f1ea2a65" containerName="kube-rbac-proxy"
Jun 02 16:04:55.808370 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:04:55.808217    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0061fcda-676e-40ef-a3af-7c90f1ea2a65" containerName="storage-initializer"
Jun 02 16:04:55.808370 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:04:55.808223    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0061fcda-676e-40ef-a3af-7c90f1ea2a65" containerName="storage-initializer"
Jun 02 16:04:55.808370 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:04:55.808234    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0061fcda-676e-40ef-a3af-7c90f1ea2a65" containerName="kserve-container"
Jun 02 16:04:55.808370 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:04:55.808240    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0061fcda-676e-40ef-a3af-7c90f1ea2a65" containerName="kserve-container"
Jun 02 16:04:55.808370 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:04:55.808285    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="0061fcda-676e-40ef-a3af-7c90f1ea2a65" containerName="kserve-container"
Jun 02 16:04:55.808370 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:04:55.808295    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="0061fcda-676e-40ef-a3af-7c90f1ea2a65" containerName="kube-rbac-proxy"
Jun 02 16:04:55.811380 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:04:55.811358    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-cffc11-predictor-85fd598f88-gbsnp"
Jun 02 16:04:55.814985 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:04:55.814956    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-cffc11\""
Jun 02 16:04:55.815155 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:04:55.815014    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-cffc11-kube-rbac-proxy-sar-config\""
Jun 02 16:04:55.815155 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:04:55.815054    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jun 02 16:04:55.815155 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:04:55.815055    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-cffc11-predictor-serving-cert\""
Jun 02 16:04:55.815155 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:04:55.815150    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-cffc11-dockercfg-hcznv\""
Jun 02 16:04:55.822699 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:04:55.822668    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-cffc11-predictor-85fd598f88-gbsnp"]
Jun 02 16:04:55.906550 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:04:55.906515    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-secondary-cffc11-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/254dbfa2-2250-4019-a29c-78089b23f519-isvc-secondary-cffc11-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-cffc11-predictor-85fd598f88-gbsnp\" (UID: \"254dbfa2-2250-4019-a29c-78089b23f519\") " pod="kserve-ci-e2e-test/isvc-secondary-cffc11-predictor-85fd598f88-gbsnp"
Jun 02 16:04:55.906761 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:04:55.906566    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/254dbfa2-2250-4019-a29c-78089b23f519-cabundle-cert\") pod \"isvc-secondary-cffc11-predictor-85fd598f88-gbsnp\" (UID: \"254dbfa2-2250-4019-a29c-78089b23f519\") " pod="kserve-ci-e2e-test/isvc-secondary-cffc11-predictor-85fd598f88-gbsnp"
Jun 02 16:04:55.906761 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:04:55.906597    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/254dbfa2-2250-4019-a29c-78089b23f519-kserve-provision-location\") pod \"isvc-secondary-cffc11-predictor-85fd598f88-gbsnp\" (UID: \"254dbfa2-2250-4019-a29c-78089b23f519\") " pod="kserve-ci-e2e-test/isvc-secondary-cffc11-predictor-85fd598f88-gbsnp"
Jun 02 16:04:55.906885 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:04:55.906745    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/254dbfa2-2250-4019-a29c-78089b23f519-proxy-tls\") pod \"isvc-secondary-cffc11-predictor-85fd598f88-gbsnp\" (UID: \"254dbfa2-2250-4019-a29c-78089b23f519\") " pod="kserve-ci-e2e-test/isvc-secondary-cffc11-predictor-85fd598f88-gbsnp"
Jun 02 16:04:55.906885 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:04:55.906798    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmlwv\" (UniqueName: \"kubernetes.io/projected/254dbfa2-2250-4019-a29c-78089b23f519-kube-api-access-qmlwv\") pod \"isvc-secondary-cffc11-predictor-85fd598f88-gbsnp\" (UID: \"254dbfa2-2250-4019-a29c-78089b23f519\") " pod="kserve-ci-e2e-test/isvc-secondary-cffc11-predictor-85fd598f88-gbsnp"
Jun 02 16:04:56.008247 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:04:56.008201    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/254dbfa2-2250-4019-a29c-78089b23f519-proxy-tls\") pod \"isvc-secondary-cffc11-predictor-85fd598f88-gbsnp\" (UID: \"254dbfa2-2250-4019-a29c-78089b23f519\") " pod="kserve-ci-e2e-test/isvc-secondary-cffc11-predictor-85fd598f88-gbsnp"
Jun 02 16:04:56.008247 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:04:56.008244    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qmlwv\" (UniqueName: \"kubernetes.io/projected/254dbfa2-2250-4019-a29c-78089b23f519-kube-api-access-qmlwv\") pod \"isvc-secondary-cffc11-predictor-85fd598f88-gbsnp\" (UID: \"254dbfa2-2250-4019-a29c-78089b23f519\") " pod="kserve-ci-e2e-test/isvc-secondary-cffc11-predictor-85fd598f88-gbsnp"
Jun 02 16:04:56.008492 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:04:56.008305    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-secondary-cffc11-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/254dbfa2-2250-4019-a29c-78089b23f519-isvc-secondary-cffc11-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-cffc11-predictor-85fd598f88-gbsnp\" (UID: \"254dbfa2-2250-4019-a29c-78089b23f519\") " pod="kserve-ci-e2e-test/isvc-secondary-cffc11-predictor-85fd598f88-gbsnp"
Jun 02 16:04:56.008492 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:04:56.008325    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/254dbfa2-2250-4019-a29c-78089b23f519-cabundle-cert\") pod \"isvc-secondary-cffc11-predictor-85fd598f88-gbsnp\" (UID: \"254dbfa2-2250-4019-a29c-78089b23f519\") " pod="kserve-ci-e2e-test/isvc-secondary-cffc11-predictor-85fd598f88-gbsnp"
Jun 02 16:04:56.008492 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:04:56.008341    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/254dbfa2-2250-4019-a29c-78089b23f519-kserve-provision-location\") pod \"isvc-secondary-cffc11-predictor-85fd598f88-gbsnp\" (UID: \"254dbfa2-2250-4019-a29c-78089b23f519\") " pod="kserve-ci-e2e-test/isvc-secondary-cffc11-predictor-85fd598f88-gbsnp"
Jun 02 16:04:56.008803 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:04:56.008774    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/254dbfa2-2250-4019-a29c-78089b23f519-kserve-provision-location\") pod \"isvc-secondary-cffc11-predictor-85fd598f88-gbsnp\" (UID: \"254dbfa2-2250-4019-a29c-78089b23f519\") " pod="kserve-ci-e2e-test/isvc-secondary-cffc11-predictor-85fd598f88-gbsnp"
Jun 02 16:04:56.009104 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:04:56.009084    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/254dbfa2-2250-4019-a29c-78089b23f519-cabundle-cert\") pod \"isvc-secondary-cffc11-predictor-85fd598f88-gbsnp\" (UID: \"254dbfa2-2250-4019-a29c-78089b23f519\") " pod="kserve-ci-e2e-test/isvc-secondary-cffc11-predictor-85fd598f88-gbsnp"
Jun 02 16:04:56.009187 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:04:56.009129    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-secondary-cffc11-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/254dbfa2-2250-4019-a29c-78089b23f519-isvc-secondary-cffc11-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-cffc11-predictor-85fd598f88-gbsnp\" (UID: \"254dbfa2-2250-4019-a29c-78089b23f519\") " pod="kserve-ci-e2e-test/isvc-secondary-cffc11-predictor-85fd598f88-gbsnp"
Jun 02 16:04:56.011181 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:04:56.011157    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/254dbfa2-2250-4019-a29c-78089b23f519-proxy-tls\") pod \"isvc-secondary-cffc11-predictor-85fd598f88-gbsnp\" (UID: \"254dbfa2-2250-4019-a29c-78089b23f519\") " pod="kserve-ci-e2e-test/isvc-secondary-cffc11-predictor-85fd598f88-gbsnp"
Jun 02 16:04:56.018707 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:04:56.018668    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmlwv\" (UniqueName: \"kubernetes.io/projected/254dbfa2-2250-4019-a29c-78089b23f519-kube-api-access-qmlwv\") pod \"isvc-secondary-cffc11-predictor-85fd598f88-gbsnp\" (UID: \"254dbfa2-2250-4019-a29c-78089b23f519\") " pod="kserve-ci-e2e-test/isvc-secondary-cffc11-predictor-85fd598f88-gbsnp"
Jun 02 16:04:56.123482 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:04:56.123375    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-cffc11-predictor-85fd598f88-gbsnp"
Jun 02 16:04:56.267417 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:04:56.267387    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-cffc11-predictor-85fd598f88-gbsnp"]
Jun 02 16:04:56.270135 ip-10-0-140-246 kubenswrapper[2578]: W0602 16:04:56.270098    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod254dbfa2_2250_4019_a29c_78089b23f519.slice/crio-6453754d6a9d967d7ade869c209cdffed1b22c3d9120ebd170d55fd6457325b3 WatchSource:0}: Error finding container 6453754d6a9d967d7ade869c209cdffed1b22c3d9120ebd170d55fd6457325b3: Status 404 returned error can't find the container with id 6453754d6a9d967d7ade869c209cdffed1b22c3d9120ebd170d55fd6457325b3
Jun 02 16:04:56.272194 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:04:56.272173    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 16:04:56.494583 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:04:56.494545    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-cffc11-predictor-85fd598f88-gbsnp" event={"ID":"254dbfa2-2250-4019-a29c-78089b23f519","Type":"ContainerStarted","Data":"625ce37f3953b4ad829973813dd6076451f4b52114d2d544939605225e51555f"}
Jun 02 16:04:56.494583 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:04:56.494584    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-cffc11-predictor-85fd598f88-gbsnp" event={"ID":"254dbfa2-2250-4019-a29c-78089b23f519","Type":"ContainerStarted","Data":"6453754d6a9d967d7ade869c209cdffed1b22c3d9120ebd170d55fd6457325b3"}
Jun 02 16:05:02.513092 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:02.513055    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cffc11-predictor-85fd598f88-gbsnp_254dbfa2-2250-4019-a29c-78089b23f519/storage-initializer/0.log"
Jun 02 16:05:02.513521 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:02.513103    2578 generic.go:358] "Generic (PLEG): container finished" podID="254dbfa2-2250-4019-a29c-78089b23f519" containerID="625ce37f3953b4ad829973813dd6076451f4b52114d2d544939605225e51555f" exitCode=1
Jun 02 16:05:02.513521 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:02.513195    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-cffc11-predictor-85fd598f88-gbsnp" event={"ID":"254dbfa2-2250-4019-a29c-78089b23f519","Type":"ContainerDied","Data":"625ce37f3953b4ad829973813dd6076451f4b52114d2d544939605225e51555f"}
Jun 02 16:05:03.518129 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:03.518095    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cffc11-predictor-85fd598f88-gbsnp_254dbfa2-2250-4019-a29c-78089b23f519/storage-initializer/0.log"
Jun 02 16:05:03.518646 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:03.518213    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-cffc11-predictor-85fd598f88-gbsnp" event={"ID":"254dbfa2-2250-4019-a29c-78089b23f519","Type":"ContainerStarted","Data":"59949998db9ee7b6283bee61f35b54f5fa4f222a014c593e2c55adb630c83886"}
Jun 02 16:05:05.525688 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:05.525604    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cffc11-predictor-85fd598f88-gbsnp_254dbfa2-2250-4019-a29c-78089b23f519/storage-initializer/1.log"
Jun 02 16:05:05.526127 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:05.525949    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cffc11-predictor-85fd598f88-gbsnp_254dbfa2-2250-4019-a29c-78089b23f519/storage-initializer/0.log"
Jun 02 16:05:05.526127 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:05.525987    2578 generic.go:358] "Generic (PLEG): container finished" podID="254dbfa2-2250-4019-a29c-78089b23f519" containerID="59949998db9ee7b6283bee61f35b54f5fa4f222a014c593e2c55adb630c83886" exitCode=1
Jun 02 16:05:05.526127 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:05.526034    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-cffc11-predictor-85fd598f88-gbsnp" event={"ID":"254dbfa2-2250-4019-a29c-78089b23f519","Type":"ContainerDied","Data":"59949998db9ee7b6283bee61f35b54f5fa4f222a014c593e2c55adb630c83886"}
Jun 02 16:05:05.526127 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:05.526078    2578 scope.go:117] "RemoveContainer" containerID="625ce37f3953b4ad829973813dd6076451f4b52114d2d544939605225e51555f"
Jun 02 16:05:05.526499 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:05.526462    2578 scope.go:117] "RemoveContainer" containerID="625ce37f3953b4ad829973813dd6076451f4b52114d2d544939605225e51555f"
Jun 02 16:05:05.537247 ip-10-0-140-246 kubenswrapper[2578]: E0602 16:05:05.537208    2578 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-cffc11-predictor-85fd598f88-gbsnp_kserve-ci-e2e-test_254dbfa2-2250-4019-a29c-78089b23f519_0 in pod sandbox 6453754d6a9d967d7ade869c209cdffed1b22c3d9120ebd170d55fd6457325b3 from index: no such id: '625ce37f3953b4ad829973813dd6076451f4b52114d2d544939605225e51555f'" containerID="625ce37f3953b4ad829973813dd6076451f4b52114d2d544939605225e51555f"
Jun 02 16:05:05.537337 ip-10-0-140-246 kubenswrapper[2578]: E0602 16:05:05.537281    2578 kuberuntime_container.go:951] "Unhandled Error" err="failed to remove pod init container \"storage-initializer\": rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-cffc11-predictor-85fd598f88-gbsnp_kserve-ci-e2e-test_254dbfa2-2250-4019-a29c-78089b23f519_0 in pod sandbox 6453754d6a9d967d7ade869c209cdffed1b22c3d9120ebd170d55fd6457325b3 from index: no such id: '625ce37f3953b4ad829973813dd6076451f4b52114d2d544939605225e51555f'; Skipping pod \"isvc-secondary-cffc11-predictor-85fd598f88-gbsnp_kserve-ci-e2e-test(254dbfa2-2250-4019-a29c-78089b23f519)\"" logger="UnhandledError"
Jun 02 16:05:05.538687 ip-10-0-140-246 kubenswrapper[2578]: E0602 16:05:05.538657    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-cffc11-predictor-85fd598f88-gbsnp_kserve-ci-e2e-test(254dbfa2-2250-4019-a29c-78089b23f519)\"" pod="kserve-ci-e2e-test/isvc-secondary-cffc11-predictor-85fd598f88-gbsnp" podUID="254dbfa2-2250-4019-a29c-78089b23f519"
Jun 02 16:05:06.531178 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:06.531142    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cffc11-predictor-85fd598f88-gbsnp_254dbfa2-2250-4019-a29c-78089b23f519/storage-initializer/1.log"
Jun 02 16:05:13.875631 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:13.875523    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44"]
Jun 02 16:05:13.876112 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:13.876009    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44" podUID="d7b6b9d3-a7bf-4793-8879-4dc2e14973a3" containerName="kserve-container" containerID="cri-o://91f3d848bc66e015dc2db927ff8718634db43c195b20fab5a27c96ee8757e122" gracePeriod=30
Jun 02 16:05:13.876112 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:13.876056    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44" podUID="d7b6b9d3-a7bf-4793-8879-4dc2e14973a3" containerName="kube-rbac-proxy" containerID="cri-o://c8c649d4ad651f4694fb54cc3a32ba8835c7016d21a7bc4a8100ab80d3d78a88" gracePeriod=30
Jun 02 16:05:13.979488 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:13.979443    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-cffc11-predictor-85fd598f88-gbsnp"]
Jun 02 16:05:14.073730 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.073692    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv"]
Jun 02 16:05:14.078016 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.077972    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv"
Jun 02 16:05:14.080294 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.080261    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-0b4fc4\""
Jun 02 16:05:14.080475 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.080334    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-0b4fc4-predictor-serving-cert\""
Jun 02 16:05:14.080475 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.080274    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-0b4fc4-kube-rbac-proxy-sar-config\""
Jun 02 16:05:14.080629 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.080614    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-0b4fc4-dockercfg-vvpbp\""
Jun 02 16:05:14.087549 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.087517    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv"]
Jun 02 16:05:14.126910 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.126827    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cffc11-predictor-85fd598f88-gbsnp_254dbfa2-2250-4019-a29c-78089b23f519/storage-initializer/1.log"
Jun 02 16:05:14.126910 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.126899    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-cffc11-predictor-85fd598f88-gbsnp"
Jun 02 16:05:14.177389 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.177347    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-init-fail-0b4fc4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/247c8ace-4646-4e2b-b9fb-f5afb563080c-isvc-init-fail-0b4fc4-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv\" (UID: \"247c8ace-4646-4e2b-b9fb-f5afb563080c\") " pod="kserve-ci-e2e-test/isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv"
Jun 02 16:05:14.177582 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.177424    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/247c8ace-4646-4e2b-b9fb-f5afb563080c-cabundle-cert\") pod \"isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv\" (UID: \"247c8ace-4646-4e2b-b9fb-f5afb563080c\") " pod="kserve-ci-e2e-test/isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv"
Jun 02 16:05:14.177582 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.177450    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/247c8ace-4646-4e2b-b9fb-f5afb563080c-kserve-provision-location\") pod \"isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv\" (UID: \"247c8ace-4646-4e2b-b9fb-f5afb563080c\") " pod="kserve-ci-e2e-test/isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv"
Jun 02 16:05:14.177582 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.177481    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z4bt\" (UniqueName: \"kubernetes.io/projected/247c8ace-4646-4e2b-b9fb-f5afb563080c-kube-api-access-5z4bt\") pod \"isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv\" (UID: \"247c8ace-4646-4e2b-b9fb-f5afb563080c\") " pod="kserve-ci-e2e-test/isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv"
Jun 02 16:05:14.177779 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.177596    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/247c8ace-4646-4e2b-b9fb-f5afb563080c-proxy-tls\") pod \"isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv\" (UID: \"247c8ace-4646-4e2b-b9fb-f5afb563080c\") " pod="kserve-ci-e2e-test/isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv"
Jun 02 16:05:14.278637 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.278598    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/254dbfa2-2250-4019-a29c-78089b23f519-kserve-provision-location\") pod \"254dbfa2-2250-4019-a29c-78089b23f519\" (UID: \"254dbfa2-2250-4019-a29c-78089b23f519\") "
Jun 02 16:05:14.278831 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.278709    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-secondary-cffc11-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/254dbfa2-2250-4019-a29c-78089b23f519-isvc-secondary-cffc11-kube-rbac-proxy-sar-config\") pod \"254dbfa2-2250-4019-a29c-78089b23f519\" (UID: \"254dbfa2-2250-4019-a29c-78089b23f519\") "
Jun 02 16:05:14.278831 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.278739    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/254dbfa2-2250-4019-a29c-78089b23f519-cabundle-cert\") pod \"254dbfa2-2250-4019-a29c-78089b23f519\" (UID: \"254dbfa2-2250-4019-a29c-78089b23f519\") "
Jun 02 16:05:14.278831 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.278808    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmlwv\" (UniqueName: \"kubernetes.io/projected/254dbfa2-2250-4019-a29c-78089b23f519-kube-api-access-qmlwv\") pod \"254dbfa2-2250-4019-a29c-78089b23f519\" (UID: \"254dbfa2-2250-4019-a29c-78089b23f519\") "
Jun 02 16:05:14.279029 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.278844    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/254dbfa2-2250-4019-a29c-78089b23f519-proxy-tls\") pod \"254dbfa2-2250-4019-a29c-78089b23f519\" (UID: \"254dbfa2-2250-4019-a29c-78089b23f519\") "
Jun 02 16:05:14.279029 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.278940    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/254dbfa2-2250-4019-a29c-78089b23f519-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "254dbfa2-2250-4019-a29c-78089b23f519" (UID: "254dbfa2-2250-4019-a29c-78089b23f519"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:05:14.279029 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.278953    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-init-fail-0b4fc4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/247c8ace-4646-4e2b-b9fb-f5afb563080c-isvc-init-fail-0b4fc4-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv\" (UID: \"247c8ace-4646-4e2b-b9fb-f5afb563080c\") " pod="kserve-ci-e2e-test/isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv"
Jun 02 16:05:14.279188 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.279080    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/247c8ace-4646-4e2b-b9fb-f5afb563080c-cabundle-cert\") pod \"isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv\" (UID: \"247c8ace-4646-4e2b-b9fb-f5afb563080c\") " pod="kserve-ci-e2e-test/isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv"
Jun 02 16:05:14.279188 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.279114    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/247c8ace-4646-4e2b-b9fb-f5afb563080c-kserve-provision-location\") pod \"isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv\" (UID: \"247c8ace-4646-4e2b-b9fb-f5afb563080c\") " pod="kserve-ci-e2e-test/isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv"
Jun 02 16:05:14.279188 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.279156    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5z4bt\" (UniqueName: \"kubernetes.io/projected/247c8ace-4646-4e2b-b9fb-f5afb563080c-kube-api-access-5z4bt\") pod \"isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv\" (UID: \"247c8ace-4646-4e2b-b9fb-f5afb563080c\") " pod="kserve-ci-e2e-test/isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv"
Jun 02 16:05:14.279343 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.279209    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/254dbfa2-2250-4019-a29c-78089b23f519-isvc-secondary-cffc11-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-secondary-cffc11-kube-rbac-proxy-sar-config") pod "254dbfa2-2250-4019-a29c-78089b23f519" (UID: "254dbfa2-2250-4019-a29c-78089b23f519"). InnerVolumeSpecName "isvc-secondary-cffc11-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:05:14.279343 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.279236    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/247c8ace-4646-4e2b-b9fb-f5afb563080c-proxy-tls\") pod \"isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv\" (UID: \"247c8ace-4646-4e2b-b9fb-f5afb563080c\") " pod="kserve-ci-e2e-test/isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv"
Jun 02 16:05:14.279343 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.279312    2578 reconciler_common.go:299] "Volume detached for volume \"isvc-secondary-cffc11-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/254dbfa2-2250-4019-a29c-78089b23f519-isvc-secondary-cffc11-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 16:05:14.279343 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.279333    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/254dbfa2-2250-4019-a29c-78089b23f519-kserve-provision-location\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 16:05:14.279585 ip-10-0-140-246 kubenswrapper[2578]: E0602 16:05:14.279419    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-0b4fc4-predictor-serving-cert: secret "isvc-init-fail-0b4fc4-predictor-serving-cert" not found
Jun 02 16:05:14.279585 ip-10-0-140-246 kubenswrapper[2578]: E0602 16:05:14.279477    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/247c8ace-4646-4e2b-b9fb-f5afb563080c-proxy-tls podName:247c8ace-4646-4e2b-b9fb-f5afb563080c nodeName:}" failed. No retries permitted until 2026-06-02 16:05:14.779458373 +0000 UTC m=+1163.481921874 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/247c8ace-4646-4e2b-b9fb-f5afb563080c-proxy-tls") pod "isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv" (UID: "247c8ace-4646-4e2b-b9fb-f5afb563080c") : secret "isvc-init-fail-0b4fc4-predictor-serving-cert" not found
Jun 02 16:05:14.279695 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.279579    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/247c8ace-4646-4e2b-b9fb-f5afb563080c-kserve-provision-location\") pod \"isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv\" (UID: \"247c8ace-4646-4e2b-b9fb-f5afb563080c\") " pod="kserve-ci-e2e-test/isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv"
Jun 02 16:05:14.279819 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.279790    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-init-fail-0b4fc4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/247c8ace-4646-4e2b-b9fb-f5afb563080c-isvc-init-fail-0b4fc4-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv\" (UID: \"247c8ace-4646-4e2b-b9fb-f5afb563080c\") " pod="kserve-ci-e2e-test/isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv"
Jun 02 16:05:14.279877 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.279808    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/247c8ace-4646-4e2b-b9fb-f5afb563080c-cabundle-cert\") pod \"isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv\" (UID: \"247c8ace-4646-4e2b-b9fb-f5afb563080c\") " pod="kserve-ci-e2e-test/isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv"
Jun 02 16:05:14.279877 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.279868    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/254dbfa2-2250-4019-a29c-78089b23f519-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "254dbfa2-2250-4019-a29c-78089b23f519" (UID: "254dbfa2-2250-4019-a29c-78089b23f519"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:05:14.281331 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.281301    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/254dbfa2-2250-4019-a29c-78089b23f519-kube-api-access-qmlwv" (OuterVolumeSpecName: "kube-api-access-qmlwv") pod "254dbfa2-2250-4019-a29c-78089b23f519" (UID: "254dbfa2-2250-4019-a29c-78089b23f519"). InnerVolumeSpecName "kube-api-access-qmlwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:05:14.281648 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.281622    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/254dbfa2-2250-4019-a29c-78089b23f519-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "254dbfa2-2250-4019-a29c-78089b23f519" (UID: "254dbfa2-2250-4019-a29c-78089b23f519"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:05:14.288557 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.288521    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z4bt\" (UniqueName: \"kubernetes.io/projected/247c8ace-4646-4e2b-b9fb-f5afb563080c-kube-api-access-5z4bt\") pod \"isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv\" (UID: \"247c8ace-4646-4e2b-b9fb-f5afb563080c\") " pod="kserve-ci-e2e-test/isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv"
Jun 02 16:05:14.380479 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.380381    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/254dbfa2-2250-4019-a29c-78089b23f519-cabundle-cert\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 16:05:14.380479 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.380422    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmlwv\" (UniqueName: \"kubernetes.io/projected/254dbfa2-2250-4019-a29c-78089b23f519-kube-api-access-qmlwv\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 16:05:14.380479 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.380438    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/254dbfa2-2250-4019-a29c-78089b23f519-proxy-tls\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 16:05:14.561592 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.561555    2578 generic.go:358] "Generic (PLEG): container finished" podID="d7b6b9d3-a7bf-4793-8879-4dc2e14973a3" containerID="c8c649d4ad651f4694fb54cc3a32ba8835c7016d21a7bc4a8100ab80d3d78a88" exitCode=2
Jun 02 16:05:14.561790 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.561589    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44" event={"ID":"d7b6b9d3-a7bf-4793-8879-4dc2e14973a3","Type":"ContainerDied","Data":"c8c649d4ad651f4694fb54cc3a32ba8835c7016d21a7bc4a8100ab80d3d78a88"}
Jun 02 16:05:14.562905 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.562882    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cffc11-predictor-85fd598f88-gbsnp_254dbfa2-2250-4019-a29c-78089b23f519/storage-initializer/1.log"
Jun 02 16:05:14.563052 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.562951    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-cffc11-predictor-85fd598f88-gbsnp" event={"ID":"254dbfa2-2250-4019-a29c-78089b23f519","Type":"ContainerDied","Data":"6453754d6a9d967d7ade869c209cdffed1b22c3d9120ebd170d55fd6457325b3"}
Jun 02 16:05:14.563052 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.562979    2578 scope.go:117] "RemoveContainer" containerID="59949998db9ee7b6283bee61f35b54f5fa4f222a014c593e2c55adb630c83886"
Jun 02 16:05:14.563052 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.563030    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-cffc11-predictor-85fd598f88-gbsnp"
Jun 02 16:05:14.600592 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.600549    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-cffc11-predictor-85fd598f88-gbsnp"]
Jun 02 16:05:14.604773 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.604735    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-cffc11-predictor-85fd598f88-gbsnp"]
Jun 02 16:05:14.784265 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.784223    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/247c8ace-4646-4e2b-b9fb-f5afb563080c-proxy-tls\") pod \"isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv\" (UID: \"247c8ace-4646-4e2b-b9fb-f5afb563080c\") " pod="kserve-ci-e2e-test/isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv"
Jun 02 16:05:14.786912 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.786874    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/247c8ace-4646-4e2b-b9fb-f5afb563080c-proxy-tls\") pod \"isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv\" (UID: \"247c8ace-4646-4e2b-b9fb-f5afb563080c\") " pod="kserve-ci-e2e-test/isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv"
Jun 02 16:05:14.993599 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:14.993556    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv"
Jun 02 16:05:15.137629 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:15.137592    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv"]
Jun 02 16:05:15.141432 ip-10-0-140-246 kubenswrapper[2578]: W0602 16:05:15.141401    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod247c8ace_4646_4e2b_b9fb_f5afb563080c.slice/crio-cc35bfcc61e0f2542bcbe4a7bc4d42d574d5f09755b324b387e90866ebffedee WatchSource:0}: Error finding container cc35bfcc61e0f2542bcbe4a7bc4d42d574d5f09755b324b387e90866ebffedee: Status 404 returned error can't find the container with id cc35bfcc61e0f2542bcbe4a7bc4d42d574d5f09755b324b387e90866ebffedee
Jun 02 16:05:15.569145 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:15.569104    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv" event={"ID":"247c8ace-4646-4e2b-b9fb-f5afb563080c","Type":"ContainerStarted","Data":"4544e8bff987c4be4336af3b7730392db9ba403565e5a950e4262cbe57940b4f"}
Jun 02 16:05:15.569145 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:15.569146    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv" event={"ID":"247c8ace-4646-4e2b-b9fb-f5afb563080c","Type":"ContainerStarted","Data":"cc35bfcc61e0f2542bcbe4a7bc4d42d574d5f09755b324b387e90866ebffedee"}
Jun 02 16:05:15.910197 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:15.910152    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="254dbfa2-2250-4019-a29c-78089b23f519" path="/var/lib/kubelet/pods/254dbfa2-2250-4019-a29c-78089b23f519/volumes"
Jun 02 16:05:18.276168 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:18.276120    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44" podUID="d7b6b9d3-a7bf-4793-8879-4dc2e14973a3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jun 02 16:05:18.280617 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:18.280574    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44" podUID="d7b6b9d3-a7bf-4793-8879-4dc2e14973a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jun 02 16:05:19.139668 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:19.139633    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44"
Jun 02 16:05:19.144400 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:19.144370    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7b6b9d3-a7bf-4793-8879-4dc2e14973a3-kserve-provision-location\") pod \"d7b6b9d3-a7bf-4793-8879-4dc2e14973a3\" (UID: \"d7b6b9d3-a7bf-4793-8879-4dc2e14973a3\") "
Jun 02 16:05:19.144590 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:19.144432    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvw9b\" (UniqueName: \"kubernetes.io/projected/d7b6b9d3-a7bf-4793-8879-4dc2e14973a3-kube-api-access-kvw9b\") pod \"d7b6b9d3-a7bf-4793-8879-4dc2e14973a3\" (UID: \"d7b6b9d3-a7bf-4793-8879-4dc2e14973a3\") "
Jun 02 16:05:19.144590 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:19.144460    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7b6b9d3-a7bf-4793-8879-4dc2e14973a3-proxy-tls\") pod \"d7b6b9d3-a7bf-4793-8879-4dc2e14973a3\" (UID: \"d7b6b9d3-a7bf-4793-8879-4dc2e14973a3\") "
Jun 02 16:05:19.144590 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:19.144517    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-primary-cffc11-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7b6b9d3-a7bf-4793-8879-4dc2e14973a3-isvc-primary-cffc11-kube-rbac-proxy-sar-config\") pod \"d7b6b9d3-a7bf-4793-8879-4dc2e14973a3\" (UID: \"d7b6b9d3-a7bf-4793-8879-4dc2e14973a3\") "
Jun 02 16:05:19.144840 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:19.144760    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b6b9d3-a7bf-4793-8879-4dc2e14973a3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d7b6b9d3-a7bf-4793-8879-4dc2e14973a3" (UID: "d7b6b9d3-a7bf-4793-8879-4dc2e14973a3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:05:19.144948 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:19.144927    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7b6b9d3-a7bf-4793-8879-4dc2e14973a3-isvc-primary-cffc11-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-primary-cffc11-kube-rbac-proxy-sar-config") pod "d7b6b9d3-a7bf-4793-8879-4dc2e14973a3" (UID: "d7b6b9d3-a7bf-4793-8879-4dc2e14973a3"). InnerVolumeSpecName "isvc-primary-cffc11-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:05:19.146728 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:19.146698    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7b6b9d3-a7bf-4793-8879-4dc2e14973a3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d7b6b9d3-a7bf-4793-8879-4dc2e14973a3" (UID: "d7b6b9d3-a7bf-4793-8879-4dc2e14973a3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:05:19.146880 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:19.146780    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7b6b9d3-a7bf-4793-8879-4dc2e14973a3-kube-api-access-kvw9b" (OuterVolumeSpecName: "kube-api-access-kvw9b") pod "d7b6b9d3-a7bf-4793-8879-4dc2e14973a3" (UID: "d7b6b9d3-a7bf-4793-8879-4dc2e14973a3"). InnerVolumeSpecName "kube-api-access-kvw9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:05:19.245481 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:19.245379    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvw9b\" (UniqueName: \"kubernetes.io/projected/d7b6b9d3-a7bf-4793-8879-4dc2e14973a3-kube-api-access-kvw9b\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 16:05:19.245481 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:19.245414    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7b6b9d3-a7bf-4793-8879-4dc2e14973a3-proxy-tls\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 16:05:19.245481 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:19.245429    2578 reconciler_common.go:299] "Volume detached for volume \"isvc-primary-cffc11-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7b6b9d3-a7bf-4793-8879-4dc2e14973a3-isvc-primary-cffc11-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 16:05:19.245481 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:19.245443    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7b6b9d3-a7bf-4793-8879-4dc2e14973a3-kserve-provision-location\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 16:05:19.583477 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:19.583376    2578 generic.go:358] "Generic (PLEG): container finished" podID="d7b6b9d3-a7bf-4793-8879-4dc2e14973a3" containerID="91f3d848bc66e015dc2db927ff8718634db43c195b20fab5a27c96ee8757e122" exitCode=0
Jun 02 16:05:19.583477 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:19.583439    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44" event={"ID":"d7b6b9d3-a7bf-4793-8879-4dc2e14973a3","Type":"ContainerDied","Data":"91f3d848bc66e015dc2db927ff8718634db43c195b20fab5a27c96ee8757e122"}
Jun 02 16:05:19.583945 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:19.583484    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44" event={"ID":"d7b6b9d3-a7bf-4793-8879-4dc2e14973a3","Type":"ContainerDied","Data":"5b0f66b94ac03898742de89b8225be121ab860cbafab0fb6f38b2e776f291071"}
Jun 02 16:05:19.583945 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:19.583492    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44"
Jun 02 16:05:19.583945 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:19.583501    2578 scope.go:117] "RemoveContainer" containerID="c8c649d4ad651f4694fb54cc3a32ba8835c7016d21a7bc4a8100ab80d3d78a88"
Jun 02 16:05:19.592283 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:19.592261    2578 scope.go:117] "RemoveContainer" containerID="91f3d848bc66e015dc2db927ff8718634db43c195b20fab5a27c96ee8757e122"
Jun 02 16:05:19.601581 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:19.601555    2578 scope.go:117] "RemoveContainer" containerID="ec458f66f9af54c5dd9aaedf7d5f87173a6828ad947f1fdb522c91c5c39d3b69"
Jun 02 16:05:19.611624 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:19.611588    2578 scope.go:117] "RemoveContainer" containerID="c8c649d4ad651f4694fb54cc3a32ba8835c7016d21a7bc4a8100ab80d3d78a88"
Jun 02 16:05:19.611973 ip-10-0-140-246 kubenswrapper[2578]: E0602 16:05:19.611948    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8c649d4ad651f4694fb54cc3a32ba8835c7016d21a7bc4a8100ab80d3d78a88\": container with ID starting with c8c649d4ad651f4694fb54cc3a32ba8835c7016d21a7bc4a8100ab80d3d78a88 not found: ID does not exist" containerID="c8c649d4ad651f4694fb54cc3a32ba8835c7016d21a7bc4a8100ab80d3d78a88"
Jun 02 16:05:19.612165 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:19.611982    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8c649d4ad651f4694fb54cc3a32ba8835c7016d21a7bc4a8100ab80d3d78a88"} err="failed to get container status \"c8c649d4ad651f4694fb54cc3a32ba8835c7016d21a7bc4a8100ab80d3d78a88\": rpc error: code = NotFound desc = could not find container \"c8c649d4ad651f4694fb54cc3a32ba8835c7016d21a7bc4a8100ab80d3d78a88\": container with ID starting with c8c649d4ad651f4694fb54cc3a32ba8835c7016d21a7bc4a8100ab80d3d78a88 not found: ID does not exist"
Jun 02 16:05:19.612165 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:19.612028    2578 scope.go:117] "RemoveContainer" containerID="91f3d848bc66e015dc2db927ff8718634db43c195b20fab5a27c96ee8757e122"
Jun 02 16:05:19.612328 ip-10-0-140-246 kubenswrapper[2578]: E0602 16:05:19.612310    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91f3d848bc66e015dc2db927ff8718634db43c195b20fab5a27c96ee8757e122\": container with ID starting with 91f3d848bc66e015dc2db927ff8718634db43c195b20fab5a27c96ee8757e122 not found: ID does not exist" containerID="91f3d848bc66e015dc2db927ff8718634db43c195b20fab5a27c96ee8757e122"
Jun 02 16:05:19.612369 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:19.612338    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91f3d848bc66e015dc2db927ff8718634db43c195b20fab5a27c96ee8757e122"} err="failed to get container status \"91f3d848bc66e015dc2db927ff8718634db43c195b20fab5a27c96ee8757e122\": rpc error: code = NotFound desc = could not find container \"91f3d848bc66e015dc2db927ff8718634db43c195b20fab5a27c96ee8757e122\": container with ID starting with 91f3d848bc66e015dc2db927ff8718634db43c195b20fab5a27c96ee8757e122 not found: ID does not exist"
Jun 02 16:05:19.612369 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:19.612362    2578 scope.go:117] "RemoveContainer" containerID="ec458f66f9af54c5dd9aaedf7d5f87173a6828ad947f1fdb522c91c5c39d3b69"
Jun 02 16:05:19.612660 ip-10-0-140-246 kubenswrapper[2578]: E0602 16:05:19.612638    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec458f66f9af54c5dd9aaedf7d5f87173a6828ad947f1fdb522c91c5c39d3b69\": container with ID starting with ec458f66f9af54c5dd9aaedf7d5f87173a6828ad947f1fdb522c91c5c39d3b69 not found: ID does not exist" containerID="ec458f66f9af54c5dd9aaedf7d5f87173a6828ad947f1fdb522c91c5c39d3b69"
Jun 02 16:05:19.612726 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:19.612661    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec458f66f9af54c5dd9aaedf7d5f87173a6828ad947f1fdb522c91c5c39d3b69"} err="failed to get container status \"ec458f66f9af54c5dd9aaedf7d5f87173a6828ad947f1fdb522c91c5c39d3b69\": rpc error: code = NotFound desc = could not find container \"ec458f66f9af54c5dd9aaedf7d5f87173a6828ad947f1fdb522c91c5c39d3b69\": container with ID starting with ec458f66f9af54c5dd9aaedf7d5f87173a6828ad947f1fdb522c91c5c39d3b69 not found: ID does not exist"
Jun 02 16:05:19.613277 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:19.613254    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44"]
Jun 02 16:05:19.618837 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:19.618803    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-cffc11-predictor-6846fdc6ff-k4x44"]
Jun 02 16:05:19.907869 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:19.907840    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7b6b9d3-a7bf-4793-8879-4dc2e14973a3" path="/var/lib/kubelet/pods/d7b6b9d3-a7bf-4793-8879-4dc2e14973a3/volumes"
Jun 02 16:05:20.588073 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:20.588044    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv_247c8ace-4646-4e2b-b9fb-f5afb563080c/storage-initializer/0.log"
Jun 02 16:05:20.588532 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:20.588086    2578 generic.go:358] "Generic (PLEG): container finished" podID="247c8ace-4646-4e2b-b9fb-f5afb563080c" containerID="4544e8bff987c4be4336af3b7730392db9ba403565e5a950e4262cbe57940b4f" exitCode=1
Jun 02 16:05:20.588532 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:20.588167    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv" event={"ID":"247c8ace-4646-4e2b-b9fb-f5afb563080c","Type":"ContainerDied","Data":"4544e8bff987c4be4336af3b7730392db9ba403565e5a950e4262cbe57940b4f"}
Jun 02 16:05:21.594179 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:21.594147    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv_247c8ace-4646-4e2b-b9fb-f5afb563080c/storage-initializer/0.log"
Jun 02 16:05:21.594602 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:21.594213    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv" event={"ID":"247c8ace-4646-4e2b-b9fb-f5afb563080c","Type":"ContainerStarted","Data":"c25315a3e0a7b7ca8512c770f98c89e5327423135ae713aa1955286cad97b4bc"}
Jun 02 16:05:24.057440 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:24.057398    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv"]
Jun 02 16:05:24.057896 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:24.057714    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv" podUID="247c8ace-4646-4e2b-b9fb-f5afb563080c" containerName="storage-initializer" containerID="cri-o://c25315a3e0a7b7ca8512c770f98c89e5327423135ae713aa1955286cad97b4bc" gracePeriod=30
Jun 02 16:05:24.199119 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:24.199073    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p"]
Jun 02 16:05:24.199492 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:24.199478    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7b6b9d3-a7bf-4793-8879-4dc2e14973a3" containerName="storage-initializer"
Jun 02 16:05:24.199532 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:24.199494    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7b6b9d3-a7bf-4793-8879-4dc2e14973a3" containerName="storage-initializer"
Jun 02 16:05:24.199532 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:24.199506    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="254dbfa2-2250-4019-a29c-78089b23f519" containerName="storage-initializer"
Jun 02 16:05:24.199532 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:24.199512    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="254dbfa2-2250-4019-a29c-78089b23f519" containerName="storage-initializer"
Jun 02 16:05:24.199532 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:24.199523    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7b6b9d3-a7bf-4793-8879-4dc2e14973a3" containerName="kube-rbac-proxy"
Jun 02 16:05:24.199532 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:24.199530    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7b6b9d3-a7bf-4793-8879-4dc2e14973a3" containerName="kube-rbac-proxy"
Jun 02 16:05:24.199710 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:24.199538    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7b6b9d3-a7bf-4793-8879-4dc2e14973a3" containerName="kserve-container"
Jun 02 16:05:24.199710 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:24.199543    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7b6b9d3-a7bf-4793-8879-4dc2e14973a3" containerName="kserve-container"
Jun 02 16:05:24.199710 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:24.199595    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="254dbfa2-2250-4019-a29c-78089b23f519" containerName="storage-initializer"
Jun 02 16:05:24.199710 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:24.199605    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="d7b6b9d3-a7bf-4793-8879-4dc2e14973a3" containerName="kserve-container"
Jun 02 16:05:24.199710 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:24.199614    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="d7b6b9d3-a7bf-4793-8879-4dc2e14973a3" containerName="kube-rbac-proxy"
Jun 02 16:05:24.199710 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:24.199671    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="254dbfa2-2250-4019-a29c-78089b23f519" containerName="storage-initializer"
Jun 02 16:05:24.199710 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:24.199677    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="254dbfa2-2250-4019-a29c-78089b23f519" containerName="storage-initializer"
Jun 02 16:05:24.199909 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:24.199747    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="254dbfa2-2250-4019-a29c-78089b23f519" containerName="storage-initializer"
Jun 02 16:05:24.202199 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:24.202170    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p"
Jun 02 16:05:24.204286 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:24.204252    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-l65kw\""
Jun 02 16:05:24.204286 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:24.204280    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-a5de4-predictor-serving-cert\""
Jun 02 16:05:24.204508 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:24.204365    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-a5de4-kube-rbac-proxy-sar-config\""
Jun 02 16:05:24.213674 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:24.213644    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p"]
Jun 02 16:05:24.285967 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:24.285917    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"raw-sklearn-a5de4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65d5bbcf-f448-4c14-94d5-b2f83613cacb-raw-sklearn-a5de4-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p\" (UID: \"65d5bbcf-f448-4c14-94d5-b2f83613cacb\") " pod="kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p"
Jun 02 16:05:24.286154 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:24.286037    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65d5bbcf-f448-4c14-94d5-b2f83613cacb-proxy-tls\") pod \"raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p\" (UID: \"65d5bbcf-f448-4c14-94d5-b2f83613cacb\") " pod="kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p"
Jun 02 16:05:24.286154 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:24.286084    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wl6z\" (UniqueName: \"kubernetes.io/projected/65d5bbcf-f448-4c14-94d5-b2f83613cacb-kube-api-access-6wl6z\") pod \"raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p\" (UID: \"65d5bbcf-f448-4c14-94d5-b2f83613cacb\") " pod="kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p"
Jun 02 16:05:24.286154 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:24.286111    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65d5bbcf-f448-4c14-94d5-b2f83613cacb-kserve-provision-location\") pod \"raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p\" (UID: \"65d5bbcf-f448-4c14-94d5-b2f83613cacb\") " pod="kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p"
Jun 02 16:05:24.386606 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:24.386481    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65d5bbcf-f448-4c14-94d5-b2f83613cacb-proxy-tls\") pod \"raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p\" (UID: \"65d5bbcf-f448-4c14-94d5-b2f83613cacb\") " pod="kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p"
Jun 02 16:05:24.386606 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:24.386580    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6wl6z\" (UniqueName: \"kubernetes.io/projected/65d5bbcf-f448-4c14-94d5-b2f83613cacb-kube-api-access-6wl6z\") pod \"raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p\" (UID: \"65d5bbcf-f448-4c14-94d5-b2f83613cacb\") " pod="kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p"
Jun 02 16:05:24.386878 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:24.386620    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65d5bbcf-f448-4c14-94d5-b2f83613cacb-kserve-provision-location\") pod \"raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p\" (UID: \"65d5bbcf-f448-4c14-94d5-b2f83613cacb\") " pod="kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p"
Jun 02 16:05:24.386878 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:24.386661    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"raw-sklearn-a5de4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65d5bbcf-f448-4c14-94d5-b2f83613cacb-raw-sklearn-a5de4-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p\" (UID: \"65d5bbcf-f448-4c14-94d5-b2f83613cacb\") " pod="kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p"
Jun 02 16:05:24.387161 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:24.387133    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65d5bbcf-f448-4c14-94d5-b2f83613cacb-kserve-provision-location\") pod \"raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p\" (UID: \"65d5bbcf-f448-4c14-94d5-b2f83613cacb\") " pod="kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p"
Jun 02 16:05:24.387501 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:24.387483    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"raw-sklearn-a5de4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65d5bbcf-f448-4c14-94d5-b2f83613cacb-raw-sklearn-a5de4-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p\" (UID: \"65d5bbcf-f448-4c14-94d5-b2f83613cacb\") " pod="kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p"
Jun 02 16:05:24.389322 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:24.389298    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65d5bbcf-f448-4c14-94d5-b2f83613cacb-proxy-tls\") pod \"raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p\" (UID: \"65d5bbcf-f448-4c14-94d5-b2f83613cacb\") " pod="kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p"
Jun 02 16:05:24.395338 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:24.395302    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wl6z\" (UniqueName: \"kubernetes.io/projected/65d5bbcf-f448-4c14-94d5-b2f83613cacb-kube-api-access-6wl6z\") pod \"raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p\" (UID: \"65d5bbcf-f448-4c14-94d5-b2f83613cacb\") " pod="kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p"
Jun 02 16:05:24.513426 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:24.513386    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p"
Jun 02 16:05:24.658332 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:24.658301    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p"]
Jun 02 16:05:24.660954 ip-10-0-140-246 kubenswrapper[2578]: W0602 16:05:24.660924    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65d5bbcf_f448_4c14_94d5_b2f83613cacb.slice/crio-f3334733a2108a60019848835b8b530a711eafeaaad0f4e894233af6f7269f07 WatchSource:0}: Error finding container f3334733a2108a60019848835b8b530a711eafeaaad0f4e894233af6f7269f07: Status 404 returned error can't find the container with id f3334733a2108a60019848835b8b530a711eafeaaad0f4e894233af6f7269f07
Jun 02 16:05:25.607704 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:25.607663    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p" event={"ID":"65d5bbcf-f448-4c14-94d5-b2f83613cacb","Type":"ContainerStarted","Data":"224750aa1c90e6bfa3ea3b299e468dfc88a6e22d6b2b7fff7a7fb03e147ad2b7"}
Jun 02 16:05:25.607704 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:25.607705    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p" event={"ID":"65d5bbcf-f448-4c14-94d5-b2f83613cacb","Type":"ContainerStarted","Data":"f3334733a2108a60019848835b8b530a711eafeaaad0f4e894233af6f7269f07"}
Jun 02 16:05:27.828731 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:27.828703    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv_247c8ace-4646-4e2b-b9fb-f5afb563080c/storage-initializer/1.log"
Jun 02 16:05:27.829157 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:27.829103    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv_247c8ace-4646-4e2b-b9fb-f5afb563080c/storage-initializer/0.log"
Jun 02 16:05:27.829204 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:27.829169    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv"
Jun 02 16:05:27.917592 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:27.917537    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5z4bt\" (UniqueName: \"kubernetes.io/projected/247c8ace-4646-4e2b-b9fb-f5afb563080c-kube-api-access-5z4bt\") pod \"247c8ace-4646-4e2b-b9fb-f5afb563080c\" (UID: \"247c8ace-4646-4e2b-b9fb-f5afb563080c\") "
Jun 02 16:05:27.917592 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:27.917600    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/247c8ace-4646-4e2b-b9fb-f5afb563080c-proxy-tls\") pod \"247c8ace-4646-4e2b-b9fb-f5afb563080c\" (UID: \"247c8ace-4646-4e2b-b9fb-f5afb563080c\") "
Jun 02 16:05:27.917866 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:27.917648    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-init-fail-0b4fc4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/247c8ace-4646-4e2b-b9fb-f5afb563080c-isvc-init-fail-0b4fc4-kube-rbac-proxy-sar-config\") pod \"247c8ace-4646-4e2b-b9fb-f5afb563080c\" (UID: \"247c8ace-4646-4e2b-b9fb-f5afb563080c\") "
Jun 02 16:05:27.917866 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:27.917707    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/247c8ace-4646-4e2b-b9fb-f5afb563080c-kserve-provision-location\") pod \"247c8ace-4646-4e2b-b9fb-f5afb563080c\" (UID: \"247c8ace-4646-4e2b-b9fb-f5afb563080c\") "
Jun 02 16:05:27.917866 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:27.917801    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/247c8ace-4646-4e2b-b9fb-f5afb563080c-cabundle-cert\") pod \"247c8ace-4646-4e2b-b9fb-f5afb563080c\" (UID: \"247c8ace-4646-4e2b-b9fb-f5afb563080c\") "
Jun 02 16:05:27.918093 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:27.918062    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/247c8ace-4646-4e2b-b9fb-f5afb563080c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "247c8ace-4646-4e2b-b9fb-f5afb563080c" (UID: "247c8ace-4646-4e2b-b9fb-f5afb563080c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:05:27.918161 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:27.918133    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/247c8ace-4646-4e2b-b9fb-f5afb563080c-isvc-init-fail-0b4fc4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-init-fail-0b4fc4-kube-rbac-proxy-sar-config") pod "247c8ace-4646-4e2b-b9fb-f5afb563080c" (UID: "247c8ace-4646-4e2b-b9fb-f5afb563080c"). InnerVolumeSpecName "isvc-init-fail-0b4fc4-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:05:27.918220 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:27.918179    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/247c8ace-4646-4e2b-b9fb-f5afb563080c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "247c8ace-4646-4e2b-b9fb-f5afb563080c" (UID: "247c8ace-4646-4e2b-b9fb-f5afb563080c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:05:27.919902 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:27.919878    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/247c8ace-4646-4e2b-b9fb-f5afb563080c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "247c8ace-4646-4e2b-b9fb-f5afb563080c" (UID: "247c8ace-4646-4e2b-b9fb-f5afb563080c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:05:27.919966 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:27.919889    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/247c8ace-4646-4e2b-b9fb-f5afb563080c-kube-api-access-5z4bt" (OuterVolumeSpecName: "kube-api-access-5z4bt") pod "247c8ace-4646-4e2b-b9fb-f5afb563080c" (UID: "247c8ace-4646-4e2b-b9fb-f5afb563080c"). InnerVolumeSpecName "kube-api-access-5z4bt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:05:28.019206 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:28.019154    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/247c8ace-4646-4e2b-b9fb-f5afb563080c-cabundle-cert\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 16:05:28.019206 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:28.019193    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z4bt\" (UniqueName: \"kubernetes.io/projected/247c8ace-4646-4e2b-b9fb-f5afb563080c-kube-api-access-5z4bt\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 16:05:28.019206 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:28.019205    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/247c8ace-4646-4e2b-b9fb-f5afb563080c-proxy-tls\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 16:05:28.019206 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:28.019216    2578 reconciler_common.go:299] "Volume detached for volume \"isvc-init-fail-0b4fc4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/247c8ace-4646-4e2b-b9fb-f5afb563080c-isvc-init-fail-0b4fc4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 16:05:28.019479 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:28.019226    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/247c8ace-4646-4e2b-b9fb-f5afb563080c-kserve-provision-location\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 16:05:28.619120 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:28.619088    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv_247c8ace-4646-4e2b-b9fb-f5afb563080c/storage-initializer/1.log"
Jun 02 16:05:28.619462 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:28.619447    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv_247c8ace-4646-4e2b-b9fb-f5afb563080c/storage-initializer/0.log"
Jun 02 16:05:28.619533 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:28.619484    2578 generic.go:358] "Generic (PLEG): container finished" podID="247c8ace-4646-4e2b-b9fb-f5afb563080c" containerID="c25315a3e0a7b7ca8512c770f98c89e5327423135ae713aa1955286cad97b4bc" exitCode=1
Jun 02 16:05:28.619603 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:28.619571    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv" event={"ID":"247c8ace-4646-4e2b-b9fb-f5afb563080c","Type":"ContainerDied","Data":"c25315a3e0a7b7ca8512c770f98c89e5327423135ae713aa1955286cad97b4bc"}
Jun 02 16:05:28.619603 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:28.619596    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv"
Jun 02 16:05:28.619706 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:28.619644    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv" event={"ID":"247c8ace-4646-4e2b-b9fb-f5afb563080c","Type":"ContainerDied","Data":"cc35bfcc61e0f2542bcbe4a7bc4d42d574d5f09755b324b387e90866ebffedee"}
Jun 02 16:05:28.619706 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:28.619668    2578 scope.go:117] "RemoveContainer" containerID="c25315a3e0a7b7ca8512c770f98c89e5327423135ae713aa1955286cad97b4bc"
Jun 02 16:05:28.621152 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:28.621123    2578 generic.go:358] "Generic (PLEG): container finished" podID="65d5bbcf-f448-4c14-94d5-b2f83613cacb" containerID="224750aa1c90e6bfa3ea3b299e468dfc88a6e22d6b2b7fff7a7fb03e147ad2b7" exitCode=0
Jun 02 16:05:28.621335 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:28.621163    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p" event={"ID":"65d5bbcf-f448-4c14-94d5-b2f83613cacb","Type":"ContainerDied","Data":"224750aa1c90e6bfa3ea3b299e468dfc88a6e22d6b2b7fff7a7fb03e147ad2b7"}
Jun 02 16:05:28.633699 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:28.633673    2578 scope.go:117] "RemoveContainer" containerID="4544e8bff987c4be4336af3b7730392db9ba403565e5a950e4262cbe57940b4f"
Jun 02 16:05:28.647759 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:28.647709    2578 scope.go:117] "RemoveContainer" containerID="c25315a3e0a7b7ca8512c770f98c89e5327423135ae713aa1955286cad97b4bc"
Jun 02 16:05:28.648136 ip-10-0-140-246 kubenswrapper[2578]: E0602 16:05:28.648116    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c25315a3e0a7b7ca8512c770f98c89e5327423135ae713aa1955286cad97b4bc\": container with ID starting with c25315a3e0a7b7ca8512c770f98c89e5327423135ae713aa1955286cad97b4bc not found: ID does not exist" containerID="c25315a3e0a7b7ca8512c770f98c89e5327423135ae713aa1955286cad97b4bc"
Jun 02 16:05:28.648199 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:28.648147    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c25315a3e0a7b7ca8512c770f98c89e5327423135ae713aa1955286cad97b4bc"} err="failed to get container status \"c25315a3e0a7b7ca8512c770f98c89e5327423135ae713aa1955286cad97b4bc\": rpc error: code = NotFound desc = could not find container \"c25315a3e0a7b7ca8512c770f98c89e5327423135ae713aa1955286cad97b4bc\": container with ID starting with c25315a3e0a7b7ca8512c770f98c89e5327423135ae713aa1955286cad97b4bc not found: ID does not exist"
Jun 02 16:05:28.648199 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:28.648167    2578 scope.go:117] "RemoveContainer" containerID="4544e8bff987c4be4336af3b7730392db9ba403565e5a950e4262cbe57940b4f"
Jun 02 16:05:28.648469 ip-10-0-140-246 kubenswrapper[2578]: E0602 16:05:28.648449    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4544e8bff987c4be4336af3b7730392db9ba403565e5a950e4262cbe57940b4f\": container with ID starting with 4544e8bff987c4be4336af3b7730392db9ba403565e5a950e4262cbe57940b4f not found: ID does not exist" containerID="4544e8bff987c4be4336af3b7730392db9ba403565e5a950e4262cbe57940b4f"
Jun 02 16:05:28.648533 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:28.648477    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4544e8bff987c4be4336af3b7730392db9ba403565e5a950e4262cbe57940b4f"} err="failed to get container status \"4544e8bff987c4be4336af3b7730392db9ba403565e5a950e4262cbe57940b4f\": rpc error: code = NotFound desc = could not find container \"4544e8bff987c4be4336af3b7730392db9ba403565e5a950e4262cbe57940b4f\": container with ID starting with 4544e8bff987c4be4336af3b7730392db9ba403565e5a950e4262cbe57940b4f not found: ID does not exist"
Jun 02 16:05:28.679388 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:28.679351    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv"]
Jun 02 16:05:28.681977 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:28.681943    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-0b4fc4-predictor-9466c49c6-mt7xv"]
Jun 02 16:05:29.631088 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:29.631052    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p" event={"ID":"65d5bbcf-f448-4c14-94d5-b2f83613cacb","Type":"ContainerStarted","Data":"93ac1a342187154fd11bfa0ac22480e6290928856c238c64a9c0b2554ec7e217"}
Jun 02 16:05:29.631489 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:29.631096    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p" event={"ID":"65d5bbcf-f448-4c14-94d5-b2f83613cacb","Type":"ContainerStarted","Data":"4c11886599111ad59b3946fc7cbe2c61ae3a8f60279b3307e6374c4fcfd7a156"}
Jun 02 16:05:29.631489 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:29.631311    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p"
Jun 02 16:05:29.650638 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:29.650574    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p" podStartSLOduration=5.650556898 podStartE2EDuration="5.650556898s" podCreationTimestamp="2026-06-02 16:05:24 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:05:29.649104495 +0000 UTC m=+1178.351568014" watchObservedRunningTime="2026-06-02 16:05:29.650556898 +0000 UTC m=+1178.353020414"
Jun 02 16:05:29.908540 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:29.908502    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="247c8ace-4646-4e2b-b9fb-f5afb563080c" path="/var/lib/kubelet/pods/247c8ace-4646-4e2b-b9fb-f5afb563080c/volumes"
Jun 02 16:05:30.635047 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:30.635015    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p"
Jun 02 16:05:30.636425 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:30.636390    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p" podUID="65d5bbcf-f448-4c14-94d5-b2f83613cacb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jun 02 16:05:31.638210 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:31.638170    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p" podUID="65d5bbcf-f448-4c14-94d5-b2f83613cacb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jun 02 16:05:36.643967 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:36.643936    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p"
Jun 02 16:05:36.644662 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:36.644627    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p" podUID="65d5bbcf-f448-4c14-94d5-b2f83613cacb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jun 02 16:05:46.645164 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:46.645126    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p" podUID="65d5bbcf-f448-4c14-94d5-b2f83613cacb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jun 02 16:05:56.644940 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:05:56.644902    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p" podUID="65d5bbcf-f448-4c14-94d5-b2f83613cacb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jun 02 16:06:06.645331 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:06.645291    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p" podUID="65d5bbcf-f448-4c14-94d5-b2f83613cacb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jun 02 16:06:16.645044 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:16.644988    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p" podUID="65d5bbcf-f448-4c14-94d5-b2f83613cacb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jun 02 16:06:26.645359 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:26.645322    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p" podUID="65d5bbcf-f448-4c14-94d5-b2f83613cacb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jun 02 16:06:36.645156 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:36.645123    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p"
Jun 02 16:06:44.280096 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:44.280025    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p"]
Jun 02 16:06:44.280607 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:44.280308    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p" podUID="65d5bbcf-f448-4c14-94d5-b2f83613cacb" containerName="kserve-container" containerID="cri-o://4c11886599111ad59b3946fc7cbe2c61ae3a8f60279b3307e6374c4fcfd7a156" gracePeriod=30
Jun 02 16:06:44.280607 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:44.280360    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p" podUID="65d5bbcf-f448-4c14-94d5-b2f83613cacb" containerName="kube-rbac-proxy" containerID="cri-o://93ac1a342187154fd11bfa0ac22480e6290928856c238c64a9c0b2554ec7e217" gracePeriod=30
Jun 02 16:06:44.378448 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:44.378414    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc"]
Jun 02 16:06:44.378743 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:44.378730    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="247c8ace-4646-4e2b-b9fb-f5afb563080c" containerName="storage-initializer"
Jun 02 16:06:44.378818 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:44.378745    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="247c8ace-4646-4e2b-b9fb-f5afb563080c" containerName="storage-initializer"
Jun 02 16:06:44.378818 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:44.378764    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="247c8ace-4646-4e2b-b9fb-f5afb563080c" containerName="storage-initializer"
Jun 02 16:06:44.378818 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:44.378769    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="247c8ace-4646-4e2b-b9fb-f5afb563080c" containerName="storage-initializer"
Jun 02 16:06:44.378971 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:44.378833    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="247c8ace-4646-4e2b-b9fb-f5afb563080c" containerName="storage-initializer"
Jun 02 16:06:44.378971 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:44.378931    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="247c8ace-4646-4e2b-b9fb-f5afb563080c" containerName="storage-initializer"
Jun 02 16:06:44.380987 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:44.380971    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc"
Jun 02 16:06:44.383224 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:44.383202    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-c84fb-predictor-serving-cert\""
Jun 02 16:06:44.383326 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:44.383205    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-c84fb-kube-rbac-proxy-sar-config\""
Jun 02 16:06:44.391904 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:44.391881    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc"]
Jun 02 16:06:44.468355 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:44.468325    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/216f876a-b63e-4ce3-a566-20ac5cd45371-proxy-tls\") pod \"raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc\" (UID: \"216f876a-b63e-4ce3-a566-20ac5cd45371\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc"
Jun 02 16:06:44.468355 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:44.468361    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/216f876a-b63e-4ce3-a566-20ac5cd45371-kserve-provision-location\") pod \"raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc\" (UID: \"216f876a-b63e-4ce3-a566-20ac5cd45371\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc"
Jun 02 16:06:44.468563 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:44.468441    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"raw-sklearn-runtime-c84fb-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/216f876a-b63e-4ce3-a566-20ac5cd45371-raw-sklearn-runtime-c84fb-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc\" (UID: \"216f876a-b63e-4ce3-a566-20ac5cd45371\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc"
Jun 02 16:06:44.468563 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:44.468469    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrm8r\" (UniqueName: \"kubernetes.io/projected/216f876a-b63e-4ce3-a566-20ac5cd45371-kube-api-access-qrm8r\") pod \"raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc\" (UID: \"216f876a-b63e-4ce3-a566-20ac5cd45371\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc"
Jun 02 16:06:44.568937 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:44.568857    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/216f876a-b63e-4ce3-a566-20ac5cd45371-proxy-tls\") pod \"raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc\" (UID: \"216f876a-b63e-4ce3-a566-20ac5cd45371\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc"
Jun 02 16:06:44.568937 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:44.568892    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/216f876a-b63e-4ce3-a566-20ac5cd45371-kserve-provision-location\") pod \"raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc\" (UID: \"216f876a-b63e-4ce3-a566-20ac5cd45371\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc"
Jun 02 16:06:44.569141 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:44.568945    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"raw-sklearn-runtime-c84fb-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/216f876a-b63e-4ce3-a566-20ac5cd45371-raw-sklearn-runtime-c84fb-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc\" (UID: \"216f876a-b63e-4ce3-a566-20ac5cd45371\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc"
Jun 02 16:06:44.569141 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:44.568966    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qrm8r\" (UniqueName: \"kubernetes.io/projected/216f876a-b63e-4ce3-a566-20ac5cd45371-kube-api-access-qrm8r\") pod \"raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc\" (UID: \"216f876a-b63e-4ce3-a566-20ac5cd45371\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc"
Jun 02 16:06:44.569412 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:44.569389    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/216f876a-b63e-4ce3-a566-20ac5cd45371-kserve-provision-location\") pod \"raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc\" (UID: \"216f876a-b63e-4ce3-a566-20ac5cd45371\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc"
Jun 02 16:06:44.569597 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:44.569576    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"raw-sklearn-runtime-c84fb-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/216f876a-b63e-4ce3-a566-20ac5cd45371-raw-sklearn-runtime-c84fb-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc\" (UID: \"216f876a-b63e-4ce3-a566-20ac5cd45371\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc"
Jun 02 16:06:44.571142 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:44.571122    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/216f876a-b63e-4ce3-a566-20ac5cd45371-proxy-tls\") pod \"raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc\" (UID: \"216f876a-b63e-4ce3-a566-20ac5cd45371\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc"
Jun 02 16:06:44.577888 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:44.577865    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrm8r\" (UniqueName: \"kubernetes.io/projected/216f876a-b63e-4ce3-a566-20ac5cd45371-kube-api-access-qrm8r\") pod \"raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc\" (UID: \"216f876a-b63e-4ce3-a566-20ac5cd45371\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc"
Jun 02 16:06:44.692685 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:44.692654    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc"
Jun 02 16:06:44.821159 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:44.821134    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc"]
Jun 02 16:06:44.823377 ip-10-0-140-246 kubenswrapper[2578]: W0602 16:06:44.823349    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod216f876a_b63e_4ce3_a566_20ac5cd45371.slice/crio-ce5ad1074ac995e49c0a35d929053bab3838089c06286b9ecda515fd477f249b WatchSource:0}: Error finding container ce5ad1074ac995e49c0a35d929053bab3838089c06286b9ecda515fd477f249b: Status 404 returned error can't find the container with id ce5ad1074ac995e49c0a35d929053bab3838089c06286b9ecda515fd477f249b
Jun 02 16:06:44.854810 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:44.854788    2578 generic.go:358] "Generic (PLEG): container finished" podID="65d5bbcf-f448-4c14-94d5-b2f83613cacb" containerID="93ac1a342187154fd11bfa0ac22480e6290928856c238c64a9c0b2554ec7e217" exitCode=2
Jun 02 16:06:44.854927 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:44.854856    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p" event={"ID":"65d5bbcf-f448-4c14-94d5-b2f83613cacb","Type":"ContainerDied","Data":"93ac1a342187154fd11bfa0ac22480e6290928856c238c64a9c0b2554ec7e217"}
Jun 02 16:06:44.855839 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:44.855820    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc" event={"ID":"216f876a-b63e-4ce3-a566-20ac5cd45371","Type":"ContainerStarted","Data":"ce5ad1074ac995e49c0a35d929053bab3838089c06286b9ecda515fd477f249b"}
Jun 02 16:06:45.859593 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:45.859548    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc" event={"ID":"216f876a-b63e-4ce3-a566-20ac5cd45371","Type":"ContainerStarted","Data":"36256e3ec69e3529e00e712c0991c5fe5ccab59f1d056d2688eacf0d63cfaed2"}
Jun 02 16:06:46.639118 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:46.639075    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p" podUID="65d5bbcf-f448-4c14-94d5-b2f83613cacb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jun 02 16:06:46.645467 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:46.645437    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p" podUID="65d5bbcf-f448-4c14-94d5-b2f83613cacb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jun 02 16:06:48.726678 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:48.726651    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p"
Jun 02 16:06:48.806328 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:48.806242    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65d5bbcf-f448-4c14-94d5-b2f83613cacb-proxy-tls\") pod \"65d5bbcf-f448-4c14-94d5-b2f83613cacb\" (UID: \"65d5bbcf-f448-4c14-94d5-b2f83613cacb\") "
Jun 02 16:06:48.806328 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:48.806289    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65d5bbcf-f448-4c14-94d5-b2f83613cacb-kserve-provision-location\") pod \"65d5bbcf-f448-4c14-94d5-b2f83613cacb\" (UID: \"65d5bbcf-f448-4c14-94d5-b2f83613cacb\") "
Jun 02 16:06:48.806517 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:48.806336    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wl6z\" (UniqueName: \"kubernetes.io/projected/65d5bbcf-f448-4c14-94d5-b2f83613cacb-kube-api-access-6wl6z\") pod \"65d5bbcf-f448-4c14-94d5-b2f83613cacb\" (UID: \"65d5bbcf-f448-4c14-94d5-b2f83613cacb\") "
Jun 02 16:06:48.806517 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:48.806394    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"raw-sklearn-a5de4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65d5bbcf-f448-4c14-94d5-b2f83613cacb-raw-sklearn-a5de4-kube-rbac-proxy-sar-config\") pod \"65d5bbcf-f448-4c14-94d5-b2f83613cacb\" (UID: \"65d5bbcf-f448-4c14-94d5-b2f83613cacb\") "
Jun 02 16:06:48.806720 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:48.806693    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d5bbcf-f448-4c14-94d5-b2f83613cacb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "65d5bbcf-f448-4c14-94d5-b2f83613cacb" (UID: "65d5bbcf-f448-4c14-94d5-b2f83613cacb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:06:48.806788 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:48.806757    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65d5bbcf-f448-4c14-94d5-b2f83613cacb-raw-sklearn-a5de4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "raw-sklearn-a5de4-kube-rbac-proxy-sar-config") pod "65d5bbcf-f448-4c14-94d5-b2f83613cacb" (UID: "65d5bbcf-f448-4c14-94d5-b2f83613cacb"). InnerVolumeSpecName "raw-sklearn-a5de4-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:06:48.808379 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:48.808353    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65d5bbcf-f448-4c14-94d5-b2f83613cacb-kube-api-access-6wl6z" (OuterVolumeSpecName: "kube-api-access-6wl6z") pod "65d5bbcf-f448-4c14-94d5-b2f83613cacb" (UID: "65d5bbcf-f448-4c14-94d5-b2f83613cacb"). InnerVolumeSpecName "kube-api-access-6wl6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:06:48.808458 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:48.808404    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65d5bbcf-f448-4c14-94d5-b2f83613cacb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "65d5bbcf-f448-4c14-94d5-b2f83613cacb" (UID: "65d5bbcf-f448-4c14-94d5-b2f83613cacb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:06:48.871008 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:48.870963    2578 generic.go:358] "Generic (PLEG): container finished" podID="65d5bbcf-f448-4c14-94d5-b2f83613cacb" containerID="4c11886599111ad59b3946fc7cbe2c61ae3a8f60279b3307e6374c4fcfd7a156" exitCode=0
Jun 02 16:06:48.871182 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:48.871038    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p" event={"ID":"65d5bbcf-f448-4c14-94d5-b2f83613cacb","Type":"ContainerDied","Data":"4c11886599111ad59b3946fc7cbe2c61ae3a8f60279b3307e6374c4fcfd7a156"}
Jun 02 16:06:48.871182 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:48.871060    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p"
Jun 02 16:06:48.871182 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:48.871075    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p" event={"ID":"65d5bbcf-f448-4c14-94d5-b2f83613cacb","Type":"ContainerDied","Data":"f3334733a2108a60019848835b8b530a711eafeaaad0f4e894233af6f7269f07"}
Jun 02 16:06:48.871182 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:48.871093    2578 scope.go:117] "RemoveContainer" containerID="93ac1a342187154fd11bfa0ac22480e6290928856c238c64a9c0b2554ec7e217"
Jun 02 16:06:48.872407 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:48.872386    2578 generic.go:358] "Generic (PLEG): container finished" podID="216f876a-b63e-4ce3-a566-20ac5cd45371" containerID="36256e3ec69e3529e00e712c0991c5fe5ccab59f1d056d2688eacf0d63cfaed2" exitCode=0
Jun 02 16:06:48.872518 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:48.872437    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc" event={"ID":"216f876a-b63e-4ce3-a566-20ac5cd45371","Type":"ContainerDied","Data":"36256e3ec69e3529e00e712c0991c5fe5ccab59f1d056d2688eacf0d63cfaed2"}
Jun 02 16:06:48.879873 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:48.879767    2578 scope.go:117] "RemoveContainer" containerID="4c11886599111ad59b3946fc7cbe2c61ae3a8f60279b3307e6374c4fcfd7a156"
Jun 02 16:06:48.886960 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:48.886936    2578 scope.go:117] "RemoveContainer" containerID="224750aa1c90e6bfa3ea3b299e468dfc88a6e22d6b2b7fff7a7fb03e147ad2b7"
Jun 02 16:06:48.893843 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:48.893821    2578 scope.go:117] "RemoveContainer" containerID="93ac1a342187154fd11bfa0ac22480e6290928856c238c64a9c0b2554ec7e217"
Jun 02 16:06:48.894105 ip-10-0-140-246 kubenswrapper[2578]: E0602 16:06:48.894087    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93ac1a342187154fd11bfa0ac22480e6290928856c238c64a9c0b2554ec7e217\": container with ID starting with 93ac1a342187154fd11bfa0ac22480e6290928856c238c64a9c0b2554ec7e217 not found: ID does not exist" containerID="93ac1a342187154fd11bfa0ac22480e6290928856c238c64a9c0b2554ec7e217"
Jun 02 16:06:48.894165 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:48.894114    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93ac1a342187154fd11bfa0ac22480e6290928856c238c64a9c0b2554ec7e217"} err="failed to get container status \"93ac1a342187154fd11bfa0ac22480e6290928856c238c64a9c0b2554ec7e217\": rpc error: code = NotFound desc = could not find container \"93ac1a342187154fd11bfa0ac22480e6290928856c238c64a9c0b2554ec7e217\": container with ID starting with 93ac1a342187154fd11bfa0ac22480e6290928856c238c64a9c0b2554ec7e217 not found: ID does not exist"
Jun 02 16:06:48.894165 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:48.894133    2578 scope.go:117] "RemoveContainer" containerID="4c11886599111ad59b3946fc7cbe2c61ae3a8f60279b3307e6374c4fcfd7a156"
Jun 02 16:06:48.894392 ip-10-0-140-246 kubenswrapper[2578]: E0602 16:06:48.894375    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c11886599111ad59b3946fc7cbe2c61ae3a8f60279b3307e6374c4fcfd7a156\": container with ID starting with 4c11886599111ad59b3946fc7cbe2c61ae3a8f60279b3307e6374c4fcfd7a156 not found: ID does not exist" containerID="4c11886599111ad59b3946fc7cbe2c61ae3a8f60279b3307e6374c4fcfd7a156"
Jun 02 16:06:48.894433 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:48.894397    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c11886599111ad59b3946fc7cbe2c61ae3a8f60279b3307e6374c4fcfd7a156"} err="failed to get container status \"4c11886599111ad59b3946fc7cbe2c61ae3a8f60279b3307e6374c4fcfd7a156\": rpc error: code = NotFound desc = could not find container \"4c11886599111ad59b3946fc7cbe2c61ae3a8f60279b3307e6374c4fcfd7a156\": container with ID starting with 4c11886599111ad59b3946fc7cbe2c61ae3a8f60279b3307e6374c4fcfd7a156 not found: ID does not exist"
Jun 02 16:06:48.894433 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:48.894413    2578 scope.go:117] "RemoveContainer" containerID="224750aa1c90e6bfa3ea3b299e468dfc88a6e22d6b2b7fff7a7fb03e147ad2b7"
Jun 02 16:06:48.894617 ip-10-0-140-246 kubenswrapper[2578]: E0602 16:06:48.894603    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"224750aa1c90e6bfa3ea3b299e468dfc88a6e22d6b2b7fff7a7fb03e147ad2b7\": container with ID starting with 224750aa1c90e6bfa3ea3b299e468dfc88a6e22d6b2b7fff7a7fb03e147ad2b7 not found: ID does not exist" containerID="224750aa1c90e6bfa3ea3b299e468dfc88a6e22d6b2b7fff7a7fb03e147ad2b7"
Jun 02 16:06:48.894658 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:48.894620    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"224750aa1c90e6bfa3ea3b299e468dfc88a6e22d6b2b7fff7a7fb03e147ad2b7"} err="failed to get container status \"224750aa1c90e6bfa3ea3b299e468dfc88a6e22d6b2b7fff7a7fb03e147ad2b7\": rpc error: code = NotFound desc = could not find container \"224750aa1c90e6bfa3ea3b299e468dfc88a6e22d6b2b7fff7a7fb03e147ad2b7\": container with ID starting with 224750aa1c90e6bfa3ea3b299e468dfc88a6e22d6b2b7fff7a7fb03e147ad2b7 not found: ID does not exist"
Jun 02 16:06:48.907930 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:48.907905    2578 reconciler_common.go:299] "Volume detached for volume \"raw-sklearn-a5de4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65d5bbcf-f448-4c14-94d5-b2f83613cacb-raw-sklearn-a5de4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 16:06:48.907930 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:48.907932    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65d5bbcf-f448-4c14-94d5-b2f83613cacb-proxy-tls\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 16:06:48.908075 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:48.907942    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65d5bbcf-f448-4c14-94d5-b2f83613cacb-kserve-provision-location\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 16:06:48.908075 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:48.907951    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wl6z\" (UniqueName: \"kubernetes.io/projected/65d5bbcf-f448-4c14-94d5-b2f83613cacb-kube-api-access-6wl6z\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 16:06:48.914358 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:48.914336    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p"]
Jun 02 16:06:48.923133 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:48.923112    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-a5de4-predictor-cd89bbb98-8cl4p"]
Jun 02 16:06:49.877524 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:49.877488    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc" event={"ID":"216f876a-b63e-4ce3-a566-20ac5cd45371","Type":"ContainerStarted","Data":"697ec75a2cd96db3ae912556333bf51ba4134eb237c6a3fc3303b35b127fa4ce"}
Jun 02 16:06:49.878336 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:49.877533    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc" event={"ID":"216f876a-b63e-4ce3-a566-20ac5cd45371","Type":"ContainerStarted","Data":"070d29aee008198f15ebba31cbe6aa1298cc87f3f998a8e127dbfa1950ffe28c"}
Jun 02 16:06:49.878336 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:49.877777    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc"
Jun 02 16:06:49.900686 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:49.900635    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc" podStartSLOduration=5.900620949 podStartE2EDuration="5.900620949s" podCreationTimestamp="2026-06-02 16:06:44 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:06:49.898561899 +0000 UTC m=+1258.601025419" watchObservedRunningTime="2026-06-02 16:06:49.900620949 +0000 UTC m=+1258.603084468"
Jun 02 16:06:49.907597 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:49.907573    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65d5bbcf-f448-4c14-94d5-b2f83613cacb" path="/var/lib/kubelet/pods/65d5bbcf-f448-4c14-94d5-b2f83613cacb/volumes"
Jun 02 16:06:50.881186 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:50.881158    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc"
Jun 02 16:06:50.882477 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:50.882453    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc" podUID="216f876a-b63e-4ce3-a566-20ac5cd45371" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jun 02 16:06:51.883712 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:51.883668    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc" podUID="216f876a-b63e-4ce3-a566-20ac5cd45371" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jun 02 16:06:56.888473 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:56.888448    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc"
Jun 02 16:06:56.889058 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:06:56.889031    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc" podUID="216f876a-b63e-4ce3-a566-20ac5cd45371" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jun 02 16:07:06.889924 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:07:06.889883    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc" podUID="216f876a-b63e-4ce3-a566-20ac5cd45371" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jun 02 16:07:16.889228 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:07:16.889193    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc" podUID="216f876a-b63e-4ce3-a566-20ac5cd45371" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jun 02 16:07:26.889709 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:07:26.889670    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc" podUID="216f876a-b63e-4ce3-a566-20ac5cd45371" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jun 02 16:07:36.889111 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:07:36.889074    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc" podUID="216f876a-b63e-4ce3-a566-20ac5cd45371" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jun 02 16:07:46.889566 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:07:46.889525    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc" podUID="216f876a-b63e-4ce3-a566-20ac5cd45371" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jun 02 16:07:56.889839 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:07:56.889811    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc"
Jun 02 16:08:04.613056 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:04.613022    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc"]
Jun 02 16:08:04.613514 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:04.613306    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc" podUID="216f876a-b63e-4ce3-a566-20ac5cd45371" containerName="kserve-container" containerID="cri-o://070d29aee008198f15ebba31cbe6aa1298cc87f3f998a8e127dbfa1950ffe28c" gracePeriod=30
Jun 02 16:08:04.613514 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:04.613344    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc" podUID="216f876a-b63e-4ce3-a566-20ac5cd45371" containerName="kube-rbac-proxy" containerID="cri-o://697ec75a2cd96db3ae912556333bf51ba4134eb237c6a3fc3303b35b127fa4ce" gracePeriod=30
Jun 02 16:08:05.092318 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:05.092284    2578 generic.go:358] "Generic (PLEG): container finished" podID="216f876a-b63e-4ce3-a566-20ac5cd45371" containerID="697ec75a2cd96db3ae912556333bf51ba4134eb237c6a3fc3303b35b127fa4ce" exitCode=2
Jun 02 16:08:05.092488 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:05.092357    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc" event={"ID":"216f876a-b63e-4ce3-a566-20ac5cd45371","Type":"ContainerDied","Data":"697ec75a2cd96db3ae912556333bf51ba4134eb237c6a3fc3303b35b127fa4ce"}
Jun 02 16:08:05.666894 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:05.666855    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9vsmp/must-gather-k2bmf"]
Jun 02 16:08:05.667377 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:05.667359    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65d5bbcf-f448-4c14-94d5-b2f83613cacb" containerName="storage-initializer"
Jun 02 16:08:05.667456 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:05.667381    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="65d5bbcf-f448-4c14-94d5-b2f83613cacb" containerName="storage-initializer"
Jun 02 16:08:05.667456 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:05.667400    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65d5bbcf-f448-4c14-94d5-b2f83613cacb" containerName="kserve-container"
Jun 02 16:08:05.667456 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:05.667408    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="65d5bbcf-f448-4c14-94d5-b2f83613cacb" containerName="kserve-container"
Jun 02 16:08:05.667456 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:05.667435    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65d5bbcf-f448-4c14-94d5-b2f83613cacb" containerName="kube-rbac-proxy"
Jun 02 16:08:05.667456 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:05.667444    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="65d5bbcf-f448-4c14-94d5-b2f83613cacb" containerName="kube-rbac-proxy"
Jun 02 16:08:05.667696 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:05.667517    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="65d5bbcf-f448-4c14-94d5-b2f83613cacb" containerName="kserve-container"
Jun 02 16:08:05.667696 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:05.667532    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="65d5bbcf-f448-4c14-94d5-b2f83613cacb" containerName="kube-rbac-proxy"
Jun 02 16:08:05.670584 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:05.670564    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9vsmp/must-gather-k2bmf"
Jun 02 16:08:05.672780 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:05.672762    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9vsmp\"/\"kube-root-ca.crt\""
Jun 02 16:08:05.672857 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:05.672765    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9vsmp\"/\"openshift-service-ca.crt\""
Jun 02 16:08:05.673207 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:05.673193    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9vsmp\"/\"default-dockercfg-vpf5r\""
Jun 02 16:08:05.678273 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:05.678253    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9vsmp/must-gather-k2bmf"]
Jun 02 16:08:05.742317 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:05.742285    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a17d3af4-c896-45ab-b6a3-9489ea37194c-must-gather-output\") pod \"must-gather-k2bmf\" (UID: \"a17d3af4-c896-45ab-b6a3-9489ea37194c\") " pod="openshift-must-gather-9vsmp/must-gather-k2bmf"
Jun 02 16:08:05.742461 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:05.742325    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xj48\" (UniqueName: \"kubernetes.io/projected/a17d3af4-c896-45ab-b6a3-9489ea37194c-kube-api-access-6xj48\") pod \"must-gather-k2bmf\" (UID: \"a17d3af4-c896-45ab-b6a3-9489ea37194c\") " pod="openshift-must-gather-9vsmp/must-gather-k2bmf"
Jun 02 16:08:05.842934 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:05.842900    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a17d3af4-c896-45ab-b6a3-9489ea37194c-must-gather-output\") pod \"must-gather-k2bmf\" (UID: \"a17d3af4-c896-45ab-b6a3-9489ea37194c\") " pod="openshift-must-gather-9vsmp/must-gather-k2bmf"
Jun 02 16:08:05.843112 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:05.842942    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6xj48\" (UniqueName: \"kubernetes.io/projected/a17d3af4-c896-45ab-b6a3-9489ea37194c-kube-api-access-6xj48\") pod \"must-gather-k2bmf\" (UID: \"a17d3af4-c896-45ab-b6a3-9489ea37194c\") " pod="openshift-must-gather-9vsmp/must-gather-k2bmf"
Jun 02 16:08:05.843260 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:05.843242    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a17d3af4-c896-45ab-b6a3-9489ea37194c-must-gather-output\") pod \"must-gather-k2bmf\" (UID: \"a17d3af4-c896-45ab-b6a3-9489ea37194c\") " pod="openshift-must-gather-9vsmp/must-gather-k2bmf"
Jun 02 16:08:05.851243 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:05.851217    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xj48\" (UniqueName: \"kubernetes.io/projected/a17d3af4-c896-45ab-b6a3-9489ea37194c-kube-api-access-6xj48\") pod \"must-gather-k2bmf\" (UID: \"a17d3af4-c896-45ab-b6a3-9489ea37194c\") " pod="openshift-must-gather-9vsmp/must-gather-k2bmf"
Jun 02 16:08:05.991088 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:05.991022    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9vsmp/must-gather-k2bmf"
Jun 02 16:08:06.116019 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:06.115974    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9vsmp/must-gather-k2bmf"]
Jun 02 16:08:06.119403 ip-10-0-140-246 kubenswrapper[2578]: W0602 16:08:06.119375    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda17d3af4_c896_45ab_b6a3_9489ea37194c.slice/crio-449eb3913abf3f8a260fc1add08957b4a57e83ee3566c260c456c04594422137 WatchSource:0}: Error finding container 449eb3913abf3f8a260fc1add08957b4a57e83ee3566c260c456c04594422137: Status 404 returned error can't find the container with id 449eb3913abf3f8a260fc1add08957b4a57e83ee3566c260c456c04594422137
Jun 02 16:08:06.884588 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:06.884540    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc" podUID="216f876a-b63e-4ce3-a566-20ac5cd45371" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jun 02 16:08:06.889214 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:06.889180    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc" podUID="216f876a-b63e-4ce3-a566-20ac5cd45371" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jun 02 16:08:07.100102 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:07.100043    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9vsmp/must-gather-k2bmf" event={"ID":"a17d3af4-c896-45ab-b6a3-9489ea37194c","Type":"ContainerStarted","Data":"449eb3913abf3f8a260fc1add08957b4a57e83ee3566c260c456c04594422137"}
Jun 02 16:08:09.937532 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:09.937512    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc"
Jun 02 16:08:09.982679 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:09.982644    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/216f876a-b63e-4ce3-a566-20ac5cd45371-kserve-provision-location\") pod \"216f876a-b63e-4ce3-a566-20ac5cd45371\" (UID: \"216f876a-b63e-4ce3-a566-20ac5cd45371\") "
Jun 02 16:08:09.982820 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:09.982707    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrm8r\" (UniqueName: \"kubernetes.io/projected/216f876a-b63e-4ce3-a566-20ac5cd45371-kube-api-access-qrm8r\") pod \"216f876a-b63e-4ce3-a566-20ac5cd45371\" (UID: \"216f876a-b63e-4ce3-a566-20ac5cd45371\") "
Jun 02 16:08:09.982820 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:09.982740    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"raw-sklearn-runtime-c84fb-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/216f876a-b63e-4ce3-a566-20ac5cd45371-raw-sklearn-runtime-c84fb-kube-rbac-proxy-sar-config\") pod \"216f876a-b63e-4ce3-a566-20ac5cd45371\" (UID: \"216f876a-b63e-4ce3-a566-20ac5cd45371\") "
Jun 02 16:08:09.982820 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:09.982780    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/216f876a-b63e-4ce3-a566-20ac5cd45371-proxy-tls\") pod \"216f876a-b63e-4ce3-a566-20ac5cd45371\" (UID: \"216f876a-b63e-4ce3-a566-20ac5cd45371\") "
Jun 02 16:08:09.983031 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:09.983005    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/216f876a-b63e-4ce3-a566-20ac5cd45371-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "216f876a-b63e-4ce3-a566-20ac5cd45371" (UID: "216f876a-b63e-4ce3-a566-20ac5cd45371"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:08:09.983116 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:09.983099    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/216f876a-b63e-4ce3-a566-20ac5cd45371-raw-sklearn-runtime-c84fb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "raw-sklearn-runtime-c84fb-kube-rbac-proxy-sar-config") pod "216f876a-b63e-4ce3-a566-20ac5cd45371" (UID: "216f876a-b63e-4ce3-a566-20ac5cd45371"). InnerVolumeSpecName "raw-sklearn-runtime-c84fb-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:08:09.984952 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:09.984918    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/216f876a-b63e-4ce3-a566-20ac5cd45371-kube-api-access-qrm8r" (OuterVolumeSpecName: "kube-api-access-qrm8r") pod "216f876a-b63e-4ce3-a566-20ac5cd45371" (UID: "216f876a-b63e-4ce3-a566-20ac5cd45371"). InnerVolumeSpecName "kube-api-access-qrm8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:08:09.985225 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:09.985207    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/216f876a-b63e-4ce3-a566-20ac5cd45371-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "216f876a-b63e-4ce3-a566-20ac5cd45371" (UID: "216f876a-b63e-4ce3-a566-20ac5cd45371"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:08:10.083452 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:10.083391    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrm8r\" (UniqueName: \"kubernetes.io/projected/216f876a-b63e-4ce3-a566-20ac5cd45371-kube-api-access-qrm8r\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 16:08:10.083452 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:10.083416    2578 reconciler_common.go:299] "Volume detached for volume \"raw-sklearn-runtime-c84fb-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/216f876a-b63e-4ce3-a566-20ac5cd45371-raw-sklearn-runtime-c84fb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 16:08:10.083452 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:10.083428    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/216f876a-b63e-4ce3-a566-20ac5cd45371-proxy-tls\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 16:08:10.083452 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:10.083437    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/216f876a-b63e-4ce3-a566-20ac5cd45371-kserve-provision-location\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 16:08:10.111457 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:10.111425    2578 generic.go:358] "Generic (PLEG): container finished" podID="216f876a-b63e-4ce3-a566-20ac5cd45371" containerID="070d29aee008198f15ebba31cbe6aa1298cc87f3f998a8e127dbfa1950ffe28c" exitCode=0
Jun 02 16:08:10.111577 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:10.111502    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc" event={"ID":"216f876a-b63e-4ce3-a566-20ac5cd45371","Type":"ContainerDied","Data":"070d29aee008198f15ebba31cbe6aa1298cc87f3f998a8e127dbfa1950ffe28c"}
Jun 02 16:08:10.111577 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:10.111522    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc"
Jun 02 16:08:10.111577 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:10.111534    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc" event={"ID":"216f876a-b63e-4ce3-a566-20ac5cd45371","Type":"ContainerDied","Data":"ce5ad1074ac995e49c0a35d929053bab3838089c06286b9ecda515fd477f249b"}
Jun 02 16:08:10.111577 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:10.111550    2578 scope.go:117] "RemoveContainer" containerID="697ec75a2cd96db3ae912556333bf51ba4134eb237c6a3fc3303b35b127fa4ce"
Jun 02 16:08:10.120416 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:10.120385    2578 scope.go:117] "RemoveContainer" containerID="070d29aee008198f15ebba31cbe6aa1298cc87f3f998a8e127dbfa1950ffe28c"
Jun 02 16:08:10.128816 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:10.128796    2578 scope.go:117] "RemoveContainer" containerID="36256e3ec69e3529e00e712c0991c5fe5ccab59f1d056d2688eacf0d63cfaed2"
Jun 02 16:08:10.135492 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:10.135474    2578 scope.go:117] "RemoveContainer" containerID="697ec75a2cd96db3ae912556333bf51ba4134eb237c6a3fc3303b35b127fa4ce"
Jun 02 16:08:10.135711 ip-10-0-140-246 kubenswrapper[2578]: E0602 16:08:10.135693    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"697ec75a2cd96db3ae912556333bf51ba4134eb237c6a3fc3303b35b127fa4ce\": container with ID starting with 697ec75a2cd96db3ae912556333bf51ba4134eb237c6a3fc3303b35b127fa4ce not found: ID does not exist" containerID="697ec75a2cd96db3ae912556333bf51ba4134eb237c6a3fc3303b35b127fa4ce"
Jun 02 16:08:10.135765 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:10.135721    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"697ec75a2cd96db3ae912556333bf51ba4134eb237c6a3fc3303b35b127fa4ce"} err="failed to get container status \"697ec75a2cd96db3ae912556333bf51ba4134eb237c6a3fc3303b35b127fa4ce\": rpc error: code = NotFound desc = could not find container \"697ec75a2cd96db3ae912556333bf51ba4134eb237c6a3fc3303b35b127fa4ce\": container with ID starting with 697ec75a2cd96db3ae912556333bf51ba4134eb237c6a3fc3303b35b127fa4ce not found: ID does not exist"
Jun 02 16:08:10.135765 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:10.135741    2578 scope.go:117] "RemoveContainer" containerID="070d29aee008198f15ebba31cbe6aa1298cc87f3f998a8e127dbfa1950ffe28c"
Jun 02 16:08:10.136100 ip-10-0-140-246 kubenswrapper[2578]: E0602 16:08:10.135982    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"070d29aee008198f15ebba31cbe6aa1298cc87f3f998a8e127dbfa1950ffe28c\": container with ID starting with 070d29aee008198f15ebba31cbe6aa1298cc87f3f998a8e127dbfa1950ffe28c not found: ID does not exist" containerID="070d29aee008198f15ebba31cbe6aa1298cc87f3f998a8e127dbfa1950ffe28c"
Jun 02 16:08:10.136100 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:10.136019    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"070d29aee008198f15ebba31cbe6aa1298cc87f3f998a8e127dbfa1950ffe28c"} err="failed to get container status \"070d29aee008198f15ebba31cbe6aa1298cc87f3f998a8e127dbfa1950ffe28c\": rpc error: code = NotFound desc = could not find container \"070d29aee008198f15ebba31cbe6aa1298cc87f3f998a8e127dbfa1950ffe28c\": container with ID starting with 070d29aee008198f15ebba31cbe6aa1298cc87f3f998a8e127dbfa1950ffe28c not found: ID does not exist"
Jun 02 16:08:10.136100 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:10.136042    2578 scope.go:117] "RemoveContainer" containerID="36256e3ec69e3529e00e712c0991c5fe5ccab59f1d056d2688eacf0d63cfaed2"
Jun 02 16:08:10.136575 ip-10-0-140-246 kubenswrapper[2578]: E0602 16:08:10.136552    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36256e3ec69e3529e00e712c0991c5fe5ccab59f1d056d2688eacf0d63cfaed2\": container with ID starting with 36256e3ec69e3529e00e712c0991c5fe5ccab59f1d056d2688eacf0d63cfaed2 not found: ID does not exist" containerID="36256e3ec69e3529e00e712c0991c5fe5ccab59f1d056d2688eacf0d63cfaed2"
Jun 02 16:08:10.136646 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:10.136582    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36256e3ec69e3529e00e712c0991c5fe5ccab59f1d056d2688eacf0d63cfaed2"} err="failed to get container status \"36256e3ec69e3529e00e712c0991c5fe5ccab59f1d056d2688eacf0d63cfaed2\": rpc error: code = NotFound desc = could not find container \"36256e3ec69e3529e00e712c0991c5fe5ccab59f1d056d2688eacf0d63cfaed2\": container with ID starting with 36256e3ec69e3529e00e712c0991c5fe5ccab59f1d056d2688eacf0d63cfaed2 not found: ID does not exist"
Jun 02 16:08:10.138442 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:10.138422    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc"]
Jun 02 16:08:10.142562 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:10.142530    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-c84fb-predictor-68c976fd68-xd6jc"]
Jun 02 16:08:11.909253 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:11.909220    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="216f876a-b63e-4ce3-a566-20ac5cd45371" path="/var/lib/kubelet/pods/216f876a-b63e-4ce3-a566-20ac5cd45371/volumes"
Jun 02 16:08:13.123194 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:13.123155    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9vsmp/must-gather-k2bmf" event={"ID":"a17d3af4-c896-45ab-b6a3-9489ea37194c","Type":"ContainerStarted","Data":"0da9b5a02004af4417cd125547cc8a36d722f2772bfd5a455ffc4afe8c44e0bc"}
Jun 02 16:08:13.123194 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:13.123192    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9vsmp/must-gather-k2bmf" event={"ID":"a17d3af4-c896-45ab-b6a3-9489ea37194c","Type":"ContainerStarted","Data":"094270c2b3a3718e909b7520ea1cfc74d56c283221f348df6ab287b423df362a"}
Jun 02 16:08:13.150583 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:13.150526    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-9vsmp/must-gather-k2bmf" podStartSLOduration=1.7678923229999999 podStartE2EDuration="8.150512247s" podCreationTimestamp="2026-06-02 16:08:05 +0000 UTC" firstStartedPulling="2026-06-02 16:08:06.120939619 +0000 UTC m=+1334.823403119" lastFinishedPulling="2026-06-02 16:08:12.503559543 +0000 UTC m=+1341.206023043" observedRunningTime="2026-06-02 16:08:13.14920492 +0000 UTC m=+1341.851668437" watchObservedRunningTime="2026-06-02 16:08:13.150512247 +0000 UTC m=+1341.852975765"
Jun 02 16:08:33.183365 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:33.183334    2578 generic.go:358] "Generic (PLEG): container finished" podID="a17d3af4-c896-45ab-b6a3-9489ea37194c" containerID="094270c2b3a3718e909b7520ea1cfc74d56c283221f348df6ab287b423df362a" exitCode=0
Jun 02 16:08:33.183760 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:33.183411    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9vsmp/must-gather-k2bmf" event={"ID":"a17d3af4-c896-45ab-b6a3-9489ea37194c","Type":"ContainerDied","Data":"094270c2b3a3718e909b7520ea1cfc74d56c283221f348df6ab287b423df362a"}
Jun 02 16:08:33.183760 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:33.183700    2578 scope.go:117] "RemoveContainer" containerID="094270c2b3a3718e909b7520ea1cfc74d56c283221f348df6ab287b423df362a"
Jun 02 16:08:33.223445 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:33.223421    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-9vsmp_must-gather-k2bmf_a17d3af4-c896-45ab-b6a3-9489ea37194c/gather/0.log"
Jun 02 16:08:33.782671 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:33.782637    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s989l/must-gather-mmjdh"]
Jun 02 16:08:33.782957 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:33.782944    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="216f876a-b63e-4ce3-a566-20ac5cd45371" containerName="kserve-container"
Jun 02 16:08:33.783024 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:33.782959    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="216f876a-b63e-4ce3-a566-20ac5cd45371" containerName="kserve-container"
Jun 02 16:08:33.783024 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:33.782973    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="216f876a-b63e-4ce3-a566-20ac5cd45371" containerName="storage-initializer"
Jun 02 16:08:33.783024 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:33.782980    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="216f876a-b63e-4ce3-a566-20ac5cd45371" containerName="storage-initializer"
Jun 02 16:08:33.783024 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:33.783006    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="216f876a-b63e-4ce3-a566-20ac5cd45371" containerName="kube-rbac-proxy"
Jun 02 16:08:33.783024 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:33.783012    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="216f876a-b63e-4ce3-a566-20ac5cd45371" containerName="kube-rbac-proxy"
Jun 02 16:08:33.783180 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:33.783068    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="216f876a-b63e-4ce3-a566-20ac5cd45371" containerName="kube-rbac-proxy"
Jun 02 16:08:33.783180 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:33.783079    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="216f876a-b63e-4ce3-a566-20ac5cd45371" containerName="kserve-container"
Jun 02 16:08:33.785188 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:33.785173    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s989l/must-gather-mmjdh"
Jun 02 16:08:33.787414 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:33.787393    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s989l\"/\"openshift-service-ca.crt\""
Jun 02 16:08:33.787949 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:33.787931    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-s989l\"/\"default-dockercfg-mv4r5\""
Jun 02 16:08:33.788082 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:33.787955    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s989l\"/\"kube-root-ca.crt\""
Jun 02 16:08:33.795303 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:33.795279    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s989l/must-gather-mmjdh"]
Jun 02 16:08:33.899130 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:33.899095    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0e77d249-abab-4d6b-aef8-565cd6698cbd-must-gather-output\") pod \"must-gather-mmjdh\" (UID: \"0e77d249-abab-4d6b-aef8-565cd6698cbd\") " pod="openshift-must-gather-s989l/must-gather-mmjdh"
Jun 02 16:08:33.899311 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:33.899173    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-262v9\" (UniqueName: \"kubernetes.io/projected/0e77d249-abab-4d6b-aef8-565cd6698cbd-kube-api-access-262v9\") pod \"must-gather-mmjdh\" (UID: \"0e77d249-abab-4d6b-aef8-565cd6698cbd\") " pod="openshift-must-gather-s989l/must-gather-mmjdh"
Jun 02 16:08:34.000293 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:34.000260    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-262v9\" (UniqueName: \"kubernetes.io/projected/0e77d249-abab-4d6b-aef8-565cd6698cbd-kube-api-access-262v9\") pod \"must-gather-mmjdh\" (UID: \"0e77d249-abab-4d6b-aef8-565cd6698cbd\") " pod="openshift-must-gather-s989l/must-gather-mmjdh"
Jun 02 16:08:34.000460 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:34.000421    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0e77d249-abab-4d6b-aef8-565cd6698cbd-must-gather-output\") pod \"must-gather-mmjdh\" (UID: \"0e77d249-abab-4d6b-aef8-565cd6698cbd\") " pod="openshift-must-gather-s989l/must-gather-mmjdh"
Jun 02 16:08:34.000756 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:34.000736    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0e77d249-abab-4d6b-aef8-565cd6698cbd-must-gather-output\") pod \"must-gather-mmjdh\" (UID: \"0e77d249-abab-4d6b-aef8-565cd6698cbd\") " pod="openshift-must-gather-s989l/must-gather-mmjdh"
Jun 02 16:08:34.009323 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:34.009302    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-262v9\" (UniqueName: \"kubernetes.io/projected/0e77d249-abab-4d6b-aef8-565cd6698cbd-kube-api-access-262v9\") pod \"must-gather-mmjdh\" (UID: \"0e77d249-abab-4d6b-aef8-565cd6698cbd\") " pod="openshift-must-gather-s989l/must-gather-mmjdh"
Jun 02 16:08:34.094117 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:34.094044    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s989l/must-gather-mmjdh"
Jun 02 16:08:34.216685 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:34.216651    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s989l/must-gather-mmjdh"]
Jun 02 16:08:34.219709 ip-10-0-140-246 kubenswrapper[2578]: W0602 16:08:34.219682    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e77d249_abab_4d6b_aef8_565cd6698cbd.slice/crio-8475688dd4f404f2afcc58d8b8b8027932af4cdcd453e28a0d836f19fedbf17e WatchSource:0}: Error finding container 8475688dd4f404f2afcc58d8b8b8027932af4cdcd453e28a0d836f19fedbf17e: Status 404 returned error can't find the container with id 8475688dd4f404f2afcc58d8b8b8027932af4cdcd453e28a0d836f19fedbf17e
Jun 02 16:08:35.193031 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:35.192933    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s989l/must-gather-mmjdh" event={"ID":"0e77d249-abab-4d6b-aef8-565cd6698cbd","Type":"ContainerStarted","Data":"9e2d4188c52796607e7a2dad23eba1f7da2050af5b25c34926e3a250de89a1f5"}
Jun 02 16:08:35.193031 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:35.192979    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s989l/must-gather-mmjdh" event={"ID":"0e77d249-abab-4d6b-aef8-565cd6698cbd","Type":"ContainerStarted","Data":"a99ed4599e6aa22bdc481ae2940f8926af49ede2284d45fc21ade3afe7d917dc"}
Jun 02 16:08:35.193031 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:35.193012    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s989l/must-gather-mmjdh" event={"ID":"0e77d249-abab-4d6b-aef8-565cd6698cbd","Type":"ContainerStarted","Data":"8475688dd4f404f2afcc58d8b8b8027932af4cdcd453e28a0d836f19fedbf17e"}
Jun 02 16:08:35.209981 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:35.209928    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-s989l/must-gather-mmjdh" podStartSLOduration=1.489899377 podStartE2EDuration="2.209913034s" podCreationTimestamp="2026-06-02 16:08:33 +0000 UTC" firstStartedPulling="2026-06-02 16:08:34.221850654 +0000 UTC m=+1362.924314151" lastFinishedPulling="2026-06-02 16:08:34.941864304 +0000 UTC m=+1363.644327808" observedRunningTime="2026-06-02 16:08:35.207730527 +0000 UTC m=+1363.910194047" watchObservedRunningTime="2026-06-02 16:08:35.209913034 +0000 UTC m=+1363.912376553"
Jun 02 16:08:36.444043 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:36.444007    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rbwl6_44384f36-6a8f-47b3-b329-083afe044047/global-pull-secret-syncer/0.log"
Jun 02 16:08:36.604387 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:36.604349    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mrkbz_c81f2766-91c6-479a-bfcb-fc1144278194/konnectivity-agent/0.log"
Jun 02 16:08:36.700844 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:36.700763    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-246.ec2.internal_36fed07468a4db314a589b059de2ede5/haproxy/0.log"
Jun 02 16:08:38.610733 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:38.610689    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-9vsmp/must-gather-k2bmf"]
Jun 02 16:08:38.611741 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:38.611699    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-must-gather-9vsmp/must-gather-k2bmf" podUID="a17d3af4-c896-45ab-b6a3-9489ea37194c" containerName="copy" containerID="cri-o://0da9b5a02004af4417cd125547cc8a36d722f2772bfd5a455ffc4afe8c44e0bc" gracePeriod=2
Jun 02 16:08:38.613915 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:38.613858    2578 status_manager.go:895] "Failed to get status for pod" podUID="a17d3af4-c896-45ab-b6a3-9489ea37194c" pod="openshift-must-gather-9vsmp/must-gather-k2bmf" err="pods \"must-gather-k2bmf\" is forbidden: User \"system:node:ip-10-0-140-246.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-9vsmp\": no relationship found between node 'ip-10-0-140-246.ec2.internal' and this object"
Jun 02 16:08:38.614916 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:38.614870    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-9vsmp/must-gather-k2bmf"]
Jun 02 16:08:38.984114 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:38.977314    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-9vsmp_must-gather-k2bmf_a17d3af4-c896-45ab-b6a3-9489ea37194c/copy/0.log"
Jun 02 16:08:38.984114 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:38.977751    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9vsmp/must-gather-k2bmf"
Jun 02 16:08:38.984114 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:38.979774    2578 status_manager.go:895] "Failed to get status for pod" podUID="a17d3af4-c896-45ab-b6a3-9489ea37194c" pod="openshift-must-gather-9vsmp/must-gather-k2bmf" err="pods \"must-gather-k2bmf\" is forbidden: User \"system:node:ip-10-0-140-246.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-9vsmp\": no relationship found between node 'ip-10-0-140-246.ec2.internal' and this object"
Jun 02 16:08:39.147151 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:39.147108    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a17d3af4-c896-45ab-b6a3-9489ea37194c-must-gather-output\") pod \"a17d3af4-c896-45ab-b6a3-9489ea37194c\" (UID: \"a17d3af4-c896-45ab-b6a3-9489ea37194c\") "
Jun 02 16:08:39.148506 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:39.148373    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a17d3af4-c896-45ab-b6a3-9489ea37194c-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "a17d3af4-c896-45ab-b6a3-9489ea37194c" (UID: "a17d3af4-c896-45ab-b6a3-9489ea37194c"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:08:39.148671 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:39.148549    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xj48\" (UniqueName: \"kubernetes.io/projected/a17d3af4-c896-45ab-b6a3-9489ea37194c-kube-api-access-6xj48\") pod \"a17d3af4-c896-45ab-b6a3-9489ea37194c\" (UID: \"a17d3af4-c896-45ab-b6a3-9489ea37194c\") "
Jun 02 16:08:39.149369 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:39.149331    2578 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a17d3af4-c896-45ab-b6a3-9489ea37194c-must-gather-output\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 16:08:39.156356 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:39.156315    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a17d3af4-c896-45ab-b6a3-9489ea37194c-kube-api-access-6xj48" (OuterVolumeSpecName: "kube-api-access-6xj48") pod "a17d3af4-c896-45ab-b6a3-9489ea37194c" (UID: "a17d3af4-c896-45ab-b6a3-9489ea37194c"). InnerVolumeSpecName "kube-api-access-6xj48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:08:39.217764 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:39.217631    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-9vsmp_must-gather-k2bmf_a17d3af4-c896-45ab-b6a3-9489ea37194c/copy/0.log"
Jun 02 16:08:39.219598 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:39.218396    2578 generic.go:358] "Generic (PLEG): container finished" podID="a17d3af4-c896-45ab-b6a3-9489ea37194c" containerID="0da9b5a02004af4417cd125547cc8a36d722f2772bfd5a455ffc4afe8c44e0bc" exitCode=143
Jun 02 16:08:39.219598 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:39.218490    2578 scope.go:117] "RemoveContainer" containerID="0da9b5a02004af4417cd125547cc8a36d722f2772bfd5a455ffc4afe8c44e0bc"
Jun 02 16:08:39.219598 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:39.218629    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9vsmp/must-gather-k2bmf"
Jun 02 16:08:39.233458 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:39.233427    2578 status_manager.go:895] "Failed to get status for pod" podUID="a17d3af4-c896-45ab-b6a3-9489ea37194c" pod="openshift-must-gather-9vsmp/must-gather-k2bmf" err="pods \"must-gather-k2bmf\" is forbidden: User \"system:node:ip-10-0-140-246.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-9vsmp\": no relationship found between node 'ip-10-0-140-246.ec2.internal' and this object"
Jun 02 16:08:39.241092 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:39.241050    2578 status_manager.go:895] "Failed to get status for pod" podUID="a17d3af4-c896-45ab-b6a3-9489ea37194c" pod="openshift-must-gather-9vsmp/must-gather-k2bmf" err="pods \"must-gather-k2bmf\" is forbidden: User \"system:node:ip-10-0-140-246.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-9vsmp\": no relationship found between node 'ip-10-0-140-246.ec2.internal' and this object"
Jun 02 16:08:39.245822 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:39.245804    2578 scope.go:117] "RemoveContainer" containerID="094270c2b3a3718e909b7520ea1cfc74d56c283221f348df6ab287b423df362a"
Jun 02 16:08:39.250112 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:39.249971    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xj48\" (UniqueName: \"kubernetes.io/projected/a17d3af4-c896-45ab-b6a3-9489ea37194c-kube-api-access-6xj48\") on node \"ip-10-0-140-246.ec2.internal\" DevicePath \"\""
Jun 02 16:08:39.269728 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:39.269702    2578 scope.go:117] "RemoveContainer" containerID="0da9b5a02004af4417cd125547cc8a36d722f2772bfd5a455ffc4afe8c44e0bc"
Jun 02 16:08:39.270125 ip-10-0-140-246 kubenswrapper[2578]: E0602 16:08:39.270099    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0da9b5a02004af4417cd125547cc8a36d722f2772bfd5a455ffc4afe8c44e0bc\": container with ID starting with 0da9b5a02004af4417cd125547cc8a36d722f2772bfd5a455ffc4afe8c44e0bc not found: ID does not exist" containerID="0da9b5a02004af4417cd125547cc8a36d722f2772bfd5a455ffc4afe8c44e0bc"
Jun 02 16:08:39.270244 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:39.270138    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0da9b5a02004af4417cd125547cc8a36d722f2772bfd5a455ffc4afe8c44e0bc"} err="failed to get container status \"0da9b5a02004af4417cd125547cc8a36d722f2772bfd5a455ffc4afe8c44e0bc\": rpc error: code = NotFound desc = could not find container \"0da9b5a02004af4417cd125547cc8a36d722f2772bfd5a455ffc4afe8c44e0bc\": container with ID starting with 0da9b5a02004af4417cd125547cc8a36d722f2772bfd5a455ffc4afe8c44e0bc not found: ID does not exist"
Jun 02 16:08:39.270244 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:39.270163    2578 scope.go:117] "RemoveContainer" containerID="094270c2b3a3718e909b7520ea1cfc74d56c283221f348df6ab287b423df362a"
Jun 02 16:08:39.270414 ip-10-0-140-246 kubenswrapper[2578]: E0602 16:08:39.270390    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"094270c2b3a3718e909b7520ea1cfc74d56c283221f348df6ab287b423df362a\": container with ID starting with 094270c2b3a3718e909b7520ea1cfc74d56c283221f348df6ab287b423df362a not found: ID does not exist" containerID="094270c2b3a3718e909b7520ea1cfc74d56c283221f348df6ab287b423df362a"
Jun 02 16:08:39.270480 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:39.270427    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"094270c2b3a3718e909b7520ea1cfc74d56c283221f348df6ab287b423df362a"} err="failed to get container status \"094270c2b3a3718e909b7520ea1cfc74d56c283221f348df6ab287b423df362a\": rpc error: code = NotFound desc = could not find container \"094270c2b3a3718e909b7520ea1cfc74d56c283221f348df6ab287b423df362a\": container with ID starting with 094270c2b3a3718e909b7520ea1cfc74d56c283221f348df6ab287b423df362a not found: ID does not exist"
Jun 02 16:08:39.911419 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:39.910966    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a17d3af4-c896-45ab-b6a3-9489ea37194c" path="/var/lib/kubelet/pods/a17d3af4-c896-45ab-b6a3-9489ea37194c/volumes"
Jun 02 16:08:40.087283 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:40.087220    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_20954fac-f654-426a-9b6a-007786d3d7b9/alertmanager/0.log"
Jun 02 16:08:40.123056 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:40.123031    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_20954fac-f654-426a-9b6a-007786d3d7b9/config-reloader/0.log"
Jun 02 16:08:40.153075 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:40.153041    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_20954fac-f654-426a-9b6a-007786d3d7b9/kube-rbac-proxy-web/0.log"
Jun 02 16:08:40.185980 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:40.185907    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_20954fac-f654-426a-9b6a-007786d3d7b9/kube-rbac-proxy/0.log"
Jun 02 16:08:40.217168 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:40.217138    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_20954fac-f654-426a-9b6a-007786d3d7b9/kube-rbac-proxy-metric/0.log"
Jun 02 16:08:40.245559 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:40.245530    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_20954fac-f654-426a-9b6a-007786d3d7b9/prom-label-proxy/0.log"
Jun 02 16:08:40.287517 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:40.287488    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_20954fac-f654-426a-9b6a-007786d3d7b9/init-config-reloader/0.log"
Jun 02 16:08:40.367661 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:40.367637    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-8469c486bb-9b5kt_c31d1126-4813-4f5d-9f89-23bac44e0919/kube-state-metrics/0.log"
Jun 02 16:08:40.395099 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:40.395069    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-8469c486bb-9b5kt_c31d1126-4813-4f5d-9f89-23bac44e0919/kube-rbac-proxy-main/0.log"
Jun 02 16:08:40.424530 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:40.424489    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-8469c486bb-9b5kt_c31d1126-4813-4f5d-9f89-23bac44e0919/kube-rbac-proxy-self/0.log"
Jun 02 16:08:40.747007 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:40.746976    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j5z2h_b64cb704-5a89-4375-9a14-d6131bf4a0fb/node-exporter/0.log"
Jun 02 16:08:40.772928 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:40.772893    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j5z2h_b64cb704-5a89-4375-9a14-d6131bf4a0fb/kube-rbac-proxy/0.log"
Jun 02 16:08:40.803604 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:40.803579    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j5z2h_b64cb704-5a89-4375-9a14-d6131bf4a0fb/init-textfile/0.log"
Jun 02 16:08:40.841752 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:40.841712    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-85658796db-nxv6k_a074a268-b658-430e-83f0-d5e98f856d03/kube-rbac-proxy-main/0.log"
Jun 02 16:08:40.872441 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:40.872379    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-85658796db-nxv6k_a074a268-b658-430e-83f0-d5e98f856d03/kube-rbac-proxy-self/0.log"
Jun 02 16:08:40.907428 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:40.907351    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-85658796db-nxv6k_a074a268-b658-430e-83f0-d5e98f856d03/openshift-state-metrics/0.log"
Jun 02 16:08:40.952193 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:40.952163    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cdb7335b-4f53-4632-8403-db5825ff15c6/prometheus/0.log"
Jun 02 16:08:40.981462 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:40.981384    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cdb7335b-4f53-4632-8403-db5825ff15c6/config-reloader/0.log"
Jun 02 16:08:41.012381 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:41.012233    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cdb7335b-4f53-4632-8403-db5825ff15c6/thanos-sidecar/0.log"
Jun 02 16:08:41.045770 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:41.045719    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cdb7335b-4f53-4632-8403-db5825ff15c6/kube-rbac-proxy-web/0.log"
Jun 02 16:08:41.088471 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:41.088447    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cdb7335b-4f53-4632-8403-db5825ff15c6/kube-rbac-proxy/0.log"
Jun 02 16:08:41.119925 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:41.119892    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cdb7335b-4f53-4632-8403-db5825ff15c6/kube-rbac-proxy-thanos/0.log"
Jun 02 16:08:41.185583 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:41.185540    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cdb7335b-4f53-4632-8403-db5825ff15c6/init-config-reloader/0.log"
Jun 02 16:08:41.332269 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:41.332235    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-59b7f8756b-vtgbm_8242da94-694d-499d-8c71-b3a479b8c726/prometheus-operator-admission-webhook/0.log"
Jun 02 16:08:41.470371 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:41.470334    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86d46bcd55-5mltm_c6a52535-c632-41ab-9a09-4f4494e7c2d8/thanos-query/0.log"
Jun 02 16:08:41.499782 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:41.499692    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86d46bcd55-5mltm_c6a52535-c632-41ab-9a09-4f4494e7c2d8/kube-rbac-proxy-web/0.log"
Jun 02 16:08:41.527575 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:41.527538    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86d46bcd55-5mltm_c6a52535-c632-41ab-9a09-4f4494e7c2d8/kube-rbac-proxy/0.log"
Jun 02 16:08:41.556123 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:41.556095    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86d46bcd55-5mltm_c6a52535-c632-41ab-9a09-4f4494e7c2d8/prom-label-proxy/0.log"
Jun 02 16:08:41.585828 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:41.585806    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86d46bcd55-5mltm_c6a52535-c632-41ab-9a09-4f4494e7c2d8/kube-rbac-proxy-rules/0.log"
Jun 02 16:08:41.640206 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:41.640182    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86d46bcd55-5mltm_c6a52535-c632-41ab-9a09-4f4494e7c2d8/kube-rbac-proxy-metrics/0.log"
Jun 02 16:08:43.619871 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:43.619837    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s989l/perf-node-gather-daemonset-62jqc"]
Jun 02 16:08:43.620473 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:43.620327    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a17d3af4-c896-45ab-b6a3-9489ea37194c" containerName="copy"
Jun 02 16:08:43.620473 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:43.620348    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a17d3af4-c896-45ab-b6a3-9489ea37194c" containerName="copy"
Jun 02 16:08:43.620473 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:43.620367    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a17d3af4-c896-45ab-b6a3-9489ea37194c" containerName="gather"
Jun 02 16:08:43.620473 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:43.620376    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a17d3af4-c896-45ab-b6a3-9489ea37194c" containerName="gather"
Jun 02 16:08:43.620473 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:43.620468    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="a17d3af4-c896-45ab-b6a3-9489ea37194c" containerName="copy"
Jun 02 16:08:43.620746 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:43.620481    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="a17d3af4-c896-45ab-b6a3-9489ea37194c" containerName="gather"
Jun 02 16:08:43.624773 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:43.624752    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s989l/perf-node-gather-daemonset-62jqc"
Jun 02 16:08:43.630093 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:43.630067    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s989l/perf-node-gather-daemonset-62jqc"]
Jun 02 16:08:43.689770 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:43.689731    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9fea8557-c07b-4ee1-abbf-f5e7defaa3c2-lib-modules\") pod \"perf-node-gather-daemonset-62jqc\" (UID: \"9fea8557-c07b-4ee1-abbf-f5e7defaa3c2\") " pod="openshift-must-gather-s989l/perf-node-gather-daemonset-62jqc"
Jun 02 16:08:43.689959 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:43.689869    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr89c\" (UniqueName: \"kubernetes.io/projected/9fea8557-c07b-4ee1-abbf-f5e7defaa3c2-kube-api-access-nr89c\") pod \"perf-node-gather-daemonset-62jqc\" (UID: \"9fea8557-c07b-4ee1-abbf-f5e7defaa3c2\") " pod="openshift-must-gather-s989l/perf-node-gather-daemonset-62jqc"
Jun 02 16:08:43.689959 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:43.689930    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9fea8557-c07b-4ee1-abbf-f5e7defaa3c2-proc\") pod \"perf-node-gather-daemonset-62jqc\" (UID: \"9fea8557-c07b-4ee1-abbf-f5e7defaa3c2\") " pod="openshift-must-gather-s989l/perf-node-gather-daemonset-62jqc"
Jun 02 16:08:43.690103 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:43.689975    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9fea8557-c07b-4ee1-abbf-f5e7defaa3c2-podres\") pod \"perf-node-gather-daemonset-62jqc\" (UID: \"9fea8557-c07b-4ee1-abbf-f5e7defaa3c2\") " pod="openshift-must-gather-s989l/perf-node-gather-daemonset-62jqc"
Jun 02 16:08:43.690103 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:43.690017    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9fea8557-c07b-4ee1-abbf-f5e7defaa3c2-sys\") pod \"perf-node-gather-daemonset-62jqc\" (UID: \"9fea8557-c07b-4ee1-abbf-f5e7defaa3c2\") " pod="openshift-must-gather-s989l/perf-node-gather-daemonset-62jqc"
Jun 02 16:08:43.790638 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:43.790594    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nr89c\" (UniqueName: \"kubernetes.io/projected/9fea8557-c07b-4ee1-abbf-f5e7defaa3c2-kube-api-access-nr89c\") pod \"perf-node-gather-daemonset-62jqc\" (UID: \"9fea8557-c07b-4ee1-abbf-f5e7defaa3c2\") " pod="openshift-must-gather-s989l/perf-node-gather-daemonset-62jqc"
Jun 02 16:08:43.790843 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:43.790666    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9fea8557-c07b-4ee1-abbf-f5e7defaa3c2-proc\") pod \"perf-node-gather-daemonset-62jqc\" (UID: \"9fea8557-c07b-4ee1-abbf-f5e7defaa3c2\") " pod="openshift-must-gather-s989l/perf-node-gather-daemonset-62jqc"
Jun 02 16:08:43.790843 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:43.790710    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9fea8557-c07b-4ee1-abbf-f5e7defaa3c2-podres\") pod \"perf-node-gather-daemonset-62jqc\" (UID: \"9fea8557-c07b-4ee1-abbf-f5e7defaa3c2\") " pod="openshift-must-gather-s989l/perf-node-gather-daemonset-62jqc"
Jun 02 16:08:43.790843 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:43.790737    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9fea8557-c07b-4ee1-abbf-f5e7defaa3c2-sys\") pod \"perf-node-gather-daemonset-62jqc\" (UID: \"9fea8557-c07b-4ee1-abbf-f5e7defaa3c2\") " pod="openshift-must-gather-s989l/perf-node-gather-daemonset-62jqc"
Jun 02 16:08:43.790843 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:43.790785    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9fea8557-c07b-4ee1-abbf-f5e7defaa3c2-lib-modules\") pod \"perf-node-gather-daemonset-62jqc\" (UID: \"9fea8557-c07b-4ee1-abbf-f5e7defaa3c2\") " pod="openshift-must-gather-s989l/perf-node-gather-daemonset-62jqc"
Jun 02 16:08:43.791110 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:43.790958    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9fea8557-c07b-4ee1-abbf-f5e7defaa3c2-lib-modules\") pod \"perf-node-gather-daemonset-62jqc\" (UID: \"9fea8557-c07b-4ee1-abbf-f5e7defaa3c2\") " pod="openshift-must-gather-s989l/perf-node-gather-daemonset-62jqc"
Jun 02 16:08:43.791200 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:43.791180    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9fea8557-c07b-4ee1-abbf-f5e7defaa3c2-sys\") pod \"perf-node-gather-daemonset-62jqc\" (UID: \"9fea8557-c07b-4ee1-abbf-f5e7defaa3c2\") " pod="openshift-must-gather-s989l/perf-node-gather-daemonset-62jqc"
Jun 02 16:08:43.791200 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:43.791191    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9fea8557-c07b-4ee1-abbf-f5e7defaa3c2-podres\") pod \"perf-node-gather-daemonset-62jqc\" (UID: \"9fea8557-c07b-4ee1-abbf-f5e7defaa3c2\") " pod="openshift-must-gather-s989l/perf-node-gather-daemonset-62jqc"
Jun 02 16:08:43.791303 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:43.791249    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9fea8557-c07b-4ee1-abbf-f5e7defaa3c2-proc\") pod \"perf-node-gather-daemonset-62jqc\" (UID: \"9fea8557-c07b-4ee1-abbf-f5e7defaa3c2\") " pod="openshift-must-gather-s989l/perf-node-gather-daemonset-62jqc"
Jun 02 16:08:43.800090 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:43.800064    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr89c\" (UniqueName: \"kubernetes.io/projected/9fea8557-c07b-4ee1-abbf-f5e7defaa3c2-kube-api-access-nr89c\") pod \"perf-node-gather-daemonset-62jqc\" (UID: \"9fea8557-c07b-4ee1-abbf-f5e7defaa3c2\") " pod="openshift-must-gather-s989l/perf-node-gather-daemonset-62jqc"
Jun 02 16:08:43.935765 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:43.935732    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s989l/perf-node-gather-daemonset-62jqc"
Jun 02 16:08:44.068274 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:44.068243    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s989l/perf-node-gather-daemonset-62jqc"]
Jun 02 16:08:44.071894 ip-10-0-140-246 kubenswrapper[2578]: W0602 16:08:44.071868    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9fea8557_c07b_4ee1_abbf_f5e7defaa3c2.slice/crio-e78798b71599a0b8a35dc8284f4edad1905a6db56aaf0972036207e9b9c88154 WatchSource:0}: Error finding container e78798b71599a0b8a35dc8284f4edad1905a6db56aaf0972036207e9b9c88154: Status 404 returned error can't find the container with id e78798b71599a0b8a35dc8284f4edad1905a6db56aaf0972036207e9b9c88154
Jun 02 16:08:44.237549 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:44.237466    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s989l/perf-node-gather-daemonset-62jqc" event={"ID":"9fea8557-c07b-4ee1-abbf-f5e7defaa3c2","Type":"ContainerStarted","Data":"4892ad9dc619ee511b3e6b51826438056f5214cba4728100100d69198a4c69c3"}
Jun 02 16:08:44.237549 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:44.237500    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s989l/perf-node-gather-daemonset-62jqc" event={"ID":"9fea8557-c07b-4ee1-abbf-f5e7defaa3c2","Type":"ContainerStarted","Data":"e78798b71599a0b8a35dc8284f4edad1905a6db56aaf0972036207e9b9c88154"}
Jun 02 16:08:44.237745 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:44.237565    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-s989l/perf-node-gather-daemonset-62jqc"
Jun 02 16:08:44.253433 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:44.253385    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-s989l/perf-node-gather-daemonset-62jqc" podStartSLOduration=1.25336933 podStartE2EDuration="1.25336933s" podCreationTimestamp="2026-06-02 16:08:43 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:08:44.253027756 +0000 UTC m=+1372.955491272" watchObservedRunningTime="2026-06-02 16:08:44.25336933 +0000 UTC m=+1372.955832852"
Jun 02 16:08:44.619982 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:44.619902    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5pxm2_2d910de6-b7ee-49ae-b0f8-2948726ae27f/dns/0.log"
Jun 02 16:08:44.650593 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:44.650572    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5pxm2_2d910de6-b7ee-49ae-b0f8-2948726ae27f/kube-rbac-proxy/0.log"
Jun 02 16:08:44.739846 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:44.739820    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4tfzk_97fe90a6-eb72-4661-b376-fba7f85330e3/dns-node-resolver/0.log"
Jun 02 16:08:45.213616 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:45.213588    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-c8f78dbd9-4qk6x_ff4aa919-595a-4e50-8a64-0e182f6c8e13/registry/0.log"
Jun 02 16:08:45.235376 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:45.235350    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-csg6c_d15a2da2-4507-43f8-913a-0414ef633f40/node-ca/0.log"
Jun 02 16:08:46.391322 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:46.391291    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-l7cml_8601b504-c1ef-44de-853b-9d130969f742/serve-healthcheck-canary/0.log"
Jun 02 16:08:46.898937 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:46.898912    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pm7zw_5167c18f-db19-477e-a673-6b8a6795bc6e/kube-rbac-proxy/0.log"
Jun 02 16:08:46.924503 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:46.924478    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pm7zw_5167c18f-db19-477e-a673-6b8a6795bc6e/exporter/0.log"
Jun 02 16:08:46.950851 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:46.950786    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pm7zw_5167c18f-db19-477e-a673-6b8a6795bc6e/extractor/0.log"
Jun 02 16:08:49.249379 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:49.249350    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-qb5bg_0256bb0d-e0cf-483e-8e40-6e9ca793f7ec/s3-init/0.log"
Jun 02 16:08:50.250255 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:50.250229    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-s989l/perf-node-gather-daemonset-62jqc"
Jun 02 16:08:55.193068 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:55.193039    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wr9zh_4b0f9d3d-2024-423c-9afd-dc5094e82d2f/kube-multus-additional-cni-plugins/0.log"
Jun 02 16:08:55.218272 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:55.218244    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wr9zh_4b0f9d3d-2024-423c-9afd-dc5094e82d2f/egress-router-binary-copy/0.log"
Jun 02 16:08:55.242811 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:55.242790    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wr9zh_4b0f9d3d-2024-423c-9afd-dc5094e82d2f/cni-plugins/0.log"
Jun 02 16:08:55.266009 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:55.265979    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wr9zh_4b0f9d3d-2024-423c-9afd-dc5094e82d2f/bond-cni-plugin/0.log"
Jun 02 16:08:55.289898 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:55.289874    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wr9zh_4b0f9d3d-2024-423c-9afd-dc5094e82d2f/routeoverride-cni/0.log"
Jun 02 16:08:55.315955 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:55.315937    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wr9zh_4b0f9d3d-2024-423c-9afd-dc5094e82d2f/whereabouts-cni-bincopy/0.log"
Jun 02 16:08:55.342185 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:55.342159    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wr9zh_4b0f9d3d-2024-423c-9afd-dc5094e82d2f/whereabouts-cni/0.log"
Jun 02 16:08:55.410874 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:55.410850    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-nnqmn_22d16389-ab0a-4b82-97ae-e548268c773e/kube-multus/0.log"
Jun 02 16:08:55.516854 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:55.516792    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8bxt7_c8a1a07e-d72f-407a-a361-9ac7ed55178f/network-metrics-daemon/0.log"
Jun 02 16:08:55.542430 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:55.542407    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8bxt7_c8a1a07e-d72f-407a-a361-9ac7ed55178f/kube-rbac-proxy/0.log"
Jun 02 16:08:56.830241 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:56.830210    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l5gxc_becbbb00-675f-4263-a999-884ce0d688b2/ovn-controller/0.log"
Jun 02 16:08:56.896393 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:56.896366    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l5gxc_becbbb00-675f-4263-a999-884ce0d688b2/ovn-acl-logging/0.log"
Jun 02 16:08:56.936789 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:56.936763    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l5gxc_becbbb00-675f-4263-a999-884ce0d688b2/kube-rbac-proxy-node/0.log"
Jun 02 16:08:56.975679 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:56.975656    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l5gxc_becbbb00-675f-4263-a999-884ce0d688b2/kube-rbac-proxy-ovn-metrics/0.log"
Jun 02 16:08:57.020065 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:57.020046    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l5gxc_becbbb00-675f-4263-a999-884ce0d688b2/northd/0.log"
Jun 02 16:08:57.057463 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:57.057438    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l5gxc_becbbb00-675f-4263-a999-884ce0d688b2/nbdb/0.log"
Jun 02 16:08:57.091823 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:57.091761    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l5gxc_becbbb00-675f-4263-a999-884ce0d688b2/sbdb/0.log"
Jun 02 16:08:57.213911 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:57.213880    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l5gxc_becbbb00-675f-4263-a999-884ce0d688b2/ovnkube-controller/0.log"
Jun 02 16:08:58.766092 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:58.766057    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-smdm5_a664bf1a-e1bf-4107-9ebb-076ddd6b6234/network-check-target-container/0.log"
Jun 02 16:08:59.726450 ip-10-0-140-246 kubenswrapper[2578]: I0602 16:08:59.726419    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-2pmfl_47c31aee-a2ce-4b2a-808d-eca33a9c31fc/iptables-alerter/0.log"