Jun 02 15:46:31.358725 ip-10-0-129-156 systemd[1]: Starting Kubernetes Kubelet...
Jun 02 15:46:31.782640 ip-10-0-129-156 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 02 15:46:31.782640 ip-10-0-129-156 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jun 02 15:46:31.782640 ip-10-0-129-156 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 02 15:46:31.782640 ip-10-0-129-156 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jun 02 15:46:31.782640 ip-10-0-129-156 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 02 15:46:31.784617 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.784523    2579 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jun 02 15:46:31.792590 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792561    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 02 15:46:31.792590 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792582    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 02 15:46:31.792590 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792585    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 02 15:46:31.792590 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792589    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 02 15:46:31.792590 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792592    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 02 15:46:31.792590 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792595    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 02 15:46:31.792590 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792598    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 02 15:46:31.792590 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792601    2579 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Jun 02 15:46:31.792590 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792603    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 02 15:46:31.792907 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792606    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 02 15:46:31.792907 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792609    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 02 15:46:31.792907 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792612    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 02 15:46:31.792907 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792615    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 02 15:46:31.792907 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792617    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 02 15:46:31.792907 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792620    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 02 15:46:31.792907 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792622    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 02 15:46:31.792907 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792626    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 02 15:46:31.792907 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792628    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Jun 02 15:46:31.792907 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792630    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 02 15:46:31.792907 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792633    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 02 15:46:31.792907 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792635    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 02 15:46:31.792907 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792638    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 02 15:46:31.792907 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792648    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 02 15:46:31.792907 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792650    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 02 15:46:31.792907 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792653    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 02 15:46:31.792907 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792655    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 02 15:46:31.792907 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792658    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 02 15:46:31.792907 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792660    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 02 15:46:31.792907 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792662    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 02 15:46:31.793440 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792665    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 02 15:46:31.793440 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792668    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 02 15:46:31.793440 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792670    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 02 15:46:31.793440 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792673    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 02 15:46:31.793440 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792675    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 02 15:46:31.793440 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792677    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 02 15:46:31.793440 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792680    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 02 15:46:31.793440 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792682    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 02 15:46:31.793440 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792685    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 02 15:46:31.793440 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792687    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 02 15:46:31.793440 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792690    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 02 15:46:31.793440 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792693    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 02 15:46:31.793440 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792696    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 02 15:46:31.793440 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792699    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 02 15:46:31.793440 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792701    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 02 15:46:31.793440 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792704    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 02 15:46:31.793440 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792707    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 02 15:46:31.793440 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792710    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 02 15:46:31.793440 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792713    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 02 15:46:31.793440 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792715    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 02 15:46:31.793959 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792718    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 02 15:46:31.793959 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792720    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 02 15:46:31.793959 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792723    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 02 15:46:31.793959 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792725    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Jun 02 15:46:31.793959 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792728    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 02 15:46:31.793959 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792730    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 02 15:46:31.793959 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792733    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 02 15:46:31.793959 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792735    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 02 15:46:31.793959 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792738    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Jun 02 15:46:31.793959 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792741    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 02 15:46:31.793959 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792743    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 02 15:46:31.793959 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792746    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 02 15:46:31.793959 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792749    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 02 15:46:31.793959 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792752    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 02 15:46:31.793959 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792755    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jun 02 15:46:31.793959 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792757    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 02 15:46:31.793959 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792760    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 02 15:46:31.793959 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792763    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 02 15:46:31.793959 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792767    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 02 15:46:31.794436 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792771    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 02 15:46:31.794436 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792774    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 02 15:46:31.794436 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792777    2579 feature_gate.go:328] unrecognized feature gate: Example
Jun 02 15:46:31.794436 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792780    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 02 15:46:31.794436 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792782    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 02 15:46:31.794436 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792785    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 02 15:46:31.794436 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792789    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 02 15:46:31.794436 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792797    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 02 15:46:31.794436 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792801    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 02 15:46:31.794436 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792804    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 02 15:46:31.794436 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792807    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 02 15:46:31.794436 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792810    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 02 15:46:31.794436 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792813    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 02 15:46:31.794436 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792815    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 02 15:46:31.794436 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792817    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 02 15:46:31.794436 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792820    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 02 15:46:31.794436 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792823    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 02 15:46:31.794436 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.792827    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 02 15:46:31.794436 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793236    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 02 15:46:31.794436 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793241    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 02 15:46:31.794919 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793243    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 02 15:46:31.794919 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793246    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 02 15:46:31.794919 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793249    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 02 15:46:31.794919 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793251    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 02 15:46:31.794919 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793254    2579 feature_gate.go:328] unrecognized feature gate: Example
Jun 02 15:46:31.794919 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793257    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 02 15:46:31.794919 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793259    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 02 15:46:31.794919 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793262    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 02 15:46:31.794919 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793264    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 02 15:46:31.794919 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793267    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 02 15:46:31.794919 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793271    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 02 15:46:31.794919 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793274    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 02 15:46:31.794919 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793277    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 02 15:46:31.794919 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793279    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 02 15:46:31.794919 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793282    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 02 15:46:31.794919 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793285    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 02 15:46:31.794919 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793288    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 02 15:46:31.794919 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793290    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 02 15:46:31.794919 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793292    2579 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Jun 02 15:46:31.794919 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793295    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 02 15:46:31.795392 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793297    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 02 15:46:31.795392 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793300    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Jun 02 15:46:31.795392 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793303    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 02 15:46:31.795392 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793305    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 02 15:46:31.795392 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793309    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 02 15:46:31.795392 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793313    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 02 15:46:31.795392 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793316    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 02 15:46:31.795392 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793319    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 02 15:46:31.795392 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793322    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 02 15:46:31.795392 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793324    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 02 15:46:31.795392 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793327    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 02 15:46:31.795392 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793329    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 02 15:46:31.795392 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793332    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 02 15:46:31.795392 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793334    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 02 15:46:31.795392 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793336    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 02 15:46:31.795392 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793340    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 02 15:46:31.795392 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793343    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Jun 02 15:46:31.795392 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793345    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 02 15:46:31.795392 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793348    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 02 15:46:31.795921 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793350    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jun 02 15:46:31.795921 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793352    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 02 15:46:31.795921 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793355    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 02 15:46:31.795921 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793358    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 02 15:46:31.795921 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793361    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 02 15:46:31.795921 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793363    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 02 15:46:31.795921 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793366    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 02 15:46:31.795921 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793369    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 02 15:46:31.795921 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793371    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 02 15:46:31.795921 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793374    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 02 15:46:31.795921 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793376    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 02 15:46:31.795921 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793378    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 02 15:46:31.795921 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793381    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 02 15:46:31.795921 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793384    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 02 15:46:31.795921 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793386    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 02 15:46:31.795921 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793388    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 02 15:46:31.795921 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793391    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 02 15:46:31.795921 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793394    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 02 15:46:31.795921 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793396    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 02 15:46:31.795921 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793415    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 02 15:46:31.796416 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793419    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 02 15:46:31.796416 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793423    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 02 15:46:31.796416 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793426    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 02 15:46:31.796416 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793428    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 02 15:46:31.796416 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793431    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 02 15:46:31.796416 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793433    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 02 15:46:31.796416 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793436    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 02 15:46:31.796416 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793438    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 02 15:46:31.796416 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793441    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 02 15:46:31.796416 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793444    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 02 15:46:31.796416 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793446    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 02 15:46:31.796416 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793449    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 02 15:46:31.796416 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793453    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 02 15:46:31.796416 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793456    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 02 15:46:31.796416 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793459    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 02 15:46:31.796416 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793462    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 02 15:46:31.796416 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793464    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 02 15:46:31.796416 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793467    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 02 15:46:31.796416 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793470    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Jun 02 15:46:31.796416 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793473    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 02 15:46:31.796902 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793476    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 02 15:46:31.796902 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793478    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 02 15:46:31.796902 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793481    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 02 15:46:31.796902 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793484    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 02 15:46:31.796902 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.793487    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 02 15:46:31.796902 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795003    2579 flags.go:64] FLAG: --address="0.0.0.0"
Jun 02 15:46:31.796902 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795013    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jun 02 15:46:31.796902 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795021    2579 flags.go:64] FLAG: --anonymous-auth="true"
Jun 02 15:46:31.796902 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795026    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jun 02 15:46:31.796902 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795031    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Jun 02 15:46:31.796902 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795034    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jun 02 15:46:31.796902 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795039    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jun 02 15:46:31.796902 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795044    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jun 02 15:46:31.796902 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795047    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jun 02 15:46:31.796902 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795050    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jun 02 15:46:31.796902 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795053    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jun 02 15:46:31.796902 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795056    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jun 02 15:46:31.796902 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795060    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jun 02 15:46:31.796902 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795063    2579 flags.go:64] FLAG: --cgroup-root=""
Jun 02 15:46:31.796902 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795065    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Jun 02 15:46:31.796902 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795069    2579 flags.go:64] FLAG: --client-ca-file=""
Jun 02 15:46:31.796902 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795071    2579 flags.go:64] FLAG: --cloud-config=""
Jun 02 15:46:31.796902 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795074    2579 flags.go:64] FLAG: --cloud-provider="external"
Jun 02 15:46:31.796902 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795077    2579 flags.go:64] FLAG: --cluster-dns="[]"
Jun 02 15:46:31.797506 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795081    2579 flags.go:64] FLAG: --cluster-domain=""
Jun 02 15:46:31.797506 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795083    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jun 02 15:46:31.797506 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795087    2579 flags.go:64] FLAG: --config-dir=""
Jun 02 15:46:31.797506 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795090    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jun 02 15:46:31.797506 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795093    2579 flags.go:64] FLAG: --container-log-max-files="5"
Jun 02 15:46:31.797506 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795096    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jun 02 15:46:31.797506 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795100    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jun 02 15:46:31.797506 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795103    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jun 02 15:46:31.797506 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795106    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jun 02 15:46:31.797506 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795109    2579 flags.go:64] FLAG: --contention-profiling="false"
Jun 02 15:46:31.797506 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795113    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jun 02 15:46:31.797506 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795116    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jun 02 15:46:31.797506 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795119    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Jun 02 15:46:31.797506 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795122    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jun 02 15:46:31.797506 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795126    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jun 02 15:46:31.797506 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795129    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jun 02 15:46:31.797506 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795132    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jun 02 15:46:31.797506 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795134    2579 flags.go:64] FLAG: --enable-load-reader="false"
Jun 02 15:46:31.797506 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795138    2579 flags.go:64] FLAG: --enable-server="true"
Jun 02 15:46:31.797506 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795141    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jun 02 15:46:31.797506 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795146    2579 flags.go:64] FLAG: --event-burst="100"
Jun 02 15:46:31.797506 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795148    2579 flags.go:64] FLAG: --event-qps="50"
Jun 02 15:46:31.797506 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795152    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jun 02 15:46:31.797506 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795155    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jun 02 15:46:31.797506 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795158    2579 flags.go:64] FLAG: --eviction-hard=""
Jun 02 15:46:31.798112 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795162    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jun 02 15:46:31.798112 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795164    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jun 02 15:46:31.798112 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795168    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jun 02 15:46:31.798112 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795171    2579 flags.go:64] FLAG: --eviction-soft=""
Jun 02 15:46:31.798112 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795173    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jun 02 15:46:31.798112 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795176    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jun 02 15:46:31.798112 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795179    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jun 02 15:46:31.798112 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795182    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Jun 02 15:46:31.798112 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795186    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Jun 02 15:46:31.798112 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795189    2579 flags.go:64] FLAG: --fail-swap-on="true"
Jun 02 15:46:31.798112 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795192    2579 flags.go:64] FLAG: --feature-gates=""
Jun 02 15:46:31.798112 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795196    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Jun 02 15:46:31.798112 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795199    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jun 02 15:46:31.798112 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795202    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jun 02 15:46:31.798112 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795206    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jun 02 15:46:31.798112 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795209    2579 flags.go:64] FLAG: --healthz-port="10248"
Jun 02 15:46:31.798112 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795212    2579 flags.go:64] FLAG: --help="false"
Jun 02 15:46:31.798112 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795215    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-129-156.ec2.internal"
Jun 02 15:46:31.798112 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795218    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Jun 02 15:46:31.798112 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795221    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Jun 02 15:46:31.798112 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795223    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jun 02 15:46:31.798112 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795227    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jun 02 15:46:31.798112 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795230    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jun 02 15:46:31.798683 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795233    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jun 02 15:46:31.798683 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795235    2579 flags.go:64] FLAG: --image-service-endpoint=""
Jun 02 15:46:31.798683 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795238    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jun 02 15:46:31.798683 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795242    2579 flags.go:64] FLAG: --kube-api-burst="100"
Jun 02 15:46:31.798683 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795245    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jun 02 15:46:31.798683 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795248    2579 flags.go:64] FLAG: --kube-api-qps="50"
Jun 02 15:46:31.798683 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795251    2579 flags.go:64] FLAG: --kube-reserved=""
Jun 02 15:46:31.798683 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795253    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jun 02 15:46:31.798683 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795256    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jun 02 15:46:31.798683 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795259    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Jun 02 15:46:31.798683 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795262    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jun 02 15:46:31.798683 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795264    2579 flags.go:64] FLAG: --lock-file=""
Jun 02 15:46:31.798683 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795267    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jun 02 15:46:31.798683 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795270    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Jun 02 15:46:31.798683 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795273    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jun 02 15:46:31.798683 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795278    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Jun 02 15:46:31.798683 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795281    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jun 02 15:46:31.798683 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795284    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Jun 02 15:46:31.798683 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795287    2579 flags.go:64] FLAG: --logging-format="text"
Jun 02 15:46:31.798683 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795290    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jun 02 15:46:31.798683 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795293    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jun 02 15:46:31.798683 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795296    2579 flags.go:64] FLAG: --manifest-url=""
Jun 02 15:46:31.798683 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795300    2579 flags.go:64] FLAG: --manifest-url-header=""
Jun 02 15:46:31.798683 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795304    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jun 02 15:46:31.798683 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795307    2579 flags.go:64] FLAG: --max-open-files="1000000"
Jun 02 15:46:31.799425 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795311    2579 flags.go:64] FLAG: --max-pods="110"
Jun 02 15:46:31.799425 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795314    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jun 02 15:46:31.799425 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795317    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jun 02 15:46:31.799425 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795320    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Jun 02 15:46:31.799425 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795323    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jun 02 15:46:31.799425 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795326    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jun 02 15:46:31.799425 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795329    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jun 02 15:46:31.799425 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795331    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jun 02 15:46:31.799425 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795339    2579 flags.go:64] FLAG: --node-status-max-images="50"
Jun 02 15:46:31.799425 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795342    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jun 02 15:46:31.799425 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795345    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Jun 02 15:46:31.799425 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795348    2579 flags.go:64] FLAG: --pod-cidr=""
Jun 02 15:46:31.799425 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795351    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f1083dc6abd9805f79797fa11c9accacc4a38caa073c94cb99097faffce05983"
Jun 02 15:46:31.799425 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795358    2579 flags.go:64] FLAG: --pod-manifest-path=""
Jun 02 15:46:31.799425 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795361    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Jun 02 15:46:31.799425 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795364    2579 flags.go:64] FLAG: --pods-per-core="0"
Jun 02 15:46:31.799425 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795367    2579 flags.go:64] FLAG: --port="10250"
Jun 02 15:46:31.799425 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795370    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jun 02 15:46:31.799425 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795373    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09b7662a42ba62ed4"
Jun 02 15:46:31.799425 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795376    2579 flags.go:64] FLAG: --qos-reserved=""
Jun 02 15:46:31.799425 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795378    2579 flags.go:64] FLAG: --read-only-port="10255"
Jun 02 15:46:31.799425 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795381    2579 flags.go:64] FLAG: --register-node="true"
Jun 02 15:46:31.799425 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795385    2579 flags.go:64] FLAG: --register-schedulable="true"
Jun 02 15:46:31.799425 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795387    2579 flags.go:64] FLAG: --register-with-taints=""
Jun 02 15:46:31.800019 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795391    2579 flags.go:64] FLAG: --registry-burst="10"
Jun 02 15:46:31.800019 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795394    2579 flags.go:64] FLAG: --registry-qps="5"
Jun 02 15:46:31.800019 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795397    2579 flags.go:64] FLAG: --reserved-cpus=""
Jun 02 15:46:31.800019 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795418    2579 flags.go:64] FLAG: --reserved-memory=""
Jun 02 15:46:31.800019 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795425    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jun 02 15:46:31.800019 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795428    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jun 02 15:46:31.800019 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795431    2579 flags.go:64] FLAG: --rotate-certificates="false"
Jun 02 15:46:31.800019 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795434    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Jun 02 15:46:31.800019 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795437    2579 flags.go:64] FLAG: --runonce="false"
Jun 02 15:46:31.800019 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795440    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jun 02 15:46:31.800019 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795443    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jun 02 15:46:31.800019 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795445    2579 flags.go:64] FLAG: --seccomp-default="false"
Jun 02 15:46:31.800019 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795448    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Jun 02 15:46:31.800019 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795451    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jun 02 15:46:31.800019 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795454    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jun 02 15:46:31.800019 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795457    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jun 02 15:46:31.800019 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795460    2579 flags.go:64] FLAG: --storage-driver-password="root"
Jun 02 15:46:31.800019 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795463    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Jun 02 15:46:31.800019 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795466    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Jun 02 15:46:31.800019 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795468    2579 flags.go:64] FLAG: --storage-driver-user="root"
Jun 02 15:46:31.800019 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795472    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jun 02 15:46:31.800019 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795475    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Jun 02 15:46:31.800019 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795481    2579 flags.go:64] FLAG: --system-cgroups=""
Jun 02 15:46:31.800019 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795484    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jun 02 15:46:31.800019 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795490    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Jun 02 15:46:31.800679 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795493    2579 flags.go:64] FLAG: --tls-cert-file=""
Jun 02 15:46:31.800679 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795496    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jun 02 15:46:31.800679 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795500    2579 flags.go:64] FLAG: --tls-min-version=""
Jun 02 15:46:31.800679 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795503    2579 flags.go:64] FLAG: --tls-private-key-file=""
Jun 02 15:46:31.800679 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795507    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Jun 02 15:46:31.800679 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795510    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Jun 02 15:46:31.800679 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795513    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Jun 02 15:46:31.800679 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795516    2579 flags.go:64] FLAG: --v="2"
Jun 02 15:46:31.800679 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795521    2579 flags.go:64] FLAG: --version="false"
Jun 02 15:46:31.800679 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795525    2579 flags.go:64] FLAG: --vmodule=""
Jun 02 15:46:31.800679 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795529    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jun 02 15:46:31.800679 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.795532    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jun 02 15:46:31.800679 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795638    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 02 15:46:31.800679 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795641    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 02 15:46:31.800679 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795644    2579 feature_gate.go:328] unrecognized feature gate: Example
Jun 02 15:46:31.800679 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795647    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 02 15:46:31.800679 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795650    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 02 15:46:31.800679 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795654    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 02 15:46:31.800679 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795656    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 02 15:46:31.800679 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795659    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 02 15:46:31.800679 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795663    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 02 15:46:31.800679 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795667    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 02 15:46:31.800679 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795670    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 02 15:46:31.801247 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795673    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 02 15:46:31.801247 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795676    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 02 15:46:31.801247 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795678    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 02 15:46:31.801247 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795681    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 02 15:46:31.801247 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795684    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 02 15:46:31.801247 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795687    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 02 15:46:31.801247 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795691    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 02 15:46:31.801247 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795693    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 02 15:46:31.801247 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795696    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 02 15:46:31.801247 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795699    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jun 02 15:46:31.801247 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795701    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 02 15:46:31.801247 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795704    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 02 15:46:31.801247 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795706    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 02 15:46:31.801247 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795710    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 02 15:46:31.801247 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795713    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 02 15:46:31.801247 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795715    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 02 15:46:31.801247 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795718    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 02 15:46:31.801247 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795720    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 02 15:46:31.801247 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795723    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 02 15:46:31.801247 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795725    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 02 15:46:31.801771 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795728    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Jun 02 15:46:31.801771 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795730    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 02 15:46:31.801771 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795733    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 02 15:46:31.801771 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795735    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 02 15:46:31.801771 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795738    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 02 15:46:31.801771 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795740    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 02 15:46:31.801771 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795743    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 02 15:46:31.801771 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795745    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 02 15:46:31.801771 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795748    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 02 15:46:31.801771 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795750    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 02 15:46:31.801771 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795753    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 02 15:46:31.801771 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795755    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 02 15:46:31.801771 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795758    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 02 15:46:31.801771 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795760    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 02 15:46:31.801771 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795762    2579 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Jun 02 15:46:31.801771 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795765    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 02 15:46:31.801771 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795767    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 02 15:46:31.801771 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795769    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 02 15:46:31.801771 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795775    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 02 15:46:31.801771 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795777    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 02 15:46:31.802250 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795780    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 02 15:46:31.802250 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795782    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 02 15:46:31.802250 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795784    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 02 15:46:31.802250 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795787    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 02 15:46:31.802250 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795789    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 02 15:46:31.802250 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795793    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 02 15:46:31.802250 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795796    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 02 15:46:31.802250 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795798    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 02 15:46:31.802250 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795800    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 02 15:46:31.802250 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795803    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 02 15:46:31.802250 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795805    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 02 15:46:31.802250 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795808    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 02 15:46:31.802250 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795810    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 02 15:46:31.802250 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795813    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 02 15:46:31.802250 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795815    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 02 15:46:31.802250 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795818    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 02 15:46:31.802250 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795821    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 02 15:46:31.802250 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795825    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 02 15:46:31.802250 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795828    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 02 15:46:31.802805 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795831    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 02 15:46:31.802805 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795834    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 02 15:46:31.802805 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795837    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 02 15:46:31.802805 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795839    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Jun 02 15:46:31.802805 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795842    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Jun 02 15:46:31.802805 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795845    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 02 15:46:31.802805 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795848    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 02 15:46:31.802805 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795850    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 02 15:46:31.802805 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795852    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 02 15:46:31.802805 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795855    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 02 15:46:31.802805 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795857    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 02 15:46:31.802805 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795865    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 02 15:46:31.802805 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795867    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 02 15:46:31.802805 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795870    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 02 15:46:31.802805 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795872    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 02 15:46:31.802805 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.795875    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 02 15:46:31.803203 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.796779    2579 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
Jun 02 15:46:31.803828 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.803807    2579 server.go:530] "Kubelet version" kubeletVersion="v1.33.12"
Jun 02 15:46:31.803865 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.803830    2579 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jun 02 15:46:31.803895 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.803881    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 02 15:46:31.803895 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.803888    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 02 15:46:31.803895 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.803891    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 02 15:46:31.803895 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.803894    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 02 15:46:31.803895 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.803897    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 02 15:46:31.804019 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.803900    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 02 15:46:31.804019 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.803904    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 02 15:46:31.804019 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.803907    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 02 15:46:31.804019 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.803909    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 02 15:46:31.804019 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.803912    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 02 15:46:31.804019 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.803915    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 02 15:46:31.804019 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.803917    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 02 15:46:31.804019 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.803920    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 02 15:46:31.804019 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.803922    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 02 15:46:31.804019 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.803924    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 02 15:46:31.804019 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.803927    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 02 15:46:31.804019 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.803929    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 02 15:46:31.804019 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.803932    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Jun 02 15:46:31.804019 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.803935    2579 feature_gate.go:328] unrecognized feature gate: Example
Jun 02 15:46:31.804019 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.803937    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 02 15:46:31.804019 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.803940    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 02 15:46:31.804019 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.803942    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 02 15:46:31.804019 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.803945    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 02 15:46:31.804019 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.803947    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 02 15:46:31.804019 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.803950    2579 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Jun 02 15:46:31.804529 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.803954    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 02 15:46:31.804529 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.803956    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 02 15:46:31.804529 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.803959    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 02 15:46:31.804529 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.803962    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 02 15:46:31.804529 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.803980    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 02 15:46:31.804529 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.803984    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 02 15:46:31.804529 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.803987    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 02 15:46:31.804529 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.803990    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 02 15:46:31.804529 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.803993    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 02 15:46:31.804529 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.803996    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 02 15:46:31.804529 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.803999    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 02 15:46:31.804529 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804001    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 02 15:46:31.804529 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804004    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 02 15:46:31.804529 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804008    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 02 15:46:31.804529 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804011    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 02 15:46:31.804529 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804013    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 02 15:46:31.804529 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804016    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 02 15:46:31.804529 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804018    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 02 15:46:31.804529 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804021    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 02 15:46:31.804990 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804024    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 02 15:46:31.804990 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804026    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 02 15:46:31.804990 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804029    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 02 15:46:31.804990 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804031    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 02 15:46:31.804990 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804034    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 02 15:46:31.804990 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804037    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 02 15:46:31.804990 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804039    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 02 15:46:31.804990 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804042    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 02 15:46:31.804990 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804044    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jun 02 15:46:31.804990 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804046    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 02 15:46:31.804990 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804049    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 02 15:46:31.804990 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804053    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 02 15:46:31.804990 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804056    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 02 15:46:31.804990 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804059    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 02 15:46:31.804990 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804062    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 02 15:46:31.804990 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804065    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 02 15:46:31.804990 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804067    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 02 15:46:31.804990 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804070    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 02 15:46:31.804990 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804072    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 02 15:46:31.804990 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804075    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 02 15:46:31.805582 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804077    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 02 15:46:31.805582 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804080    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 02 15:46:31.805582 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804082    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 02 15:46:31.805582 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804085    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 02 15:46:31.805582 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804087    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 02 15:46:31.805582 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804090    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Jun 02 15:46:31.805582 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804093    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 02 15:46:31.805582 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804096    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 02 15:46:31.805582 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804099    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 02 15:46:31.805582 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804101    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 02 15:46:31.805582 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804105    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 02 15:46:31.805582 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804109    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 02 15:46:31.805582 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804112    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 02 15:46:31.805582 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804115    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 02 15:46:31.805582 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804117    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 02 15:46:31.805582 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804120    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 02 15:46:31.805582 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804123    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 02 15:46:31.805582 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804125    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 02 15:46:31.805582 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804128    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 02 15:46:31.806053 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804130    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 02 15:46:31.806053 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804133    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Jun 02 15:46:31.806053 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804135    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 02 15:46:31.806053 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.804140    2579 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
Jun 02 15:46:31.806053 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804247    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 02 15:46:31.806053 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804253    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 02 15:46:31.806053 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804256    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 02 15:46:31.806053 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804259    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 02 15:46:31.806053 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804263    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 02 15:46:31.806053 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804267    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 02 15:46:31.806053 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804270    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 02 15:46:31.806053 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804273    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 02 15:46:31.806053 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804277    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 02 15:46:31.806053 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804280    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 02 15:46:31.806053 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804283    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 02 15:46:31.806494 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804285    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 02 15:46:31.806494 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804288    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 02 15:46:31.806494 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804291    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 02 15:46:31.806494 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804293    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 02 15:46:31.806494 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804297    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 02 15:46:31.806494 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804300    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 02 15:46:31.806494 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804302    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 02 15:46:31.806494 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804305    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 02 15:46:31.806494 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804308    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 02 15:46:31.806494 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804311    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 02 15:46:31.806494 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804313    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Jun 02 15:46:31.806494 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804316    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 02 15:46:31.806494 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804319    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 02 15:46:31.806494 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804321    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 02 15:46:31.806494 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804324    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Jun 02 15:46:31.806494 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804326    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 02 15:46:31.806494 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804329    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 02 15:46:31.806494 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804331    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 02 15:46:31.806494 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804333    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 02 15:46:31.806494 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804336    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 02 15:46:31.806973 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804338    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 02 15:46:31.806973 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804341    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 02 15:46:31.806973 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804343    2579 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Jun 02 15:46:31.806973 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804346    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 02 15:46:31.806973 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804348    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 02 15:46:31.806973 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804351    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 02 15:46:31.806973 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804353    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 02 15:46:31.806973 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804356    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 02 15:46:31.806973 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804358    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 02 15:46:31.806973 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804361    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 02 15:46:31.806973 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804363    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 02 15:46:31.806973 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804365    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 02 15:46:31.806973 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804368    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 02 15:46:31.806973 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804370    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 02 15:46:31.806973 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804373    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 02 15:46:31.806973 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804375    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 02 15:46:31.806973 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804378    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 02 15:46:31.806973 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804381    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 02 15:46:31.806973 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804383    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 02 15:46:31.806973 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804386    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 02 15:46:31.807470 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804388    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 02 15:46:31.807470 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804391    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 02 15:46:31.807470 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804393    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 02 15:46:31.807470 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804396    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 02 15:46:31.807470 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804416    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 02 15:46:31.807470 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804419    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jun 02 15:46:31.807470 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804422    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 02 15:46:31.807470 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804424    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 02 15:46:31.807470 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804427    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 02 15:46:31.807470 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804429    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 02 15:46:31.807470 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804432    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 02 15:46:31.807470 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804434    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 02 15:46:31.807470 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804437    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 02 15:46:31.807470 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804439    2579 feature_gate.go:328] unrecognized feature gate: Example
Jun 02 15:46:31.807470 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804442    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 02 15:46:31.807470 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804445    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 02 15:46:31.807470 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804448    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Jun 02 15:46:31.807470 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804451    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 02 15:46:31.807470 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804453    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 02 15:46:31.807932 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804456    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 02 15:46:31.807932 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804459    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 02 15:46:31.807932 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804462    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 02 15:46:31.807932 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804464    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 02 15:46:31.807932 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804466    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 02 15:46:31.807932 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804469    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 02 15:46:31.807932 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804471    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 02 15:46:31.807932 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804474    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 02 15:46:31.807932 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804476    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 02 15:46:31.807932 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804479    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 02 15:46:31.807932 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804481    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 02 15:46:31.807932 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804484    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 02 15:46:31.807932 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804486    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 02 15:46:31.807932 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804489    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 02 15:46:31.807932 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804491    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 02 15:46:31.807932 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:31.804494    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 02 15:46:31.808310 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.804499    2579 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
Jun 02 15:46:31.808310 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.804612    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Jun 02 15:46:31.809380 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.809364    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jun 02 15:46:31.810247 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.810235    2579 server.go:1019] "Starting client certificate rotation"
Jun 02 15:46:31.810353 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.810335    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jun 02 15:46:31.811136 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.811125    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jun 02 15:46:31.833998 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.833976    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jun 02 15:46:31.837567 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.837538    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jun 02 15:46:31.848696 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.848675    2579 log.go:25] "Validated CRI v1 runtime API"
Jun 02 15:46:31.856755 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.856735    2579 log.go:25] "Validated CRI v1 image API"
Jun 02 15:46:31.858147 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.858115    2579 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jun 02 15:46:31.865469 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.865434    2579 fs.go:135] Filesystem UUIDs: map[4abb48e5-f591-44ed-adec-0a79efedec36:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c1d169d9-b69f-4df9-8763-2f28b985a2a9:/dev/nvme0n1p4]
Jun 02 15:46:31.865540 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.865466    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jun 02 15:46:31.870214 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.870194    2579 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jun 02 15:46:31.870307 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.869902    2579 manager.go:217] Machine: {Timestamp:2026-06-02 15:46:31.869354014 +0000 UTC m=+0.395262436 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099940 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2622755cbc5451f904fff40de3eadd SystemUUID:ec262275-5cbc-5451-f904-fff40de3eadd BootID:73f731f5-735f-49f9-a580-80fa06d206a9 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:07:0c:61:5c:4d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:07:0c:61:5c:4d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:56:36:43:16:0c:ec Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jun 02 15:46:31.870990 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.870979    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jun 02 15:46:31.871128 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.871114    2579 manager.go:233] Version: {KernelVersion:5.14.0-570.117.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260521-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jun 02 15:46:31.873446 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.873203    2579 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jun 02 15:46:31.873511 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.873242    2579 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-156.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jun 02 15:46:31.873511 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.873492    2579 topology_manager.go:138] "Creating topology manager with none policy"
Jun 02 15:46:31.873511 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.873507    2579 container_manager_linux.go:306] "Creating device plugin manager"
Jun 02 15:46:31.873688 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.873518    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jun 02 15:46:31.874388 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.874375    2579 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jun 02 15:46:31.876434 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.876423    2579 state_mem.go:36] "Initialized new in-memory state store"
Jun 02 15:46:31.876551 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.876542    2579 server.go:1267] "Using root directory" path="/var/lib/kubelet"
Jun 02 15:46:31.878866 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.878855    2579 kubelet.go:491] "Attempting to sync node with API server"
Jun 02 15:46:31.878909 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.878872    2579 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
Jun 02 15:46:31.878909 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.878886    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jun 02 15:46:31.878909 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.878896    2579 kubelet.go:397] "Adding apiserver pod source"
Jun 02 15:46:31.878909 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.878907    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jun 02 15:46:31.879870 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.879857    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jun 02 15:46:31.879918 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.879877    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jun 02 15:46:31.895749 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.895728    2579 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.12-3.rhaos4.20.git64c6a00.el9" apiVersion="v1"
Jun 02 15:46:31.901228 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.901200    2579 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jun 02 15:46:31.903827 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.903803    2579 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jun 02 15:46:31.903827 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.903829    2579 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jun 02 15:46:31.903827 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.903835    2579 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jun 02 15:46:31.903994 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.903841    2579 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jun 02 15:46:31.903994 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.903851    2579 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jun 02 15:46:31.903994 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.903857    2579 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jun 02 15:46:31.903994 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.903863    2579 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jun 02 15:46:31.903994 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.903868    2579 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jun 02 15:46:31.903994 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.903875    2579 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jun 02 15:46:31.903994 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.903882    2579 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jun 02 15:46:31.903994 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.903891    2579 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jun 02 15:46:31.903994 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.903896    2579 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jun 02 15:46:31.904637 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.904627    2579 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jun 02 15:46:31.904637 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.904637    2579 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jun 02 15:46:31.905930 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.905907    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lpljt"
Jun 02 15:46:31.907552 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.907533    2579 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-156.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jun 02 15:46:31.908688 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.908674    2579 watchdog_linux.go:99] "Systemd watchdog is not enabled"
Jun 02 15:46:31.908755 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.908708    2579 server.go:1295] "Started kubelet"
Jun 02 15:46:31.908941 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:31.908918    2579 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jun 02 15:46:31.909028 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:31.909012    2579 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-156.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jun 02 15:46:31.909337 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.909313    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jun 02 15:46:31.909701 ip-10-0-129-156 systemd[1]: Started Kubernetes Kubelet.
Jun 02 15:46:31.910126 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.909931    2579 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jun 02 15:46:31.910126 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.909995    2579 server_v1.go:47] "podresources" method="list" useActivePods=true
Jun 02 15:46:31.911379 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.911236    2579 server.go:317] "Adding debug handlers to kubelet server"
Jun 02 15:46:31.911479 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.911455    2579 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jun 02 15:46:31.916052 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:31.915124    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-156.ec2.internal.18b54e2f1a6d1d4c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-156.ec2.internal,UID:ip-10-0-129-156.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-156.ec2.internal,},FirstTimestamp:2026-06-02 15:46:31.90868718 +0000 UTC m=+0.434595601,LastTimestamp:2026-06-02 15:46:31.90868718 +0000 UTC m=+0.434595601,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-156.ec2.internal,}"
Jun 02 15:46:31.918046 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.918029    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lpljt"
Jun 02 15:46:31.918669 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.918654    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jun 02 15:46:31.918753 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.918726    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jun 02 15:46:31.918927 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.918893    2579 volume_manager.go:295] "The desired_state_of_world populator starts"
Jun 02 15:46:31.918927 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.918909    2579 volume_manager.go:297] "Starting Kubelet Volume Manager"
Jun 02 15:46:31.919074 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.919058    2579 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
Jun 02 15:46:31.919165 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.919124    2579 reconstruct.go:97] "Volume reconstruction finished"
Jun 02 15:46:31.919165 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.919132    2579 reconciler.go:26] "Reconciler: start to sync state"
Jun 02 15:46:31.919653 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:31.919631    2579 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-156.ec2.internal\" not found"
Jun 02 15:46:31.919653 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.919642    2579 factory.go:55] Registering systemd factory
Jun 02 15:46:31.919789 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:31.919676    2579 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jun 02 15:46:31.919789 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.919682    2579 factory.go:223] Registration of the systemd container factory successfully
Jun 02 15:46:31.920010 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.919992    2579 factory.go:153] Registering CRI-O factory
Jun 02 15:46:31.920010 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.920010    2579 factory.go:223] Registration of the crio container factory successfully
Jun 02 15:46:31.920174 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.920120    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jun 02 15:46:31.920174 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.920145    2579 factory.go:103] Registering Raw factory
Jun 02 15:46:31.920274 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.920178    2579 manager.go:1196] Started watching for new ooms in manager
Jun 02 15:46:31.920887 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.920872    2579 manager.go:319] Starting recovery of all containers
Jun 02 15:46:31.930518 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.930372    2579 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
Jun 02 15:46:31.934462 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:31.934435    2579 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-156.ec2.internal\" not found" node="ip-10-0-129-156.ec2.internal"
Jun 02 15:46:31.935808 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.935782    2579 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 02 15:46:31.935912 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.935870    2579 manager.go:324] Recovery completed
Jun 02 15:46:31.937431 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:31.937389    2579 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jun 02 15:46:31.940381 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.940367    2579 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Jun 02 15:46:31.942484 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.942470    2579 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-156.ec2.internal" event="NodeHasSufficientMemory"
Jun 02 15:46:31.942529 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.942500    2579 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-156.ec2.internal" event="NodeHasNoDiskPressure"
Jun 02 15:46:31.942529 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.942510    2579 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-156.ec2.internal" event="NodeHasSufficientPID"
Jun 02 15:46:31.943045 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.943024    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jun 02 15:46:31.943045 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.943044    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jun 02 15:46:31.943150 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.943061    2579 state_mem.go:36] "Initialized new in-memory state store"
Jun 02 15:46:31.946163 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.946150    2579 policy_none.go:49] "None policy: Start"
Jun 02 15:46:31.946208 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.946169    2579 memory_manager.go:186] "Starting memorymanager" policy="None"
Jun 02 15:46:31.946208 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.946180    2579 state_mem.go:35] "Initializing new in-memory state store"
Jun 02 15:46:31.983036 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.983013    2579 manager.go:341] "Starting Device Plugin manager"
Jun 02 15:46:31.995127 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:31.983121    2579 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jun 02 15:46:31.995127 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.983137    2579 server.go:85] "Starting device plugin registration server"
Jun 02 15:46:31.995127 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.983374    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Jun 02 15:46:31.995127 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.983383    2579 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jun 02 15:46:31.995127 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.983535    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jun 02 15:46:31.995127 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.983657    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jun 02 15:46:31.995127 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:31.983669    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jun 02 15:46:31.995127 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:31.984114    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jun 02 15:46:31.995127 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:31.984156    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-156.ec2.internal\" not found"
Jun 02 15:46:32.058874 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.058802    2579 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
Jun 02 15:46:32.058874 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.058833    2579 status_manager.go:230] "Starting to sync pod status with apiserver"
Jun 02 15:46:32.059068 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.058904    2579 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jun 02 15:46:32.059068 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.058915    2579 kubelet.go:2451] "Starting kubelet main sync loop"
Jun 02 15:46:32.059068 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:32.058965    2579 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jun 02 15:46:32.062796 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.062768    2579 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 02 15:46:32.084198 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.084177    2579 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Jun 02 15:46:32.085389 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.085371    2579 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-156.ec2.internal" event="NodeHasSufficientMemory"
Jun 02 15:46:32.085519 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.085427    2579 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-156.ec2.internal" event="NodeHasNoDiskPressure"
Jun 02 15:46:32.085519 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.085445    2579 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-156.ec2.internal" event="NodeHasSufficientPID"
Jun 02 15:46:32.085519 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.085479    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-156.ec2.internal"
Jun 02 15:46:32.096145 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.096121    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-156.ec2.internal"
Jun 02 15:46:32.096256 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:32.096149    2579 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-156.ec2.internal\": node \"ip-10-0-129-156.ec2.internal\" not found"
Jun 02 15:46:32.111907 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:32.111883    2579 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-156.ec2.internal\" not found"
Jun 02 15:46:32.159508 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.159454    2579 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-156.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-156.ec2.internal"]
Jun 02 15:46:32.159662 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.159579    2579 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Jun 02 15:46:32.160655 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.160636    2579 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-156.ec2.internal" event="NodeHasSufficientMemory"
Jun 02 15:46:32.160746 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.160667    2579 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-156.ec2.internal" event="NodeHasNoDiskPressure"
Jun 02 15:46:32.160746 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.160677    2579 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-156.ec2.internal" event="NodeHasSufficientPID"
Jun 02 15:46:32.161913 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.161896    2579 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Jun 02 15:46:32.162079 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.162059    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-156.ec2.internal"
Jun 02 15:46:32.162145 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.162101    2579 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Jun 02 15:46:32.162577 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.162560    2579 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-156.ec2.internal" event="NodeHasSufficientMemory"
Jun 02 15:46:32.162648 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.162588    2579 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-156.ec2.internal" event="NodeHasNoDiskPressure"
Jun 02 15:46:32.162648 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.162603    2579 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-156.ec2.internal" event="NodeHasSufficientPID"
Jun 02 15:46:32.162648 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.162622    2579 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-156.ec2.internal" event="NodeHasSufficientMemory"
Jun 02 15:46:32.162648 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.162639    2579 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-156.ec2.internal" event="NodeHasNoDiskPressure"
Jun 02 15:46:32.162791 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.162651    2579 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-156.ec2.internal" event="NodeHasSufficientPID"
Jun 02 15:46:32.163904 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.163888    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-156.ec2.internal"
Jun 02 15:46:32.163951 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.163923    2579 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Jun 02 15:46:32.164581 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.164567    2579 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-156.ec2.internal" event="NodeHasSufficientMemory"
Jun 02 15:46:32.164649 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.164593    2579 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-156.ec2.internal" event="NodeHasNoDiskPressure"
Jun 02 15:46:32.164649 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.164606    2579 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-156.ec2.internal" event="NodeHasSufficientPID"
Jun 02 15:46:32.181974 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:32.181951    2579 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-156.ec2.internal\" not found" node="ip-10-0-129-156.ec2.internal"
Jun 02 15:46:32.186201 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:32.186185    2579 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-156.ec2.internal\" not found" node="ip-10-0-129-156.ec2.internal"
Jun 02 15:46:32.212686 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:32.212654    2579 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-156.ec2.internal\" not found"
Jun 02 15:46:32.220974 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.220947    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/42688cc70769c2fbb9ca611c4f39c7ef-config\") pod \"kube-apiserver-proxy-ip-10-0-129-156.ec2.internal\" (UID: \"42688cc70769c2fbb9ca611c4f39c7ef\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-156.ec2.internal"
Jun 02 15:46:32.221056 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.220980    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/23859d63aa1dc8f73c323540603ab2ba-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-156.ec2.internal\" (UID: \"23859d63aa1dc8f73c323540603ab2ba\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-156.ec2.internal"
Jun 02 15:46:32.221056 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.221005    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23859d63aa1dc8f73c323540603ab2ba-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-156.ec2.internal\" (UID: \"23859d63aa1dc8f73c323540603ab2ba\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-156.ec2.internal"
Jun 02 15:46:32.312815 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:32.312733    2579 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-156.ec2.internal\" not found"
Jun 02 15:46:32.322054 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.322025    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/42688cc70769c2fbb9ca611c4f39c7ef-config\") pod \"kube-apiserver-proxy-ip-10-0-129-156.ec2.internal\" (UID: \"42688cc70769c2fbb9ca611c4f39c7ef\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-156.ec2.internal"
Jun 02 15:46:32.322148 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.321994    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/42688cc70769c2fbb9ca611c4f39c7ef-config\") pod \"kube-apiserver-proxy-ip-10-0-129-156.ec2.internal\" (UID: \"42688cc70769c2fbb9ca611c4f39c7ef\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-156.ec2.internal"
Jun 02 15:46:32.322148 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.322141    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/23859d63aa1dc8f73c323540603ab2ba-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-156.ec2.internal\" (UID: \"23859d63aa1dc8f73c323540603ab2ba\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-156.ec2.internal"
Jun 02 15:46:32.322249 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.322186    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23859d63aa1dc8f73c323540603ab2ba-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-156.ec2.internal\" (UID: \"23859d63aa1dc8f73c323540603ab2ba\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-156.ec2.internal"
Jun 02 15:46:32.322249 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.322196    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/23859d63aa1dc8f73c323540603ab2ba-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-156.ec2.internal\" (UID: \"23859d63aa1dc8f73c323540603ab2ba\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-156.ec2.internal"
Jun 02 15:46:32.322249 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.322222    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23859d63aa1dc8f73c323540603ab2ba-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-156.ec2.internal\" (UID: \"23859d63aa1dc8f73c323540603ab2ba\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-156.ec2.internal"
Jun 02 15:46:32.413030 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:32.412979    2579 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-156.ec2.internal\" not found"
Jun 02 15:46:32.484505 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.484460    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-156.ec2.internal"
Jun 02 15:46:32.490500 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.490477    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-156.ec2.internal"
Jun 02 15:46:32.513350 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:32.513321    2579 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-156.ec2.internal\" not found"
Jun 02 15:46:32.613761 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:32.613669    2579 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-156.ec2.internal\" not found"
Jun 02 15:46:32.714191 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:32.714159    2579 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-156.ec2.internal\" not found"
Jun 02 15:46:32.732129 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.732101    2579 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 02 15:46:32.810524 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.810485    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jun 02 15:46:32.811235 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.810670    2579 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jun 02 15:46:32.811235 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.810687    2579 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jun 02 15:46:32.811235 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.810687    2579 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jun 02 15:46:32.814632 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:32.814614    2579 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-156.ec2.internal\" not found"
Jun 02 15:46:32.915340 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:32.915297    2579 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-156.ec2.internal\" not found"
Jun 02 15:46:32.920304 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.920266    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-06-01 15:41:31 +0000 UTC" deadline="2028-01-18 04:55:43.804194287 +0000 UTC"
Jun 02 15:46:32.920304 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.920292    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jun 02 15:46:32.920304 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.920301    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14269h9m10.883897774s"
Jun 02 15:46:32.930228 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.930204    2579 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jun 02 15:46:32.974861 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.974823    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2z7b6"
Jun 02 15:46:32.982116 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.982089    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2z7b6"
Jun 02 15:46:32.988349 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:32.988318    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23859d63aa1dc8f73c323540603ab2ba.slice/crio-b6493a079430ca7d2caa3c343ff5b8a5ffe631fa6631dfda0c953b0a453a51df WatchSource:0}: Error finding container b6493a079430ca7d2caa3c343ff5b8a5ffe631fa6631dfda0c953b0a453a51df: Status 404 returned error can't find the container with id b6493a079430ca7d2caa3c343ff5b8a5ffe631fa6631dfda0c953b0a453a51df
Jun 02 15:46:32.988564 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:32.988544    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42688cc70769c2fbb9ca611c4f39c7ef.slice/crio-725b760016ef6267fad6125a4422ca8c53d605d7c9165857d29199d32edb571d WatchSource:0}: Error finding container 725b760016ef6267fad6125a4422ca8c53d605d7c9165857d29199d32edb571d: Status 404 returned error can't find the container with id 725b760016ef6267fad6125a4422ca8c53d605d7c9165857d29199d32edb571d
Jun 02 15:46:32.991693 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:32.991679    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 15:46:33.016106 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:33.016079    2579 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-156.ec2.internal\" not found"
Jun 02 15:46:33.062637 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.062585    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-156.ec2.internal" event={"ID":"23859d63aa1dc8f73c323540603ab2ba","Type":"ContainerStarted","Data":"b6493a079430ca7d2caa3c343ff5b8a5ffe631fa6631dfda0c953b0a453a51df"}
Jun 02 15:46:33.063612 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.063588    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-156.ec2.internal" event={"ID":"42688cc70769c2fbb9ca611c4f39c7ef","Type":"ContainerStarted","Data":"725b760016ef6267fad6125a4422ca8c53d605d7c9165857d29199d32edb571d"}
Jun 02 15:46:33.116803 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:33.116772    2579 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-156.ec2.internal\" not found"
Jun 02 15:46:33.217342 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:33.217250    2579 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-156.ec2.internal\" not found"
Jun 02 15:46:33.317784 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:33.317745    2579 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-156.ec2.internal\" not found"
Jun 02 15:46:33.325766 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.325735    2579 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 02 15:46:33.419574 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.419506    2579 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-156.ec2.internal"
Jun 02 15:46:33.430873 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.430733    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jun 02 15:46:33.432054 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.431816    2579 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-156.ec2.internal"
Jun 02 15:46:33.442264 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.442139    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jun 02 15:46:33.880082 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.880050    2579 apiserver.go:52] "Watching apiserver"
Jun 02 15:46:33.888751 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.888721    2579 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jun 02 15:46:33.889113 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.889087    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-th978","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-156.ec2.internal","openshift-multus/multus-lg8nx","openshift-multus/network-metrics-daemon-6lvc8","openshift-network-diagnostics/network-check-target-zzt45","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhfk7","openshift-cluster-node-tuning-operator/tuned-mllz2","openshift-image-registry/node-ca-cwjmt","openshift-multus/multus-additional-cni-plugins-dc6wj","openshift-network-operator/iptables-alerter-zqsll","openshift-ovn-kubernetes/ovnkube-node-lqx9k","kube-system/konnectivity-agent-zt7mm","kube-system/kube-apiserver-proxy-ip-10-0-129-156.ec2.internal"]
Jun 02 15:46:33.890688 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.890667    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:33.891961 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.891919    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:33.893361 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.893263    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6lvc8"
Jun 02 15:46:33.893470 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:33.893355    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6lvc8" podUID="eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca"
Jun 02 15:46:33.893700 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.893677    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jun 02 15:46:33.893840 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.893823    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jun 02 15:46:33.894085 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.894062    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jun 02 15:46:33.894344 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.894325    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6wggr\""
Jun 02 15:46:33.894451 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.894443    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jun 02 15:46:33.894662 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.894642    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jun 02 15:46:33.894739 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.894715    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bbx5n\""
Jun 02 15:46:33.894815 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.894649    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jun 02 15:46:33.896670 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.896648    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzt45"
Jun 02 15:46:33.896784 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:33.896748    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzt45" podUID="43370289-88af-4e83-b409-4e8a98e09857"
Jun 02 15:46:33.897949 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.897926    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhfk7"
Jun 02 15:46:33.898044 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.898019    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-th978"
Jun 02 15:46:33.899274 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.899254    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cwjmt"
Jun 02 15:46:33.900605 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.900582    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dc6wj"
Jun 02 15:46:33.901878 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.901859    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zqsll"
Jun 02 15:46:33.902488 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.902468    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jun 02 15:46:33.903460 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.903443    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jun 02 15:46:33.903680 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.903667    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jun 02 15:46:33.903892 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.903878    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jun 02 15:46:33.904051 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.904033    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bhsjt\""
Jun 02 15:46:33.904130 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.904116    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jun 02 15:46:33.904617 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.904597    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gqlm4\""
Jun 02 15:46:33.904887 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.904870    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jun 02 15:46:33.905204 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.905188    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jun 02 15:46:33.905241 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.905219    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9w56j\""
Jun 02 15:46:33.905438 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.905422    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jun 02 15:46:33.905504 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.905490    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6mmw6\""
Jun 02 15:46:33.905812 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.905785    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jun 02 15:46:33.905932 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.905913    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rjt4s\""
Jun 02 15:46:33.906150 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.906131    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jun 02 15:46:33.906308 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.906290    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jun 02 15:46:33.906504 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.906485    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zt7mm"
Jun 02 15:46:33.906584 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.906546    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jun 02 15:46:33.906669 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.906654    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:33.909973 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.909955    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jun 02 15:46:33.910184 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.910168    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jun 02 15:46:33.910260 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.910217    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-v5jf5\""
Jun 02 15:46:33.910746 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.910724    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jun 02 15:46:33.911444 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.911427    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-55ljc\""
Jun 02 15:46:33.915361 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.914779    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jun 02 15:46:33.915361 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.914847    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jun 02 15:46:33.915361 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.914914    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jun 02 15:46:33.915361 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.915033    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jun 02 15:46:33.915361 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.915057    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jun 02 15:46:33.915361 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.915234    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jun 02 15:46:33.920328 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.920308    2579 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
Jun 02 15:46:33.931147 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.931126    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/498f7354-865b-4ad1-9e9f-fabd57febd4a-etc-tuned\") pod \"tuned-mllz2\" (UID: \"498f7354-865b-4ad1-9e9f-fabd57febd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:33.931260 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.931158    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/73b94606-f892-4c47-9022-5d42f71bb77d-os-release\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:33.931260 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.931185    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a4a82f10-0682-47f3-b029-9b45d89383d3-device-dir\") pod \"aws-ebs-csi-driver-node-bhfk7\" (UID: \"a4a82f10-0682-47f3-b029-9b45d89383d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhfk7"
Jun 02 15:46:33.931260 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.931203    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/810b8e3e-5836-4214-a141-c340d2220639-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dc6wj\" (UID: \"810b8e3e-5836-4214-a141-c340d2220639\") " pod="openshift-multus/multus-additional-cni-plugins-dc6wj"
Jun 02 15:46:33.931260 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.931223    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/498f7354-865b-4ad1-9e9f-fabd57febd4a-tmp\") pod \"tuned-mllz2\" (UID: \"498f7354-865b-4ad1-9e9f-fabd57febd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:33.931260 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.931247    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qclv4\" (UniqueName: \"kubernetes.io/projected/a4a82f10-0682-47f3-b029-9b45d89383d3-kube-api-access-qclv4\") pod \"aws-ebs-csi-driver-node-bhfk7\" (UID: \"a4a82f10-0682-47f3-b029-9b45d89383d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhfk7"
Jun 02 15:46:33.931466 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.931308    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6127e33c-aa77-437e-b443-e956037afedb-ovnkube-script-lib\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:33.931466 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.931338    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/498f7354-865b-4ad1-9e9f-fabd57febd4a-sys\") pod \"tuned-mllz2\" (UID: \"498f7354-865b-4ad1-9e9f-fabd57febd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:33.931466 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.931356    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/498f7354-865b-4ad1-9e9f-fabd57febd4a-host\") pod \"tuned-mllz2\" (UID: \"498f7354-865b-4ad1-9e9f-fabd57febd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:33.931466 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.931372    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/73b94606-f892-4c47-9022-5d42f71bb77d-multus-cni-dir\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:33.931466 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.931389    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/73b94606-f892-4c47-9022-5d42f71bb77d-host-run-k8s-cni-cncf-io\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:33.931466 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.931436    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/73b94606-f892-4c47-9022-5d42f71bb77d-multus-daemon-config\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:33.931466 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.931464    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a4a82f10-0682-47f3-b029-9b45d89383d3-registration-dir\") pod \"aws-ebs-csi-driver-node-bhfk7\" (UID: \"a4a82f10-0682-47f3-b029-9b45d89383d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhfk7"
Jun 02 15:46:33.931715 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.931489    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/810b8e3e-5836-4214-a141-c340d2220639-cni-binary-copy\") pod \"multus-additional-cni-plugins-dc6wj\" (UID: \"810b8e3e-5836-4214-a141-c340d2220639\") " pod="openshift-multus/multus-additional-cni-plugins-dc6wj"
Jun 02 15:46:33.931715 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.931532    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-node-log\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:33.931715 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.931561    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/498f7354-865b-4ad1-9e9f-fabd57febd4a-run\") pod \"tuned-mllz2\" (UID: \"498f7354-865b-4ad1-9e9f-fabd57febd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:33.931715 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.931586    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/73b94606-f892-4c47-9022-5d42f71bb77d-host-run-multus-certs\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:33.931715 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.931646    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a4a82f10-0682-47f3-b029-9b45d89383d3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bhfk7\" (UID: \"a4a82f10-0682-47f3-b029-9b45d89383d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhfk7"
Jun 02 15:46:33.931715 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.931678    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a4a82f10-0682-47f3-b029-9b45d89383d3-etc-selinux\") pod \"aws-ebs-csi-driver-node-bhfk7\" (UID: \"a4a82f10-0682-47f3-b029-9b45d89383d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhfk7"
Jun 02 15:46:33.931715 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.931706    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d87j5\" (UniqueName: \"kubernetes.io/projected/43370289-88af-4e83-b409-4e8a98e09857-kube-api-access-d87j5\") pod \"network-check-target-zzt45\" (UID: \"43370289-88af-4e83-b409-4e8a98e09857\") " pod="openshift-network-diagnostics/network-check-target-zzt45"
Jun 02 15:46:33.931966 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.931729    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-host-slash\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:33.931966 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.931751    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-run-systemd\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:33.931966 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.931778    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:33.931966 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.931816    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/73b94606-f892-4c47-9022-5d42f71bb77d-cnibin\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:33.931966 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.931837    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c0c551c2-7ea0-4de3-b24a-735cdb0c536d-iptables-alerter-script\") pod \"iptables-alerter-zqsll\" (UID: \"c0c551c2-7ea0-4de3-b24a-735cdb0c536d\") " pod="openshift-network-operator/iptables-alerter-zqsll"
Jun 02 15:46:33.931966 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.931860    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk4bb\" (UniqueName: \"kubernetes.io/projected/c0c551c2-7ea0-4de3-b24a-735cdb0c536d-kube-api-access-rk4bb\") pod \"iptables-alerter-zqsll\" (UID: \"c0c551c2-7ea0-4de3-b24a-735cdb0c536d\") " pod="openshift-network-operator/iptables-alerter-zqsll"
Jun 02 15:46:33.931966 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.931875    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v668f\" (UniqueName: \"kubernetes.io/projected/eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca-kube-api-access-v668f\") pod \"network-metrics-daemon-6lvc8\" (UID: \"eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca\") " pod="openshift-multus/network-metrics-daemon-6lvc8"
Jun 02 15:46:33.931966 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.931889    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/810b8e3e-5836-4214-a141-c340d2220639-system-cni-dir\") pod \"multus-additional-cni-plugins-dc6wj\" (UID: \"810b8e3e-5836-4214-a141-c340d2220639\") " pod="openshift-multus/multus-additional-cni-plugins-dc6wj"
Jun 02 15:46:33.931966 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.931902    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/810b8e3e-5836-4214-a141-c340d2220639-cnibin\") pod \"multus-additional-cni-plugins-dc6wj\" (UID: \"810b8e3e-5836-4214-a141-c340d2220639\") " pod="openshift-multus/multus-additional-cni-plugins-dc6wj"
Jun 02 15:46:33.931966 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.931925    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6127e33c-aa77-437e-b443-e956037afedb-ovnkube-config\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:33.931966 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.931961    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/498f7354-865b-4ad1-9e9f-fabd57febd4a-var-lib-kubelet\") pod \"tuned-mllz2\" (UID: \"498f7354-865b-4ad1-9e9f-fabd57febd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:33.932306 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.932006    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/73b94606-f892-4c47-9022-5d42f71bb77d-multus-socket-dir-parent\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:33.932306 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.932038    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/73b94606-f892-4c47-9022-5d42f71bb77d-host-var-lib-cni-multus\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:33.932306 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.932064    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/810b8e3e-5836-4214-a141-c340d2220639-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dc6wj\" (UID: \"810b8e3e-5836-4214-a141-c340d2220639\") " pod="openshift-multus/multus-additional-cni-plugins-dc6wj"
Jun 02 15:46:33.932306 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.932087    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq2ml\" (UniqueName: \"kubernetes.io/projected/810b8e3e-5836-4214-a141-c340d2220639-kube-api-access-hq2ml\") pod \"multus-additional-cni-plugins-dc6wj\" (UID: \"810b8e3e-5836-4214-a141-c340d2220639\") " pod="openshift-multus/multus-additional-cni-plugins-dc6wj"
Jun 02 15:46:33.932306 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.932111    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-etc-openvswitch\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:33.932306 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.932135    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6127e33c-aa77-437e-b443-e956037afedb-env-overrides\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:33.932306 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.932159    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7b0e93e2-a6b5-4b5a-be51-e09f43184379-agent-certs\") pod \"konnectivity-agent-zt7mm\" (UID: \"7b0e93e2-a6b5-4b5a-be51-e09f43184379\") " pod="kube-system/konnectivity-agent-zt7mm"
Jun 02 15:46:33.932306 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.932179    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/73b94606-f892-4c47-9022-5d42f71bb77d-system-cni-dir\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:33.932306 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.932193    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-host-run-netns\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:33.932306 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.932206    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-host-cni-bin\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:33.932306 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.932245    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/498f7354-865b-4ad1-9e9f-fabd57febd4a-etc-sysctl-conf\") pod \"tuned-mllz2\" (UID: \"498f7354-865b-4ad1-9e9f-fabd57febd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:33.932306 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.932282    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3bee98d4-d583-4ecb-b07b-9f422a9810da-host\") pod \"node-ca-cwjmt\" (UID: \"3bee98d4-d583-4ecb-b07b-9f422a9810da\") " pod="openshift-image-registry/node-ca-cwjmt"
Jun 02 15:46:33.932758 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.932316    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l47g\" (UniqueName: \"kubernetes.io/projected/3bee98d4-d583-4ecb-b07b-9f422a9810da-kube-api-access-8l47g\") pod \"node-ca-cwjmt\" (UID: \"3bee98d4-d583-4ecb-b07b-9f422a9810da\") " pod="openshift-image-registry/node-ca-cwjmt"
Jun 02 15:46:33.932758 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.932360    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4290cf74-a174-4c3f-88b4-66b05bbe4750-tmp-dir\") pod \"node-resolver-th978\" (UID: \"4290cf74-a174-4c3f-88b4-66b05bbe4750\") " pod="openshift-dns/node-resolver-th978"
Jun 02 15:46:33.932758 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.932384    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-host-kubelet\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:33.932758 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.932424    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/498f7354-865b-4ad1-9e9f-fabd57febd4a-lib-modules\") pod \"tuned-mllz2\" (UID: \"498f7354-865b-4ad1-9e9f-fabd57febd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:33.932758 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.932449    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/73b94606-f892-4c47-9022-5d42f71bb77d-host-var-lib-cni-bin\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:33.932758 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.932471    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzbl5\" (UniqueName: \"kubernetes.io/projected/73b94606-f892-4c47-9022-5d42f71bb77d-kube-api-access-bzbl5\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:33.932758 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.932497    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca-metrics-certs\") pod \"network-metrics-daemon-6lvc8\" (UID: \"eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca\") " pod="openshift-multus/network-metrics-daemon-6lvc8"
Jun 02 15:46:33.932758 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.932523    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4290cf74-a174-4c3f-88b4-66b05bbe4750-hosts-file\") pod \"node-resolver-th978\" (UID: \"4290cf74-a174-4c3f-88b4-66b05bbe4750\") " pod="openshift-dns/node-resolver-th978"
Jun 02 15:46:33.932758 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.932547    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4d97\" (UniqueName: \"kubernetes.io/projected/4290cf74-a174-4c3f-88b4-66b05bbe4750-kube-api-access-x4d97\") pod \"node-resolver-th978\" (UID: \"4290cf74-a174-4c3f-88b4-66b05bbe4750\") " pod="openshift-dns/node-resolver-th978"
Jun 02 15:46:33.932758 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.932589    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-var-lib-openvswitch\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:33.932758 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.932614    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-run-ovn\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:33.932758 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.932680    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/498f7354-865b-4ad1-9e9f-fabd57febd4a-etc-sysconfig\") pod \"tuned-mllz2\" (UID: \"498f7354-865b-4ad1-9e9f-fabd57febd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:33.932758 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.932710    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/73b94606-f892-4c47-9022-5d42f71bb77d-cni-binary-copy\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:33.932758 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.932737    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3bee98d4-d583-4ecb-b07b-9f422a9810da-serviceca\") pod \"node-ca-cwjmt\" (UID: \"3bee98d4-d583-4ecb-b07b-9f422a9810da\") " pod="openshift-image-registry/node-ca-cwjmt"
Jun 02 15:46:33.932758 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.932760    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c0c551c2-7ea0-4de3-b24a-735cdb0c536d-host-slash\") pod \"iptables-alerter-zqsll\" (UID: \"c0c551c2-7ea0-4de3-b24a-735cdb0c536d\") " pod="openshift-network-operator/iptables-alerter-zqsll"
Jun 02 15:46:33.933334 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.932776    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6127e33c-aa77-437e-b443-e956037afedb-ovn-node-metrics-cert\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:33.933334 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.932793    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs6wj\" (UniqueName: \"kubernetes.io/projected/498f7354-865b-4ad1-9e9f-fabd57febd4a-kube-api-access-zs6wj\") pod \"tuned-mllz2\" (UID: \"498f7354-865b-4ad1-9e9f-fabd57febd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:33.933334 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.932816    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/73b94606-f892-4c47-9022-5d42f71bb77d-host-var-lib-kubelet\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:33.933334 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.932838    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73b94606-f892-4c47-9022-5d42f71bb77d-etc-kubernetes\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:33.933334 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.932862    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/498f7354-865b-4ad1-9e9f-fabd57febd4a-etc-kubernetes\") pod \"tuned-mllz2\" (UID: \"498f7354-865b-4ad1-9e9f-fabd57febd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:33.933334 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.932882    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/498f7354-865b-4ad1-9e9f-fabd57febd4a-etc-sysctl-d\") pod \"tuned-mllz2\" (UID: \"498f7354-865b-4ad1-9e9f-fabd57febd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:33.933334 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.932916    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/498f7354-865b-4ad1-9e9f-fabd57febd4a-etc-systemd\") pod \"tuned-mllz2\" (UID: \"498f7354-865b-4ad1-9e9f-fabd57febd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:33.933334 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.932940    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/73b94606-f892-4c47-9022-5d42f71bb77d-host-run-netns\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:33.933334 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.932979    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-log-socket\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:33.933334 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.933004    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqndc\" (UniqueName: \"kubernetes.io/projected/6127e33c-aa77-437e-b443-e956037afedb-kube-api-access-bqndc\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:33.933334 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.933030    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7b0e93e2-a6b5-4b5a-be51-e09f43184379-konnectivity-ca\") pod \"konnectivity-agent-zt7mm\" (UID: \"7b0e93e2-a6b5-4b5a-be51-e09f43184379\") " pod="kube-system/konnectivity-agent-zt7mm"
Jun 02 15:46:33.933334 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.933050    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/498f7354-865b-4ad1-9e9f-fabd57febd4a-etc-modprobe-d\") pod \"tuned-mllz2\" (UID: \"498f7354-865b-4ad1-9e9f-fabd57febd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:33.933334 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.933066    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/73b94606-f892-4c47-9022-5d42f71bb77d-hostroot\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:33.933334 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.933080    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/73b94606-f892-4c47-9022-5d42f71bb77d-multus-conf-dir\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:33.933334 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.933094    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/810b8e3e-5836-4214-a141-c340d2220639-os-release\") pod \"multus-additional-cni-plugins-dc6wj\" (UID: \"810b8e3e-5836-4214-a141-c340d2220639\") " pod="openshift-multus/multus-additional-cni-plugins-dc6wj"
Jun 02 15:46:33.933334 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.933109    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-run-openvswitch\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:33.933334 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.933130    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-host-run-ovn-kubernetes\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:33.933828 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.933170    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a4a82f10-0682-47f3-b029-9b45d89383d3-socket-dir\") pod \"aws-ebs-csi-driver-node-bhfk7\" (UID: \"a4a82f10-0682-47f3-b029-9b45d89383d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhfk7"
Jun 02 15:46:33.933828 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.933189    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a4a82f10-0682-47f3-b029-9b45d89383d3-sys-fs\") pod \"aws-ebs-csi-driver-node-bhfk7\" (UID: \"a4a82f10-0682-47f3-b029-9b45d89383d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhfk7"
Jun 02 15:46:33.933828 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.933209    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/810b8e3e-5836-4214-a141-c340d2220639-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dc6wj\" (UID: \"810b8e3e-5836-4214-a141-c340d2220639\") " pod="openshift-multus/multus-additional-cni-plugins-dc6wj"
Jun 02 15:46:33.933828 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.933249    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-systemd-units\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:33.933828 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.933285    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-host-cni-netd\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:33.954122 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.954096    2579 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 02 15:46:33.963526 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.963498    2579 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 02 15:46:33.983258 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.983216    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-01 15:41:32 +0000 UTC" deadline="2027-12-30 17:05:42.521807967 +0000 UTC"
Jun 02 15:46:33.983258 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:33.983255    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13825h19m8.538557042s"
Jun 02 15:46:34.037057 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.033538    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/73b94606-f892-4c47-9022-5d42f71bb77d-system-cni-dir\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:34.037057 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.033593    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-host-run-netns\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:34.037057 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.033617    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-host-cni-bin\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:34.037057 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.033623    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/73b94606-f892-4c47-9022-5d42f71bb77d-system-cni-dir\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:34.037057 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.033648    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/498f7354-865b-4ad1-9e9f-fabd57febd4a-etc-sysctl-conf\") pod \"tuned-mllz2\" (UID: \"498f7354-865b-4ad1-9e9f-fabd57febd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:34.037057 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.033671    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3bee98d4-d583-4ecb-b07b-9f422a9810da-host\") pod \"node-ca-cwjmt\" (UID: \"3bee98d4-d583-4ecb-b07b-9f422a9810da\") " pod="openshift-image-registry/node-ca-cwjmt"
Jun 02 15:46:34.037057 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.033682    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-host-cni-bin\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:34.037057 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.033696    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8l47g\" (UniqueName: \"kubernetes.io/projected/3bee98d4-d583-4ecb-b07b-9f422a9810da-kube-api-access-8l47g\") pod \"node-ca-cwjmt\" (UID: \"3bee98d4-d583-4ecb-b07b-9f422a9810da\") " pod="openshift-image-registry/node-ca-cwjmt"
Jun 02 15:46:34.037057 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.033719    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4290cf74-a174-4c3f-88b4-66b05bbe4750-tmp-dir\") pod \"node-resolver-th978\" (UID: \"4290cf74-a174-4c3f-88b4-66b05bbe4750\") " pod="openshift-dns/node-resolver-th978"
Jun 02 15:46:34.037057 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.033716    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-host-run-netns\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:34.037057 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.033738    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-host-kubelet\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:34.037057 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.033766    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-host-kubelet\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:34.037057 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.033779    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/498f7354-865b-4ad1-9e9f-fabd57febd4a-lib-modules\") pod \"tuned-mllz2\" (UID: \"498f7354-865b-4ad1-9e9f-fabd57febd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:34.037057 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.033800    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3bee98d4-d583-4ecb-b07b-9f422a9810da-host\") pod \"node-ca-cwjmt\" (UID: \"3bee98d4-d583-4ecb-b07b-9f422a9810da\") " pod="openshift-image-registry/node-ca-cwjmt"
Jun 02 15:46:34.037057 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.033806    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/73b94606-f892-4c47-9022-5d42f71bb77d-host-var-lib-cni-bin\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:34.037057 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.033815    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/498f7354-865b-4ad1-9e9f-fabd57febd4a-etc-sysctl-conf\") pod \"tuned-mllz2\" (UID: \"498f7354-865b-4ad1-9e9f-fabd57febd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:34.037057 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.033834    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bzbl5\" (UniqueName: \"kubernetes.io/projected/73b94606-f892-4c47-9022-5d42f71bb77d-kube-api-access-bzbl5\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:34.037057 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.033854    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/73b94606-f892-4c47-9022-5d42f71bb77d-host-var-lib-cni-bin\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:34.038038 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.033864    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca-metrics-certs\") pod \"network-metrics-daemon-6lvc8\" (UID: \"eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca\") " pod="openshift-multus/network-metrics-daemon-6lvc8"
Jun 02 15:46:34.038038 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.033890    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4290cf74-a174-4c3f-88b4-66b05bbe4750-hosts-file\") pod \"node-resolver-th978\" (UID: \"4290cf74-a174-4c3f-88b4-66b05bbe4750\") " pod="openshift-dns/node-resolver-th978"
Jun 02 15:46:34.038038 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.033914    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-x4d97\" (UniqueName: \"kubernetes.io/projected/4290cf74-a174-4c3f-88b4-66b05bbe4750-kube-api-access-x4d97\") pod \"node-resolver-th978\" (UID: \"4290cf74-a174-4c3f-88b4-66b05bbe4750\") " pod="openshift-dns/node-resolver-th978"
Jun 02 15:46:34.038038 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.033927    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/498f7354-865b-4ad1-9e9f-fabd57febd4a-lib-modules\") pod \"tuned-mllz2\" (UID: \"498f7354-865b-4ad1-9e9f-fabd57febd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:34.038038 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.033955    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-var-lib-openvswitch\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:34.038038 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.033976    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-run-ovn\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:34.038038 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.033994    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/498f7354-865b-4ad1-9e9f-fabd57febd4a-etc-sysconfig\") pod \"tuned-mllz2\" (UID: \"498f7354-865b-4ad1-9e9f-fabd57febd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:34.038038 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034001    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4290cf74-a174-4c3f-88b4-66b05bbe4750-hosts-file\") pod \"node-resolver-th978\" (UID: \"4290cf74-a174-4c3f-88b4-66b05bbe4750\") " pod="openshift-dns/node-resolver-th978"
Jun 02 15:46:34.038038 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034010    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/73b94606-f892-4c47-9022-5d42f71bb77d-cni-binary-copy\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:34.038038 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034026    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3bee98d4-d583-4ecb-b07b-9f422a9810da-serviceca\") pod \"node-ca-cwjmt\" (UID: \"3bee98d4-d583-4ecb-b07b-9f422a9810da\") " pod="openshift-image-registry/node-ca-cwjmt"
Jun 02 15:46:34.038038 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034042    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c0c551c2-7ea0-4de3-b24a-735cdb0c536d-host-slash\") pod \"iptables-alerter-zqsll\" (UID: \"c0c551c2-7ea0-4de3-b24a-735cdb0c536d\") " pod="openshift-network-operator/iptables-alerter-zqsll"
Jun 02 15:46:34.038038 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034051    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-var-lib-openvswitch\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:34.038038 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034075    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c0c551c2-7ea0-4de3-b24a-735cdb0c536d-host-slash\") pod \"iptables-alerter-zqsll\" (UID: \"c0c551c2-7ea0-4de3-b24a-735cdb0c536d\") " pod="openshift-network-operator/iptables-alerter-zqsll"
Jun 02 15:46:34.038038 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034114    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-run-ovn\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:34.038038 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:34.034124    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:34.038038 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034140    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4290cf74-a174-4c3f-88b4-66b05bbe4750-tmp-dir\") pod \"node-resolver-th978\" (UID: \"4290cf74-a174-4c3f-88b4-66b05bbe4750\") " pod="openshift-dns/node-resolver-th978"
Jun 02 15:46:34.038038 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034187    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/498f7354-865b-4ad1-9e9f-fabd57febd4a-etc-sysconfig\") pod \"tuned-mllz2\" (UID: \"498f7354-865b-4ad1-9e9f-fabd57febd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:34.038739 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:34.034230    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca-metrics-certs podName:eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca nodeName:}" failed. No retries permitted until 2026-06-02 15:46:34.534177208 +0000 UTC m=+3.060085639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca-metrics-certs") pod "network-metrics-daemon-6lvc8" (UID: "eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:34.038739 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034257    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6127e33c-aa77-437e-b443-e956037afedb-ovn-node-metrics-cert\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:34.038739 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034291    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zs6wj\" (UniqueName: \"kubernetes.io/projected/498f7354-865b-4ad1-9e9f-fabd57febd4a-kube-api-access-zs6wj\") pod \"tuned-mllz2\" (UID: \"498f7354-865b-4ad1-9e9f-fabd57febd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:34.038739 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034318    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/73b94606-f892-4c47-9022-5d42f71bb77d-host-var-lib-kubelet\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:34.038739 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034347    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73b94606-f892-4c47-9022-5d42f71bb77d-etc-kubernetes\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:34.038739 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034374    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/498f7354-865b-4ad1-9e9f-fabd57febd4a-etc-kubernetes\") pod \"tuned-mllz2\" (UID: \"498f7354-865b-4ad1-9e9f-fabd57febd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:34.038739 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034417    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/498f7354-865b-4ad1-9e9f-fabd57febd4a-etc-sysctl-d\") pod \"tuned-mllz2\" (UID: \"498f7354-865b-4ad1-9e9f-fabd57febd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:34.038739 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034441    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/498f7354-865b-4ad1-9e9f-fabd57febd4a-etc-systemd\") pod \"tuned-mllz2\" (UID: \"498f7354-865b-4ad1-9e9f-fabd57febd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:34.038739 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034440    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/73b94606-f892-4c47-9022-5d42f71bb77d-host-var-lib-kubelet\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:34.038739 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034468    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/73b94606-f892-4c47-9022-5d42f71bb77d-host-run-netns\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:34.038739 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034493    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-log-socket\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:34.038739 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034520    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/498f7354-865b-4ad1-9e9f-fabd57febd4a-etc-kubernetes\") pod \"tuned-mllz2\" (UID: \"498f7354-865b-4ad1-9e9f-fabd57febd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:34.038739 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034520    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bqndc\" (UniqueName: \"kubernetes.io/projected/6127e33c-aa77-437e-b443-e956037afedb-kube-api-access-bqndc\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:34.038739 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034561    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3bee98d4-d583-4ecb-b07b-9f422a9810da-serviceca\") pod \"node-ca-cwjmt\" (UID: \"3bee98d4-d583-4ecb-b07b-9f422a9810da\") " pod="openshift-image-registry/node-ca-cwjmt"
Jun 02 15:46:34.038739 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034558    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/498f7354-865b-4ad1-9e9f-fabd57febd4a-etc-systemd\") pod \"tuned-mllz2\" (UID: \"498f7354-865b-4ad1-9e9f-fabd57febd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:34.038739 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034577    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7b0e93e2-a6b5-4b5a-be51-e09f43184379-konnectivity-ca\") pod \"konnectivity-agent-zt7mm\" (UID: \"7b0e93e2-a6b5-4b5a-be51-e09f43184379\") " pod="kube-system/konnectivity-agent-zt7mm"
Jun 02 15:46:34.038739 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034620    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/498f7354-865b-4ad1-9e9f-fabd57febd4a-etc-modprobe-d\") pod \"tuned-mllz2\" (UID: \"498f7354-865b-4ad1-9e9f-fabd57febd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:34.039299 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034638    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73b94606-f892-4c47-9022-5d42f71bb77d-etc-kubernetes\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:34.039299 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034643    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/73b94606-f892-4c47-9022-5d42f71bb77d-hostroot\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:34.039299 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034662    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jun 02 15:46:34.039299 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034676    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/73b94606-f892-4c47-9022-5d42f71bb77d-multus-conf-dir\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:34.039299 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034688    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/73b94606-f892-4c47-9022-5d42f71bb77d-host-run-netns\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:34.039299 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034694    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/810b8e3e-5836-4214-a141-c340d2220639-os-release\") pod \"multus-additional-cni-plugins-dc6wj\" (UID: \"810b8e3e-5836-4214-a141-c340d2220639\") " pod="openshift-multus/multus-additional-cni-plugins-dc6wj"
Jun 02 15:46:34.039299 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034719    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-run-openvswitch\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:34.039299 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034739    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/810b8e3e-5836-4214-a141-c340d2220639-os-release\") pod \"multus-additional-cni-plugins-dc6wj\" (UID: \"810b8e3e-5836-4214-a141-c340d2220639\") " pod="openshift-multus/multus-additional-cni-plugins-dc6wj"
Jun 02 15:46:34.039299 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034745    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-host-run-ovn-kubernetes\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:34.039299 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034757    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/498f7354-865b-4ad1-9e9f-fabd57febd4a-etc-sysctl-d\") pod \"tuned-mllz2\" (UID: \"498f7354-865b-4ad1-9e9f-fabd57febd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:34.039299 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034760    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/498f7354-865b-4ad1-9e9f-fabd57febd4a-etc-modprobe-d\") pod \"tuned-mllz2\" (UID: \"498f7354-865b-4ad1-9e9f-fabd57febd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:34.039299 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034771    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a4a82f10-0682-47f3-b029-9b45d89383d3-socket-dir\") pod \"aws-ebs-csi-driver-node-bhfk7\" (UID: \"a4a82f10-0682-47f3-b029-9b45d89383d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhfk7"
Jun 02 15:46:34.039299 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034799    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/73b94606-f892-4c47-9022-5d42f71bb77d-multus-conf-dir\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:34.039299 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034802    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-log-socket\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:34.039299 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034803    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-run-openvswitch\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:34.039299 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034777    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/73b94606-f892-4c47-9022-5d42f71bb77d-hostroot\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:34.039299 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034805    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a4a82f10-0682-47f3-b029-9b45d89383d3-sys-fs\") pod \"aws-ebs-csi-driver-node-bhfk7\" (UID: \"a4a82f10-0682-47f3-b029-9b45d89383d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhfk7"
Jun 02 15:46:34.039299 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034675    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/73b94606-f892-4c47-9022-5d42f71bb77d-cni-binary-copy\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:34.039959 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034846    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-host-run-ovn-kubernetes\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:34.039959 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034854    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/810b8e3e-5836-4214-a141-c340d2220639-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dc6wj\" (UID: \"810b8e3e-5836-4214-a141-c340d2220639\") " pod="openshift-multus/multus-additional-cni-plugins-dc6wj"
Jun 02 15:46:34.039959 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034880    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a4a82f10-0682-47f3-b029-9b45d89383d3-sys-fs\") pod \"aws-ebs-csi-driver-node-bhfk7\" (UID: \"a4a82f10-0682-47f3-b029-9b45d89383d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhfk7"
Jun 02 15:46:34.039959 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034881    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-systemd-units\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:34.039959 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034904    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-host-cni-netd\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:34.039959 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034922    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/498f7354-865b-4ad1-9e9f-fabd57febd4a-etc-tuned\") pod \"tuned-mllz2\" (UID: \"498f7354-865b-4ad1-9e9f-fabd57febd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:34.039959 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034930    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-systemd-units\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:34.039959 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034938    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/73b94606-f892-4c47-9022-5d42f71bb77d-os-release\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:34.039959 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034964    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a4a82f10-0682-47f3-b029-9b45d89383d3-device-dir\") pod \"aws-ebs-csi-driver-node-bhfk7\" (UID: \"a4a82f10-0682-47f3-b029-9b45d89383d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhfk7"
Jun 02 15:46:34.039959 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034978    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/73b94606-f892-4c47-9022-5d42f71bb77d-os-release\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:34.039959 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034920    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a4a82f10-0682-47f3-b029-9b45d89383d3-socket-dir\") pod \"aws-ebs-csi-driver-node-bhfk7\" (UID: \"a4a82f10-0682-47f3-b029-9b45d89383d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhfk7"
Jun 02 15:46:34.039959 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034980    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-host-cni-netd\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:34.039959 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.034989    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/810b8e3e-5836-4214-a141-c340d2220639-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dc6wj\" (UID: \"810b8e3e-5836-4214-a141-c340d2220639\") " pod="openshift-multus/multus-additional-cni-plugins-dc6wj"
Jun 02 15:46:34.039959 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.035016    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/498f7354-865b-4ad1-9e9f-fabd57febd4a-tmp\") pod \"tuned-mllz2\" (UID: \"498f7354-865b-4ad1-9e9f-fabd57febd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:34.039959 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.035043    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qclv4\" (UniqueName: \"kubernetes.io/projected/a4a82f10-0682-47f3-b029-9b45d89383d3-kube-api-access-qclv4\") pod \"aws-ebs-csi-driver-node-bhfk7\" (UID: \"a4a82f10-0682-47f3-b029-9b45d89383d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhfk7"
Jun 02 15:46:34.039959 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.035053    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a4a82f10-0682-47f3-b029-9b45d89383d3-device-dir\") pod \"aws-ebs-csi-driver-node-bhfk7\" (UID: \"a4a82f10-0682-47f3-b029-9b45d89383d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhfk7"
Jun 02 15:46:34.039959 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.035071    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6127e33c-aa77-437e-b443-e956037afedb-ovnkube-script-lib\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:34.040450 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.035114    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/498f7354-865b-4ad1-9e9f-fabd57febd4a-sys\") pod \"tuned-mllz2\" (UID: \"498f7354-865b-4ad1-9e9f-fabd57febd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:34.040450 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.035136    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/498f7354-865b-4ad1-9e9f-fabd57febd4a-host\") pod \"tuned-mllz2\" (UID: \"498f7354-865b-4ad1-9e9f-fabd57febd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:34.040450 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.035161    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/73b94606-f892-4c47-9022-5d42f71bb77d-multus-cni-dir\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:34.040450 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.035189    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/73b94606-f892-4c47-9022-5d42f71bb77d-host-run-k8s-cni-cncf-io\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:34.040450 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.035223    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7b0e93e2-a6b5-4b5a-be51-e09f43184379-konnectivity-ca\") pod \"konnectivity-agent-zt7mm\" (UID: \"7b0e93e2-a6b5-4b5a-be51-e09f43184379\") " pod="kube-system/konnectivity-agent-zt7mm"
Jun 02 15:46:34.040450 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.035213    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/73b94606-f892-4c47-9022-5d42f71bb77d-multus-daemon-config\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:34.040450 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.035312    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/73b94606-f892-4c47-9022-5d42f71bb77d-host-run-k8s-cni-cncf-io\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:34.040450 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.035367    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/73b94606-f892-4c47-9022-5d42f71bb77d-multus-cni-dir\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:34.040450 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.035387    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/810b8e3e-5836-4214-a141-c340d2220639-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dc6wj\" (UID: \"810b8e3e-5836-4214-a141-c340d2220639\") " pod="openshift-multus/multus-additional-cni-plugins-dc6wj"
Jun 02 15:46:34.040450 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.035476    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/498f7354-865b-4ad1-9e9f-fabd57febd4a-sys\") pod \"tuned-mllz2\" (UID: \"498f7354-865b-4ad1-9e9f-fabd57febd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:34.040450 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.035508    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a4a82f10-0682-47f3-b029-9b45d89383d3-registration-dir\") pod \"aws-ebs-csi-driver-node-bhfk7\" (UID: \"a4a82f10-0682-47f3-b029-9b45d89383d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhfk7"
Jun 02 15:46:34.040450 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.035537    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/810b8e3e-5836-4214-a141-c340d2220639-cni-binary-copy\") pod \"multus-additional-cni-plugins-dc6wj\" (UID: \"810b8e3e-5836-4214-a141-c340d2220639\") " pod="openshift-multus/multus-additional-cni-plugins-dc6wj"
Jun 02 15:46:34.040450 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.035563    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-node-log\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:34.040450 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.035587    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/498f7354-865b-4ad1-9e9f-fabd57febd4a-run\") pod \"tuned-mllz2\" (UID: \"498f7354-865b-4ad1-9e9f-fabd57febd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:34.040450 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.035611    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/73b94606-f892-4c47-9022-5d42f71bb77d-host-run-multus-certs\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:34.040450 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.035636    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a4a82f10-0682-47f3-b029-9b45d89383d3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bhfk7\" (UID: \"a4a82f10-0682-47f3-b029-9b45d89383d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhfk7"
Jun 02 15:46:34.040450 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.035661    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a4a82f10-0682-47f3-b029-9b45d89383d3-etc-selinux\") pod \"aws-ebs-csi-driver-node-bhfk7\" (UID: \"a4a82f10-0682-47f3-b029-9b45d89383d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhfk7"
Jun 02 15:46:34.040891 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.035685    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d87j5\" (UniqueName: \"kubernetes.io/projected/43370289-88af-4e83-b409-4e8a98e09857-kube-api-access-d87j5\") pod \"network-check-target-zzt45\" (UID: \"43370289-88af-4e83-b409-4e8a98e09857\") " pod="openshift-network-diagnostics/network-check-target-zzt45"
Jun 02 15:46:34.040891 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.035688    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/810b8e3e-5836-4214-a141-c340d2220639-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dc6wj\" (UID: \"810b8e3e-5836-4214-a141-c340d2220639\") " pod="openshift-multus/multus-additional-cni-plugins-dc6wj"
Jun 02 15:46:34.040891 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.035712    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-host-slash\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:34.040891 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.035741    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/498f7354-865b-4ad1-9e9f-fabd57febd4a-host\") pod \"tuned-mllz2\" (UID: \"498f7354-865b-4ad1-9e9f-fabd57febd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:34.040891 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.035741    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-run-systemd\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:34.040891 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.035778    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-run-systemd\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:34.040891 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.035786    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/73b94606-f892-4c47-9022-5d42f71bb77d-multus-daemon-config\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:34.040891 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.035779    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:34.040891 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.035811    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:34.040891 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.035824    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a4a82f10-0682-47f3-b029-9b45d89383d3-registration-dir\") pod \"aws-ebs-csi-driver-node-bhfk7\" (UID: \"a4a82f10-0682-47f3-b029-9b45d89383d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhfk7"
Jun 02 15:46:34.040891 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.035841    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/73b94606-f892-4c47-9022-5d42f71bb77d-cnibin\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:34.040891 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.035856    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/73b94606-f892-4c47-9022-5d42f71bb77d-host-run-multus-certs\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:34.040891 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.035868    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a4a82f10-0682-47f3-b029-9b45d89383d3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bhfk7\" (UID: \"a4a82f10-0682-47f3-b029-9b45d89383d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhfk7"
Jun 02 15:46:34.040891 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.035867    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c0c551c2-7ea0-4de3-b24a-735cdb0c536d-iptables-alerter-script\") pod \"iptables-alerter-zqsll\" (UID: \"c0c551c2-7ea0-4de3-b24a-735cdb0c536d\") " pod="openshift-network-operator/iptables-alerter-zqsll"
Jun 02 15:46:34.040891 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.035920    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a4a82f10-0682-47f3-b029-9b45d89383d3-etc-selinux\") pod \"aws-ebs-csi-driver-node-bhfk7\" (UID: \"a4a82f10-0682-47f3-b029-9b45d89383d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhfk7"
Jun 02 15:46:34.040891 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.035955    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/498f7354-865b-4ad1-9e9f-fabd57febd4a-run\") pod \"tuned-mllz2\" (UID: \"498f7354-865b-4ad1-9e9f-fabd57febd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:34.040891 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.036093    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/73b94606-f892-4c47-9022-5d42f71bb77d-cnibin\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:34.041606 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.036107    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6127e33c-aa77-437e-b443-e956037afedb-ovnkube-script-lib\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:34.041606 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.036140    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-host-slash\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:34.041606 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.036155    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rk4bb\" (UniqueName: \"kubernetes.io/projected/c0c551c2-7ea0-4de3-b24a-735cdb0c536d-kube-api-access-rk4bb\") pod \"iptables-alerter-zqsll\" (UID: \"c0c551c2-7ea0-4de3-b24a-735cdb0c536d\") " pod="openshift-network-operator/iptables-alerter-zqsll"
Jun 02 15:46:34.041606 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.036186    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-v668f\" (UniqueName: \"kubernetes.io/projected/eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca-kube-api-access-v668f\") pod \"network-metrics-daemon-6lvc8\" (UID: \"eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca\") " pod="openshift-multus/network-metrics-daemon-6lvc8"
Jun 02 15:46:34.041606 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.036215    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/810b8e3e-5836-4214-a141-c340d2220639-system-cni-dir\") pod \"multus-additional-cni-plugins-dc6wj\" (UID: \"810b8e3e-5836-4214-a141-c340d2220639\") " pod="openshift-multus/multus-additional-cni-plugins-dc6wj"
Jun 02 15:46:34.041606 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.036241    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/810b8e3e-5836-4214-a141-c340d2220639-cnibin\") pod \"multus-additional-cni-plugins-dc6wj\" (UID: \"810b8e3e-5836-4214-a141-c340d2220639\") " pod="openshift-multus/multus-additional-cni-plugins-dc6wj"
Jun 02 15:46:34.041606 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.036266    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6127e33c-aa77-437e-b443-e956037afedb-ovnkube-config\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:34.041606 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.036323    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/498f7354-865b-4ad1-9e9f-fabd57febd4a-var-lib-kubelet\") pod \"tuned-mllz2\" (UID: \"498f7354-865b-4ad1-9e9f-fabd57febd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:34.041606 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.036348    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/810b8e3e-5836-4214-a141-c340d2220639-cni-binary-copy\") pod \"multus-additional-cni-plugins-dc6wj\" (UID: \"810b8e3e-5836-4214-a141-c340d2220639\") " pod="openshift-multus/multus-additional-cni-plugins-dc6wj"
Jun 02 15:46:34.041606 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.036361    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/73b94606-f892-4c47-9022-5d42f71bb77d-multus-socket-dir-parent\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:34.041606 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.036370    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/810b8e3e-5836-4214-a141-c340d2220639-system-cni-dir\") pod \"multus-additional-cni-plugins-dc6wj\" (UID: \"810b8e3e-5836-4214-a141-c340d2220639\") " pod="openshift-multus/multus-additional-cni-plugins-dc6wj"
Jun 02 15:46:34.041606 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.036418    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c0c551c2-7ea0-4de3-b24a-735cdb0c536d-iptables-alerter-script\") pod \"iptables-alerter-zqsll\" (UID: \"c0c551c2-7ea0-4de3-b24a-735cdb0c536d\") " pod="openshift-network-operator/iptables-alerter-zqsll"
Jun 02 15:46:34.041606 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.036419    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/73b94606-f892-4c47-9022-5d42f71bb77d-host-var-lib-cni-multus\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:34.041606 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.036463    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/810b8e3e-5836-4214-a141-c340d2220639-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dc6wj\" (UID: \"810b8e3e-5836-4214-a141-c340d2220639\") " pod="openshift-multus/multus-additional-cni-plugins-dc6wj"
Jun 02 15:46:34.041606 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.036480    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/73b94606-f892-4c47-9022-5d42f71bb77d-multus-socket-dir-parent\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:34.041606 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.036506    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-node-log\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:34.041606 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.036531    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/498f7354-865b-4ad1-9e9f-fabd57febd4a-var-lib-kubelet\") pod \"tuned-mllz2\" (UID: \"498f7354-865b-4ad1-9e9f-fabd57febd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:34.042283 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.036540    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hq2ml\" (UniqueName: \"kubernetes.io/projected/810b8e3e-5836-4214-a141-c340d2220639-kube-api-access-hq2ml\") pod \"multus-additional-cni-plugins-dc6wj\" (UID: \"810b8e3e-5836-4214-a141-c340d2220639\") " pod="openshift-multus/multus-additional-cni-plugins-dc6wj"
Jun 02 15:46:34.042283 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.036581    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-etc-openvswitch\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:34.042283 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.036607    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6127e33c-aa77-437e-b443-e956037afedb-env-overrides\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:34.042283 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.036610    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/73b94606-f892-4c47-9022-5d42f71bb77d-host-var-lib-cni-multus\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:34.042283 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.036632    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7b0e93e2-a6b5-4b5a-be51-e09f43184379-agent-certs\") pod \"konnectivity-agent-zt7mm\" (UID: \"7b0e93e2-a6b5-4b5a-be51-e09f43184379\") " pod="kube-system/konnectivity-agent-zt7mm"
Jun 02 15:46:34.042283 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.036756    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/810b8e3e-5836-4214-a141-c340d2220639-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dc6wj\" (UID: \"810b8e3e-5836-4214-a141-c340d2220639\") " pod="openshift-multus/multus-additional-cni-plugins-dc6wj"
Jun 02 15:46:34.042283 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.036807    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6127e33c-aa77-437e-b443-e956037afedb-etc-openvswitch\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:34.042283 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.036843    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6127e33c-aa77-437e-b443-e956037afedb-ovnkube-config\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:34.042283 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.036876    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/810b8e3e-5836-4214-a141-c340d2220639-cnibin\") pod \"multus-additional-cni-plugins-dc6wj\" (UID: \"810b8e3e-5836-4214-a141-c340d2220639\") " pod="openshift-multus/multus-additional-cni-plugins-dc6wj"
Jun 02 15:46:34.042283 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.037799    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6127e33c-aa77-437e-b443-e956037afedb-env-overrides\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:34.042283 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.038232    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/498f7354-865b-4ad1-9e9f-fabd57febd4a-etc-tuned\") pod \"tuned-mllz2\" (UID: \"498f7354-865b-4ad1-9e9f-fabd57febd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:34.042283 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.038451    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/498f7354-865b-4ad1-9e9f-fabd57febd4a-tmp\") pod \"tuned-mllz2\" (UID: \"498f7354-865b-4ad1-9e9f-fabd57febd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:34.042283 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.038863    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6127e33c-aa77-437e-b443-e956037afedb-ovn-node-metrics-cert\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:34.042283 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.039616    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7b0e93e2-a6b5-4b5a-be51-e09f43184379-agent-certs\") pod \"konnectivity-agent-zt7mm\" (UID: \"7b0e93e2-a6b5-4b5a-be51-e09f43184379\") " pod="kube-system/konnectivity-agent-zt7mm"
Jun 02 15:46:34.044334 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.044302    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzbl5\" (UniqueName: \"kubernetes.io/projected/73b94606-f892-4c47-9022-5d42f71bb77d-kube-api-access-bzbl5\") pod \"multus-lg8nx\" (UID: \"73b94606-f892-4c47-9022-5d42f71bb77d\") " pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:34.044475 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.044302    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4d97\" (UniqueName: \"kubernetes.io/projected/4290cf74-a174-4c3f-88b4-66b05bbe4750-kube-api-access-x4d97\") pod \"node-resolver-th978\" (UID: \"4290cf74-a174-4c3f-88b4-66b05bbe4750\") " pod="openshift-dns/node-resolver-th978"
Jun 02 15:46:34.044620 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:34.044598    2579 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 02 15:46:34.044683 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:34.044627    2579 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 02 15:46:34.044683 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:34.044642    2579 projected.go:194] Error preparing data for projected volume kube-api-access-d87j5 for pod openshift-network-diagnostics/network-check-target-zzt45: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:34.044836 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:34.044715    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43370289-88af-4e83-b409-4e8a98e09857-kube-api-access-d87j5 podName:43370289-88af-4e83-b409-4e8a98e09857 nodeName:}" failed. No retries permitted until 2026-06-02 15:46:34.544694983 +0000 UTC m=+3.070603393 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-d87j5" (UniqueName: "kubernetes.io/projected/43370289-88af-4e83-b409-4e8a98e09857-kube-api-access-d87j5") pod "network-check-target-zzt45" (UID: "43370289-88af-4e83-b409-4e8a98e09857") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:34.045698 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.045604    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk4bb\" (UniqueName: \"kubernetes.io/projected/c0c551c2-7ea0-4de3-b24a-735cdb0c536d-kube-api-access-rk4bb\") pod \"iptables-alerter-zqsll\" (UID: \"c0c551c2-7ea0-4de3-b24a-735cdb0c536d\") " pod="openshift-network-operator/iptables-alerter-zqsll"
Jun 02 15:46:34.045698 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.045698    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l47g\" (UniqueName: \"kubernetes.io/projected/3bee98d4-d583-4ecb-b07b-9f422a9810da-kube-api-access-8l47g\") pod \"node-ca-cwjmt\" (UID: \"3bee98d4-d583-4ecb-b07b-9f422a9810da\") " pod="openshift-image-registry/node-ca-cwjmt"
Jun 02 15:46:34.046389 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.046367    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqndc\" (UniqueName: \"kubernetes.io/projected/6127e33c-aa77-437e-b443-e956037afedb-kube-api-access-bqndc\") pod \"ovnkube-node-lqx9k\" (UID: \"6127e33c-aa77-437e-b443-e956037afedb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:34.046972 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.046548    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs6wj\" (UniqueName: \"kubernetes.io/projected/498f7354-865b-4ad1-9e9f-fabd57febd4a-kube-api-access-zs6wj\") pod \"tuned-mllz2\" (UID: \"498f7354-865b-4ad1-9e9f-fabd57febd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:34.046972 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.046924    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq2ml\" (UniqueName: \"kubernetes.io/projected/810b8e3e-5836-4214-a141-c340d2220639-kube-api-access-hq2ml\") pod \"multus-additional-cni-plugins-dc6wj\" (UID: \"810b8e3e-5836-4214-a141-c340d2220639\") " pod="openshift-multus/multus-additional-cni-plugins-dc6wj"
Jun 02 15:46:34.047587 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.047571    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v668f\" (UniqueName: \"kubernetes.io/projected/eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca-kube-api-access-v668f\") pod \"network-metrics-daemon-6lvc8\" (UID: \"eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca\") " pod="openshift-multus/network-metrics-daemon-6lvc8"
Jun 02 15:46:34.048525 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.048510    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qclv4\" (UniqueName: \"kubernetes.io/projected/a4a82f10-0682-47f3-b029-9b45d89383d3-kube-api-access-qclv4\") pod \"aws-ebs-csi-driver-node-bhfk7\" (UID: \"a4a82f10-0682-47f3-b029-9b45d89383d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhfk7"
Jun 02 15:46:34.203330 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.203252    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mllz2"
Jun 02 15:46:34.212088 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.212059    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lg8nx"
Jun 02 15:46:34.220806 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.220783    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhfk7"
Jun 02 15:46:34.225513 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.225487    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-th978"
Jun 02 15:46:34.232146 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.232121    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cwjmt"
Jun 02 15:46:34.241276 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.241251    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dc6wj"
Jun 02 15:46:34.245981 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.245963    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zqsll"
Jun 02 15:46:34.252588 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.252567    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zt7mm"
Jun 02 15:46:34.258263 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.258244    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:46:34.541745 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.541629    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca-metrics-certs\") pod \"network-metrics-daemon-6lvc8\" (UID: \"eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca\") " pod="openshift-multus/network-metrics-daemon-6lvc8"
Jun 02 15:46:34.541881 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:34.541847    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:34.541932 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:34.541907    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca-metrics-certs podName:eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca nodeName:}" failed. No retries permitted until 2026-06-02 15:46:35.541888807 +0000 UTC m=+4.067797230 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca-metrics-certs") pod "network-metrics-daemon-6lvc8" (UID: "eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:34.547897 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:34.547668    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4a82f10_0682_47f3_b029_9b45d89383d3.slice/crio-973796ade989ae97d1849d458206e69bf74bd3a6ef51e873c7ce71004933f27e WatchSource:0}: Error finding container 973796ade989ae97d1849d458206e69bf74bd3a6ef51e873c7ce71004933f27e: Status 404 returned error can't find the container with id 973796ade989ae97d1849d458206e69bf74bd3a6ef51e873c7ce71004933f27e
Jun 02 15:46:34.548763 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:34.548737    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bee98d4_d583_4ecb_b07b_9f422a9810da.slice/crio-5a064d5abe66069bd79346abeeb421d57d5f2a1786382fff786ee1013d4cc49b WatchSource:0}: Error finding container 5a064d5abe66069bd79346abeeb421d57d5f2a1786382fff786ee1013d4cc49b: Status 404 returned error can't find the container with id 5a064d5abe66069bd79346abeeb421d57d5f2a1786382fff786ee1013d4cc49b
Jun 02 15:46:34.553740 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:34.553719    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b0e93e2_a6b5_4b5a_be51_e09f43184379.slice/crio-6794c75baaa33c68de1ff048eee97ba1d0ad28bc13782598fffe65a705f0e57a WatchSource:0}: Error finding container 6794c75baaa33c68de1ff048eee97ba1d0ad28bc13782598fffe65a705f0e57a: Status 404 returned error can't find the container with id 6794c75baaa33c68de1ff048eee97ba1d0ad28bc13782598fffe65a705f0e57a
Jun 02 15:46:34.554630 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:34.554597    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73b94606_f892_4c47_9022_5d42f71bb77d.slice/crio-20dc3b5e3302ecce50316666ec913f35326aafb83fbe5786ea8ed9f554cbcd7a WatchSource:0}: Error finding container 20dc3b5e3302ecce50316666ec913f35326aafb83fbe5786ea8ed9f554cbcd7a: Status 404 returned error can't find the container with id 20dc3b5e3302ecce50316666ec913f35326aafb83fbe5786ea8ed9f554cbcd7a
Jun 02 15:46:34.555624 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:34.555603    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6127e33c_aa77_437e_b443_e956037afedb.slice/crio-5c8dadc7b49e89c2fa9921d33cb47238a1b37f47ba0bdfec7fac9bcdb5417b82 WatchSource:0}: Error finding container 5c8dadc7b49e89c2fa9921d33cb47238a1b37f47ba0bdfec7fac9bcdb5417b82: Status 404 returned error can't find the container with id 5c8dadc7b49e89c2fa9921d33cb47238a1b37f47ba0bdfec7fac9bcdb5417b82
Jun 02 15:46:34.557162 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:34.557140    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod810b8e3e_5836_4214_a141_c340d2220639.slice/crio-f07fbad0bc0aa0b09bf155fe7eda71ac3bffbc2f26a946645be4ae74a8340f6c WatchSource:0}: Error finding container f07fbad0bc0aa0b09bf155fe7eda71ac3bffbc2f26a946645be4ae74a8340f6c: Status 404 returned error can't find the container with id f07fbad0bc0aa0b09bf155fe7eda71ac3bffbc2f26a946645be4ae74a8340f6c
Jun 02 15:46:34.561525 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:34.561501    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0c551c2_7ea0_4de3_b24a_735cdb0c536d.slice/crio-989492242e8e2325b0fbccc73091d26dbd2d61bde43a70c96ce07bc0dad328dd WatchSource:0}: Error finding container 989492242e8e2325b0fbccc73091d26dbd2d61bde43a70c96ce07bc0dad328dd: Status 404 returned error can't find the container with id 989492242e8e2325b0fbccc73091d26dbd2d61bde43a70c96ce07bc0dad328dd
Jun 02 15:46:34.563699 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:46:34.563675    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod498f7354_865b_4ad1_9e9f_fabd57febd4a.slice/crio-76ef85d883cdf3eea41d70575b9632a733408dad99fff3de2dc0860afe14e4cb WatchSource:0}: Error finding container 76ef85d883cdf3eea41d70575b9632a733408dad99fff3de2dc0860afe14e4cb: Status 404 returned error can't find the container with id 76ef85d883cdf3eea41d70575b9632a733408dad99fff3de2dc0860afe14e4cb
Jun 02 15:46:34.642824 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.642796    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d87j5\" (UniqueName: \"kubernetes.io/projected/43370289-88af-4e83-b409-4e8a98e09857-kube-api-access-d87j5\") pod \"network-check-target-zzt45\" (UID: \"43370289-88af-4e83-b409-4e8a98e09857\") " pod="openshift-network-diagnostics/network-check-target-zzt45"
Jun 02 15:46:34.642975 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:34.642953    2579 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 02 15:46:34.643021 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:34.642975    2579 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 02 15:46:34.643021 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:34.642985    2579 projected.go:194] Error preparing data for projected volume kube-api-access-d87j5 for pod openshift-network-diagnostics/network-check-target-zzt45: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:34.643085 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:34.643037    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43370289-88af-4e83-b409-4e8a98e09857-kube-api-access-d87j5 podName:43370289-88af-4e83-b409-4e8a98e09857 nodeName:}" failed. No retries permitted until 2026-06-02 15:46:35.643019994 +0000 UTC m=+4.168928405 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-d87j5" (UniqueName: "kubernetes.io/projected/43370289-88af-4e83-b409-4e8a98e09857-kube-api-access-d87j5") pod "network-check-target-zzt45" (UID: "43370289-88af-4e83-b409-4e8a98e09857") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:34.984232 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.984164    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-01 15:41:32 +0000 UTC" deadline="2027-11-21 20:24:48.321497564 +0000 UTC"
Jun 02 15:46:34.984232 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:34.984220    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12892h38m13.33728158s"
Jun 02 15:46:35.059970 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:35.059942    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6lvc8"
Jun 02 15:46:35.060135 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:35.060082    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6lvc8" podUID="eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca"
Jun 02 15:46:35.074608 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:35.074527    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-156.ec2.internal" event={"ID":"42688cc70769c2fbb9ca611c4f39c7ef","Type":"ContainerStarted","Data":"da51fbd7a9e1ac1e7c46673bde78044d51f8756ef79b1d2197e7d25d4f024d69"}
Jun 02 15:46:35.087556 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:35.087360    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mllz2" event={"ID":"498f7354-865b-4ad1-9e9f-fabd57febd4a","Type":"ContainerStarted","Data":"76ef85d883cdf3eea41d70575b9632a733408dad99fff3de2dc0860afe14e4cb"}
Jun 02 15:46:35.091855 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:35.091796    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-156.ec2.internal" podStartSLOduration=2.091780025 podStartE2EDuration="2.091780025s" podCreationTimestamp="2026-06-02 15:46:33 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 15:46:35.091564589 +0000 UTC m=+3.617473046" watchObservedRunningTime="2026-06-02 15:46:35.091780025 +0000 UTC m=+3.617688458"
Jun 02 15:46:35.095158 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:35.095103    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dc6wj" event={"ID":"810b8e3e-5836-4214-a141-c340d2220639","Type":"ContainerStarted","Data":"f07fbad0bc0aa0b09bf155fe7eda71ac3bffbc2f26a946645be4ae74a8340f6c"}
Jun 02 15:46:35.097542 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:35.097478    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k" event={"ID":"6127e33c-aa77-437e-b443-e956037afedb","Type":"ContainerStarted","Data":"5c8dadc7b49e89c2fa9921d33cb47238a1b37f47ba0bdfec7fac9bcdb5417b82"}
Jun 02 15:46:35.100237 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:35.100197    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lg8nx" event={"ID":"73b94606-f892-4c47-9022-5d42f71bb77d","Type":"ContainerStarted","Data":"20dc3b5e3302ecce50316666ec913f35326aafb83fbe5786ea8ed9f554cbcd7a"}
Jun 02 15:46:35.104896 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:35.104830    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cwjmt" event={"ID":"3bee98d4-d583-4ecb-b07b-9f422a9810da","Type":"ContainerStarted","Data":"5a064d5abe66069bd79346abeeb421d57d5f2a1786382fff786ee1013d4cc49b"}
Jun 02 15:46:35.115002 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:35.114969    2579 generic.go:358] "Generic (PLEG): container finished" podID="23859d63aa1dc8f73c323540603ab2ba" containerID="250774d587d1b998c20d0948732e9f056c2005c3af32b300c5d2179f665b578c" exitCode=0
Jun 02 15:46:35.115117 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:35.115052    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-156.ec2.internal" event={"ID":"23859d63aa1dc8f73c323540603ab2ba","Type":"ContainerDied","Data":"250774d587d1b998c20d0948732e9f056c2005c3af32b300c5d2179f665b578c"}
Jun 02 15:46:35.125305 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:35.125276    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zqsll" event={"ID":"c0c551c2-7ea0-4de3-b24a-735cdb0c536d","Type":"ContainerStarted","Data":"989492242e8e2325b0fbccc73091d26dbd2d61bde43a70c96ce07bc0dad328dd"}
Jun 02 15:46:35.133360 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:35.133327    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-th978" event={"ID":"4290cf74-a174-4c3f-88b4-66b05bbe4750","Type":"ContainerStarted","Data":"f747175ec6fcec169cdfef53d597c0c707503602c2105db4e9e2bff6bfababc7"}
Jun 02 15:46:35.145255 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:35.145214    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zt7mm" event={"ID":"7b0e93e2-a6b5-4b5a-be51-e09f43184379","Type":"ContainerStarted","Data":"6794c75baaa33c68de1ff048eee97ba1d0ad28bc13782598fffe65a705f0e57a"}
Jun 02 15:46:35.153981 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:35.153912    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhfk7" event={"ID":"a4a82f10-0682-47f3-b029-9b45d89383d3","Type":"ContainerStarted","Data":"973796ade989ae97d1849d458206e69bf74bd3a6ef51e873c7ce71004933f27e"}
Jun 02 15:46:35.562375 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:35.562335    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca-metrics-certs\") pod \"network-metrics-daemon-6lvc8\" (UID: \"eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca\") " pod="openshift-multus/network-metrics-daemon-6lvc8"
Jun 02 15:46:35.562587 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:35.562552    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:35.562691 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:35.562677    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca-metrics-certs podName:eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca nodeName:}" failed. No retries permitted until 2026-06-02 15:46:37.562655212 +0000 UTC m=+6.088563635 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca-metrics-certs") pod "network-metrics-daemon-6lvc8" (UID: "eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:35.663390 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:35.663299    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d87j5\" (UniqueName: \"kubernetes.io/projected/43370289-88af-4e83-b409-4e8a98e09857-kube-api-access-d87j5\") pod \"network-check-target-zzt45\" (UID: \"43370289-88af-4e83-b409-4e8a98e09857\") " pod="openshift-network-diagnostics/network-check-target-zzt45"
Jun 02 15:46:35.663575 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:35.663495    2579 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 02 15:46:35.663575 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:35.663515    2579 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 02 15:46:35.663575 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:35.663527    2579 projected.go:194] Error preparing data for projected volume kube-api-access-d87j5 for pod openshift-network-diagnostics/network-check-target-zzt45: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:35.663734 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:35.663582    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43370289-88af-4e83-b409-4e8a98e09857-kube-api-access-d87j5 podName:43370289-88af-4e83-b409-4e8a98e09857 nodeName:}" failed. No retries permitted until 2026-06-02 15:46:37.663567983 +0000 UTC m=+6.189476392 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-d87j5" (UniqueName: "kubernetes.io/projected/43370289-88af-4e83-b409-4e8a98e09857-kube-api-access-d87j5") pod "network-check-target-zzt45" (UID: "43370289-88af-4e83-b409-4e8a98e09857") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:35.797098 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:35.797059    2579 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 02 15:46:36.062520 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:36.062441    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzt45"
Jun 02 15:46:36.062977 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:36.062573    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzt45" podUID="43370289-88af-4e83-b409-4e8a98e09857"
Jun 02 15:46:36.194809 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:36.194773    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-156.ec2.internal" event={"ID":"23859d63aa1dc8f73c323540603ab2ba","Type":"ContainerStarted","Data":"ce5dbb2f76edbfe7fbe5f1c479bde16fb618cfcd519b458f078f7d4f790ee558"}
Jun 02 15:46:37.059364 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:37.059325    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6lvc8"
Jun 02 15:46:37.059578 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:37.059497    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6lvc8" podUID="eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca"
Jun 02 15:46:37.578741 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:37.578700    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca-metrics-certs\") pod \"network-metrics-daemon-6lvc8\" (UID: \"eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca\") " pod="openshift-multus/network-metrics-daemon-6lvc8"
Jun 02 15:46:37.579240 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:37.578904    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:37.579240 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:37.578974    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca-metrics-certs podName:eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca nodeName:}" failed. No retries permitted until 2026-06-02 15:46:41.578952357 +0000 UTC m=+10.104860785 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca-metrics-certs") pod "network-metrics-daemon-6lvc8" (UID: "eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:37.679761 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:37.679723    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d87j5\" (UniqueName: \"kubernetes.io/projected/43370289-88af-4e83-b409-4e8a98e09857-kube-api-access-d87j5\") pod \"network-check-target-zzt45\" (UID: \"43370289-88af-4e83-b409-4e8a98e09857\") " pod="openshift-network-diagnostics/network-check-target-zzt45"
Jun 02 15:46:37.679920 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:37.679874    2579 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 02 15:46:37.679920 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:37.679891    2579 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 02 15:46:37.679920 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:37.679902    2579 projected.go:194] Error preparing data for projected volume kube-api-access-d87j5 for pod openshift-network-diagnostics/network-check-target-zzt45: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:37.680028 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:37.679951    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43370289-88af-4e83-b409-4e8a98e09857-kube-api-access-d87j5 podName:43370289-88af-4e83-b409-4e8a98e09857 nodeName:}" failed. No retries permitted until 2026-06-02 15:46:41.67993802 +0000 UTC m=+10.205846432 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-d87j5" (UniqueName: "kubernetes.io/projected/43370289-88af-4e83-b409-4e8a98e09857-kube-api-access-d87j5") pod "network-check-target-zzt45" (UID: "43370289-88af-4e83-b409-4e8a98e09857") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:38.059753 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:38.059673    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzt45"
Jun 02 15:46:38.059907 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:38.059828    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzt45" podUID="43370289-88af-4e83-b409-4e8a98e09857"
Jun 02 15:46:39.059435 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:39.059382    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6lvc8"
Jun 02 15:46:39.059885 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:39.059551    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6lvc8" podUID="eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca"
Jun 02 15:46:40.060354 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:40.059831    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzt45"
Jun 02 15:46:40.060354 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:40.059953    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzt45" podUID="43370289-88af-4e83-b409-4e8a98e09857"
Jun 02 15:46:41.060658 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:41.060100    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6lvc8"
Jun 02 15:46:41.060658 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:41.060259    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6lvc8" podUID="eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca"
Jun 02 15:46:41.617447 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:41.617384    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca-metrics-certs\") pod \"network-metrics-daemon-6lvc8\" (UID: \"eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca\") " pod="openshift-multus/network-metrics-daemon-6lvc8"
Jun 02 15:46:41.617645 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:41.617571    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:41.617731 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:41.617648    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca-metrics-certs podName:eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca nodeName:}" failed. No retries permitted until 2026-06-02 15:46:49.617620191 +0000 UTC m=+18.143528615 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca-metrics-certs") pod "network-metrics-daemon-6lvc8" (UID: "eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:41.718815 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:41.718775    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d87j5\" (UniqueName: \"kubernetes.io/projected/43370289-88af-4e83-b409-4e8a98e09857-kube-api-access-d87j5\") pod \"network-check-target-zzt45\" (UID: \"43370289-88af-4e83-b409-4e8a98e09857\") " pod="openshift-network-diagnostics/network-check-target-zzt45"
Jun 02 15:46:41.719010 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:41.718973    2579 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 02 15:46:41.719010 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:41.718991    2579 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 02 15:46:41.719010 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:41.719004    2579 projected.go:194] Error preparing data for projected volume kube-api-access-d87j5 for pod openshift-network-diagnostics/network-check-target-zzt45: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:41.719180 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:41.719060    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43370289-88af-4e83-b409-4e8a98e09857-kube-api-access-d87j5 podName:43370289-88af-4e83-b409-4e8a98e09857 nodeName:}" failed. No retries permitted until 2026-06-02 15:46:49.719041296 +0000 UTC m=+18.244949705 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-d87j5" (UniqueName: "kubernetes.io/projected/43370289-88af-4e83-b409-4e8a98e09857-kube-api-access-d87j5") pod "network-check-target-zzt45" (UID: "43370289-88af-4e83-b409-4e8a98e09857") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:42.059945 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:42.059870    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzt45"
Jun 02 15:46:42.060097 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:42.059965    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzt45" podUID="43370289-88af-4e83-b409-4e8a98e09857"
Jun 02 15:46:43.059961 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:43.059923    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6lvc8"
Jun 02 15:46:43.060393 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:43.060081    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6lvc8" podUID="eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca"
Jun 02 15:46:43.931868 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:43.931802    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-156.ec2.internal" podStartSLOduration=10.931782465 podStartE2EDuration="10.931782465s" podCreationTimestamp="2026-06-02 15:46:33 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 15:46:36.208588166 +0000 UTC m=+4.734496598" watchObservedRunningTime="2026-06-02 15:46:43.931782465 +0000 UTC m=+12.457690897"
Jun 02 15:46:43.932141 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:43.932121    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-vh7pj"]
Jun 02 15:46:43.937383 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:43.937356    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vh7pj"
Jun 02 15:46:43.937559 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:43.937461    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vh7pj" podUID="32e4470f-53cd-4fda-b78b-404f9e5d45c3"
Jun 02 15:46:44.039926 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:44.039891    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/32e4470f-53cd-4fda-b78b-404f9e5d45c3-kubelet-config\") pod \"global-pull-secret-syncer-vh7pj\" (UID: \"32e4470f-53cd-4fda-b78b-404f9e5d45c3\") " pod="kube-system/global-pull-secret-syncer-vh7pj"
Jun 02 15:46:44.039926 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:44.039931    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32e4470f-53cd-4fda-b78b-404f9e5d45c3-original-pull-secret\") pod \"global-pull-secret-syncer-vh7pj\" (UID: \"32e4470f-53cd-4fda-b78b-404f9e5d45c3\") " pod="kube-system/global-pull-secret-syncer-vh7pj"
Jun 02 15:46:44.040127 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:44.039990    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/32e4470f-53cd-4fda-b78b-404f9e5d45c3-dbus\") pod \"global-pull-secret-syncer-vh7pj\" (UID: \"32e4470f-53cd-4fda-b78b-404f9e5d45c3\") " pod="kube-system/global-pull-secret-syncer-vh7pj"
Jun 02 15:46:44.060009 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:44.059974    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzt45"
Jun 02 15:46:44.060469 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:44.060107    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzt45" podUID="43370289-88af-4e83-b409-4e8a98e09857"
Jun 02 15:46:44.140388 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:44.140347    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/32e4470f-53cd-4fda-b78b-404f9e5d45c3-kubelet-config\") pod \"global-pull-secret-syncer-vh7pj\" (UID: \"32e4470f-53cd-4fda-b78b-404f9e5d45c3\") " pod="kube-system/global-pull-secret-syncer-vh7pj"
Jun 02 15:46:44.140388 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:44.140391    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32e4470f-53cd-4fda-b78b-404f9e5d45c3-original-pull-secret\") pod \"global-pull-secret-syncer-vh7pj\" (UID: \"32e4470f-53cd-4fda-b78b-404f9e5d45c3\") " pod="kube-system/global-pull-secret-syncer-vh7pj"
Jun 02 15:46:44.140637 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:44.140481    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/32e4470f-53cd-4fda-b78b-404f9e5d45c3-kubelet-config\") pod \"global-pull-secret-syncer-vh7pj\" (UID: \"32e4470f-53cd-4fda-b78b-404f9e5d45c3\") " pod="kube-system/global-pull-secret-syncer-vh7pj"
Jun 02 15:46:44.140637 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:44.140530    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/32e4470f-53cd-4fda-b78b-404f9e5d45c3-dbus\") pod \"global-pull-secret-syncer-vh7pj\" (UID: \"32e4470f-53cd-4fda-b78b-404f9e5d45c3\") " pod="kube-system/global-pull-secret-syncer-vh7pj"
Jun 02 15:46:44.140637 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:44.140486    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 02 15:46:44.140637 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:44.140589    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32e4470f-53cd-4fda-b78b-404f9e5d45c3-original-pull-secret podName:32e4470f-53cd-4fda-b78b-404f9e5d45c3 nodeName:}" failed. No retries permitted until 2026-06-02 15:46:44.640572209 +0000 UTC m=+13.166480622 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32e4470f-53cd-4fda-b78b-404f9e5d45c3-original-pull-secret") pod "global-pull-secret-syncer-vh7pj" (UID: "32e4470f-53cd-4fda-b78b-404f9e5d45c3") : object "kube-system"/"original-pull-secret" not registered
Jun 02 15:46:44.140762 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:44.140667    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/32e4470f-53cd-4fda-b78b-404f9e5d45c3-dbus\") pod \"global-pull-secret-syncer-vh7pj\" (UID: \"32e4470f-53cd-4fda-b78b-404f9e5d45c3\") " pod="kube-system/global-pull-secret-syncer-vh7pj"
Jun 02 15:46:44.645299 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:44.645263    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32e4470f-53cd-4fda-b78b-404f9e5d45c3-original-pull-secret\") pod \"global-pull-secret-syncer-vh7pj\" (UID: \"32e4470f-53cd-4fda-b78b-404f9e5d45c3\") " pod="kube-system/global-pull-secret-syncer-vh7pj"
Jun 02 15:46:44.645496 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:44.645454    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 02 15:46:44.645545 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:44.645540    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32e4470f-53cd-4fda-b78b-404f9e5d45c3-original-pull-secret podName:32e4470f-53cd-4fda-b78b-404f9e5d45c3 nodeName:}" failed. No retries permitted until 2026-06-02 15:46:45.645521899 +0000 UTC m=+14.171430321 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32e4470f-53cd-4fda-b78b-404f9e5d45c3-original-pull-secret") pod "global-pull-secret-syncer-vh7pj" (UID: "32e4470f-53cd-4fda-b78b-404f9e5d45c3") : object "kube-system"/"original-pull-secret" not registered
Jun 02 15:46:45.060160 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:45.060082    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6lvc8"
Jun 02 15:46:45.060529 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:45.060184    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6lvc8" podUID="eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca"
Jun 02 15:46:45.653416 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:45.653313    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32e4470f-53cd-4fda-b78b-404f9e5d45c3-original-pull-secret\") pod \"global-pull-secret-syncer-vh7pj\" (UID: \"32e4470f-53cd-4fda-b78b-404f9e5d45c3\") " pod="kube-system/global-pull-secret-syncer-vh7pj"
Jun 02 15:46:45.653568 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:45.653527    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 02 15:46:45.653646 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:45.653622    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32e4470f-53cd-4fda-b78b-404f9e5d45c3-original-pull-secret podName:32e4470f-53cd-4fda-b78b-404f9e5d45c3 nodeName:}" failed. No retries permitted until 2026-06-02 15:46:47.653598695 +0000 UTC m=+16.179507119 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32e4470f-53cd-4fda-b78b-404f9e5d45c3-original-pull-secret") pod "global-pull-secret-syncer-vh7pj" (UID: "32e4470f-53cd-4fda-b78b-404f9e5d45c3") : object "kube-system"/"original-pull-secret" not registered
Jun 02 15:46:46.059530 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:46.059443    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vh7pj"
Jun 02 15:46:46.059530 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:46.059488    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzt45"
Jun 02 15:46:46.059776 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:46.059577    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vh7pj" podUID="32e4470f-53cd-4fda-b78b-404f9e5d45c3"
Jun 02 15:46:46.060042 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:46.060011    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzt45" podUID="43370289-88af-4e83-b409-4e8a98e09857"
Jun 02 15:46:47.059375 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:47.059333    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6lvc8"
Jun 02 15:46:47.059796 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:47.059490    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6lvc8" podUID="eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca"
Jun 02 15:46:47.669515 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:47.669467    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32e4470f-53cd-4fda-b78b-404f9e5d45c3-original-pull-secret\") pod \"global-pull-secret-syncer-vh7pj\" (UID: \"32e4470f-53cd-4fda-b78b-404f9e5d45c3\") " pod="kube-system/global-pull-secret-syncer-vh7pj"
Jun 02 15:46:47.669717 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:47.669637    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 02 15:46:47.669769 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:47.669717    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32e4470f-53cd-4fda-b78b-404f9e5d45c3-original-pull-secret podName:32e4470f-53cd-4fda-b78b-404f9e5d45c3 nodeName:}" failed. No retries permitted until 2026-06-02 15:46:51.66970035 +0000 UTC m=+20.195608779 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32e4470f-53cd-4fda-b78b-404f9e5d45c3-original-pull-secret") pod "global-pull-secret-syncer-vh7pj" (UID: "32e4470f-53cd-4fda-b78b-404f9e5d45c3") : object "kube-system"/"original-pull-secret" not registered
Jun 02 15:46:48.059776 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:48.059682    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vh7pj"
Jun 02 15:46:48.059776 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:48.059733    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzt45"
Jun 02 15:46:48.060309 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:48.059819    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vh7pj" podUID="32e4470f-53cd-4fda-b78b-404f9e5d45c3"
Jun 02 15:46:48.060309 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:48.059933    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzt45" podUID="43370289-88af-4e83-b409-4e8a98e09857"
Jun 02 15:46:49.060136 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:49.060091    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6lvc8"
Jun 02 15:46:49.060681 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:49.060268    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6lvc8" podUID="eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca"
Jun 02 15:46:49.687708 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:49.687670    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca-metrics-certs\") pod \"network-metrics-daemon-6lvc8\" (UID: \"eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca\") " pod="openshift-multus/network-metrics-daemon-6lvc8"
Jun 02 15:46:49.687921 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:49.687845    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:49.687990 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:49.687930    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca-metrics-certs podName:eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca nodeName:}" failed. No retries permitted until 2026-06-02 15:47:05.687907498 +0000 UTC m=+34.213815922 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca-metrics-certs") pod "network-metrics-daemon-6lvc8" (UID: "eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:49.788918 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:49.788870    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d87j5\" (UniqueName: \"kubernetes.io/projected/43370289-88af-4e83-b409-4e8a98e09857-kube-api-access-d87j5\") pod \"network-check-target-zzt45\" (UID: \"43370289-88af-4e83-b409-4e8a98e09857\") " pod="openshift-network-diagnostics/network-check-target-zzt45"
Jun 02 15:46:49.789119 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:49.789061    2579 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 02 15:46:49.789119 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:49.789086    2579 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 02 15:46:49.789119 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:49.789099    2579 projected.go:194] Error preparing data for projected volume kube-api-access-d87j5 for pod openshift-network-diagnostics/network-check-target-zzt45: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:49.789271 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:49.789172    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43370289-88af-4e83-b409-4e8a98e09857-kube-api-access-d87j5 podName:43370289-88af-4e83-b409-4e8a98e09857 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:05.789152185 +0000 UTC m=+34.315060613 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-d87j5" (UniqueName: "kubernetes.io/projected/43370289-88af-4e83-b409-4e8a98e09857-kube-api-access-d87j5") pod "network-check-target-zzt45" (UID: "43370289-88af-4e83-b409-4e8a98e09857") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:50.059243 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:50.059156    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzt45"
Jun 02 15:46:50.059243 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:50.059209    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vh7pj"
Jun 02 15:46:50.059486 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:50.059292    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzt45" podUID="43370289-88af-4e83-b409-4e8a98e09857"
Jun 02 15:46:50.059486 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:50.059440    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vh7pj" podUID="32e4470f-53cd-4fda-b78b-404f9e5d45c3"
Jun 02 15:46:51.059489 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:51.059455    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6lvc8"
Jun 02 15:46:51.059957 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:51.059599    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6lvc8" podUID="eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca"
Jun 02 15:46:51.701608 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:51.701573    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32e4470f-53cd-4fda-b78b-404f9e5d45c3-original-pull-secret\") pod \"global-pull-secret-syncer-vh7pj\" (UID: \"32e4470f-53cd-4fda-b78b-404f9e5d45c3\") " pod="kube-system/global-pull-secret-syncer-vh7pj"
Jun 02 15:46:51.701806 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:51.701746    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 02 15:46:51.701850 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:51.701819    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32e4470f-53cd-4fda-b78b-404f9e5d45c3-original-pull-secret podName:32e4470f-53cd-4fda-b78b-404f9e5d45c3 nodeName:}" failed. No retries permitted until 2026-06-02 15:46:59.701804076 +0000 UTC m=+28.227712488 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32e4470f-53cd-4fda-b78b-404f9e5d45c3-original-pull-secret") pod "global-pull-secret-syncer-vh7pj" (UID: "32e4470f-53cd-4fda-b78b-404f9e5d45c3") : object "kube-system"/"original-pull-secret" not registered
Jun 02 15:46:52.060953 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:52.060628    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vh7pj"
Jun 02 15:46:52.060953 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:52.060757    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vh7pj" podUID="32e4470f-53cd-4fda-b78b-404f9e5d45c3"
Jun 02 15:46:52.060953 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:52.060796    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzt45"
Jun 02 15:46:52.060953 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:52.060870    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzt45" podUID="43370289-88af-4e83-b409-4e8a98e09857"
Jun 02 15:46:53.059498 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:53.059458    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6lvc8"
Jun 02 15:46:53.059688 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:53.059605    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6lvc8" podUID="eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca"
Jun 02 15:46:54.059645 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:54.059280    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vh7pj"
Jun 02 15:46:54.060222 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:54.059326    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzt45"
Jun 02 15:46:54.060222 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:54.059746    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vh7pj" podUID="32e4470f-53cd-4fda-b78b-404f9e5d45c3"
Jun 02 15:46:54.060222 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:54.059819    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzt45" podUID="43370289-88af-4e83-b409-4e8a98e09857"
Jun 02 15:46:54.229351 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:54.229321    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-th978" event={"ID":"4290cf74-a174-4c3f-88b4-66b05bbe4750","Type":"ContainerStarted","Data":"73ffac96ef70d74fae0bbde0aac64bf130a32f39192f849a462464cc39706e38"}
Jun 02 15:46:54.233525 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:54.232834    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zt7mm" event={"ID":"7b0e93e2-a6b5-4b5a-be51-e09f43184379","Type":"ContainerStarted","Data":"c83148de2978e0807cb2f05e5c332c123d2eece17c4efa1cb38926b339c5f365"}
Jun 02 15:46:54.235142 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:54.235108    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhfk7" event={"ID":"a4a82f10-0682-47f3-b029-9b45d89383d3","Type":"ContainerStarted","Data":"556727f6a203c2cd8cc56c747805639167836f8a7c1338d579138f5bc95872a3"}
Jun 02 15:46:54.238985 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:54.238689    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mllz2" event={"ID":"498f7354-865b-4ad1-9e9f-fabd57febd4a","Type":"ContainerStarted","Data":"9c3f72118dd0ad02da7831ea2a3f1147aa8c82a26e8fd7b4b33540feee616318"}
Jun 02 15:46:54.240355 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:54.240284    2579 generic.go:358] "Generic (PLEG): container finished" podID="810b8e3e-5836-4214-a141-c340d2220639" containerID="6b78609aff739452423ec389cfb64f6d86f787e3fac19aea9899aa4e85791020" exitCode=0
Jun 02 15:46:54.240499 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:54.240358    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dc6wj" event={"ID":"810b8e3e-5836-4214-a141-c340d2220639","Type":"ContainerDied","Data":"6b78609aff739452423ec389cfb64f6d86f787e3fac19aea9899aa4e85791020"}
Jun 02 15:46:54.243073 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:54.243054    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lqx9k_6127e33c-aa77-437e-b443-e956037afedb/ovn-acl-logging/0.log"
Jun 02 15:46:54.243459 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:54.243433    2579 generic.go:358] "Generic (PLEG): container finished" podID="6127e33c-aa77-437e-b443-e956037afedb" containerID="d11204c4c8fb42a26b5a6c7ab488e97d6a7050f2cf44f919646d46d187aee116" exitCode=1
Jun 02 15:46:54.243550 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:54.243496    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k" event={"ID":"6127e33c-aa77-437e-b443-e956037afedb","Type":"ContainerStarted","Data":"2fdc8567e6ffd01523786f0275ec701b4bf3f681bfb082350a326c64fd934982"}
Jun 02 15:46:54.243550 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:54.243527    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k" event={"ID":"6127e33c-aa77-437e-b443-e956037afedb","Type":"ContainerStarted","Data":"7a99bec2f0a81cb03aa3120e303e6cd9b8bce9b53ef2d77314ccbf330c7f66cb"}
Jun 02 15:46:54.243550 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:54.243541    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k" event={"ID":"6127e33c-aa77-437e-b443-e956037afedb","Type":"ContainerStarted","Data":"a5e7c05e234c50ea14fcaa11983225f615dae4d3c7a2cff0f49a5f34937331e1"}
Jun 02 15:46:54.243686 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:54.243553    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k" event={"ID":"6127e33c-aa77-437e-b443-e956037afedb","Type":"ContainerStarted","Data":"8b3badf30c707a0efdfefa0b88ea07196d14530dd2037f3eaed66c86adb0e9e2"}
Jun 02 15:46:54.243686 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:54.243566    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k" event={"ID":"6127e33c-aa77-437e-b443-e956037afedb","Type":"ContainerDied","Data":"d11204c4c8fb42a26b5a6c7ab488e97d6a7050f2cf44f919646d46d187aee116"}
Jun 02 15:46:54.243686 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:54.243584    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k" event={"ID":"6127e33c-aa77-437e-b443-e956037afedb","Type":"ContainerStarted","Data":"1a00428fb0d91052fbfd684fb63da38cb2f904cc274f4f760acf9a1a3e98ea02"}
Jun 02 15:46:54.245005 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:54.244985    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lg8nx" event={"ID":"73b94606-f892-4c47-9022-5d42f71bb77d","Type":"ContainerStarted","Data":"a97c8bfd55fe84a9f7289fd082917d135a19f9b3652c69e261342e463d42dded"}
Jun 02 15:46:54.246183 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:54.246158    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cwjmt" event={"ID":"3bee98d4-d583-4ecb-b07b-9f422a9810da","Type":"ContainerStarted","Data":"0997b55f557e222737d22edb692fc13709d136fca53f526288962f4825c655df"}
Jun 02 15:46:54.255030 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:54.254987    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-th978" podStartSLOduration=3.207587375 podStartE2EDuration="22.254965745s" podCreationTimestamp="2026-06-02 15:46:32 +0000 UTC" firstStartedPulling="2026-06-02 15:46:34.562523035 +0000 UTC m=+3.088431446" lastFinishedPulling="2026-06-02 15:46:53.609901405 +0000 UTC m=+22.135809816" observedRunningTime="2026-06-02 15:46:54.254777965 +0000 UTC m=+22.780686396" watchObservedRunningTime="2026-06-02 15:46:54.254965745 +0000 UTC m=+22.780874176"
Jun 02 15:46:54.280307 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:54.280142    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zt7mm" podStartSLOduration=3.2836131650000002 podStartE2EDuration="22.28012133s" podCreationTimestamp="2026-06-02 15:46:32 +0000 UTC" firstStartedPulling="2026-06-02 15:46:34.555855985 +0000 UTC m=+3.081764412" lastFinishedPulling="2026-06-02 15:46:53.552364169 +0000 UTC m=+22.078272577" observedRunningTime="2026-06-02 15:46:54.279692012 +0000 UTC m=+22.805600439" watchObservedRunningTime="2026-06-02 15:46:54.28012133 +0000 UTC m=+22.806029763"
Jun 02 15:46:54.299174 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:54.299132    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mllz2" podStartSLOduration=3.278674043 podStartE2EDuration="22.299112516s" podCreationTimestamp="2026-06-02 15:46:32 +0000 UTC" firstStartedPulling="2026-06-02 15:46:34.565100805 +0000 UTC m=+3.091009214" lastFinishedPulling="2026-06-02 15:46:53.585539278 +0000 UTC m=+22.111447687" observedRunningTime="2026-06-02 15:46:54.298263448 +0000 UTC m=+22.824171879" watchObservedRunningTime="2026-06-02 15:46:54.299112516 +0000 UTC m=+22.825020947"
Jun 02 15:46:54.344028 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:54.343961    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-lg8nx" podStartSLOduration=3.29623306 podStartE2EDuration="22.343939516s" podCreationTimestamp="2026-06-02 15:46:32 +0000 UTC" firstStartedPulling="2026-06-02 15:46:34.558595432 +0000 UTC m=+3.084503866" lastFinishedPulling="2026-06-02 15:46:53.60630191 +0000 UTC m=+22.132210322" observedRunningTime="2026-06-02 15:46:54.343741993 +0000 UTC m=+22.869650436" watchObservedRunningTime="2026-06-02 15:46:54.343939516 +0000 UTC m=+22.869847948"
Jun 02 15:46:54.364762 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:54.364715    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cwjmt" podStartSLOduration=3.364045402 podStartE2EDuration="22.364699687s" podCreationTimestamp="2026-06-02 15:46:32 +0000 UTC" firstStartedPulling="2026-06-02 15:46:34.551628106 +0000 UTC m=+3.077536515" lastFinishedPulling="2026-06-02 15:46:53.552282382 +0000 UTC m=+22.078190800" observedRunningTime="2026-06-02 15:46:54.364475069 +0000 UTC m=+22.890383501" watchObservedRunningTime="2026-06-02 15:46:54.364699687 +0000 UTC m=+22.890608117"
Jun 02 15:46:55.059645 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:55.059624    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6lvc8"
Jun 02 15:46:55.059806 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:55.059731    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6lvc8" podUID="eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca"
Jun 02 15:46:55.092324 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:55.092296    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jun 02 15:46:55.249906 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:55.249809    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zqsll" event={"ID":"c0c551c2-7ea0-4de3-b24a-735cdb0c536d","Type":"ContainerStarted","Data":"315de66e5b52c5f88527048299688938ddf841f090f95b0c2fcce70b022ca80d"}
Jun 02 15:46:55.251997 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:55.251964    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhfk7" event={"ID":"a4a82f10-0682-47f3-b029-9b45d89383d3","Type":"ContainerStarted","Data":"e6f2faa6c68e59a378276509937ed85df7a7b8922f0ff3762a0dc593d633cc95"}
Jun 02 15:46:55.266053 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:55.266000    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zqsll" podStartSLOduration=4.226446275 podStartE2EDuration="23.265984631s" podCreationTimestamp="2026-06-02 15:46:32 +0000 UTC" firstStartedPulling="2026-06-02 15:46:34.564920978 +0000 UTC m=+3.090829402" lastFinishedPulling="2026-06-02 15:46:53.604459329 +0000 UTC m=+22.130367758" observedRunningTime="2026-06-02 15:46:55.265412797 +0000 UTC m=+23.791321221" watchObservedRunningTime="2026-06-02 15:46:55.265984631 +0000 UTC m=+23.791893061"
Jun 02 15:46:55.996595 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:55.996481    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-06-02T15:46:55.092313627Z","UUID":"ff32173e-d88d-43ff-accc-7ebd54832f11","Handler":null,"Name":"","Endpoint":""}
Jun 02 15:46:55.999843 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:55.999815    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jun 02 15:46:55.999977 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:55.999852    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jun 02 15:46:56.059508 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:56.059474    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vh7pj"
Jun 02 15:46:56.059508 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:56.059510    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzt45"
Jun 02 15:46:56.059743 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:56.059598    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vh7pj" podUID="32e4470f-53cd-4fda-b78b-404f9e5d45c3"
Jun 02 15:46:56.059743 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:56.059687    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzt45" podUID="43370289-88af-4e83-b409-4e8a98e09857"
Jun 02 15:46:57.059683 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:57.059467    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6lvc8"
Jun 02 15:46:57.060132 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:57.059785    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6lvc8" podUID="eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca"
Jun 02 15:46:57.259747 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:57.259717    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lqx9k_6127e33c-aa77-437e-b443-e956037afedb/ovn-acl-logging/0.log"
Jun 02 15:46:57.260153 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:57.260123    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k" event={"ID":"6127e33c-aa77-437e-b443-e956037afedb","Type":"ContainerStarted","Data":"9dc8fdca6c609e0bbef25ef955974d977d55c5d7beeae6df84fc12aac0060cb4"}
Jun 02 15:46:57.261996 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:57.261975    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhfk7" event={"ID":"a4a82f10-0682-47f3-b029-9b45d89383d3","Type":"ContainerStarted","Data":"f11baf7ac5c38e4f4e0dc5d13fbabbad15edd38a1ace8e8bf8a9b6edc5ffde08"}
Jun 02 15:46:57.316224 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:57.316123    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhfk7" podStartSLOduration=3.534201154 podStartE2EDuration="25.316108632s" podCreationTimestamp="2026-06-02 15:46:32 +0000 UTC" firstStartedPulling="2026-06-02 15:46:34.550429699 +0000 UTC m=+3.076338121" lastFinishedPulling="2026-06-02 15:46:56.332337172 +0000 UTC m=+24.858245599" observedRunningTime="2026-06-02 15:46:57.315928447 +0000 UTC m=+25.841836878" watchObservedRunningTime="2026-06-02 15:46:57.316108632 +0000 UTC m=+25.842017063"
Jun 02 15:46:58.062866 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:58.062781    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vh7pj"
Jun 02 15:46:58.062866 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:58.062801    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzt45"
Jun 02 15:46:58.063472 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:58.062909    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vh7pj" podUID="32e4470f-53cd-4fda-b78b-404f9e5d45c3"
Jun 02 15:46:58.063472 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:58.063031    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzt45" podUID="43370289-88af-4e83-b409-4e8a98e09857"
Jun 02 15:46:58.651313 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:58.651275    2579 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zt7mm"
Jun 02 15:46:58.652067 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:58.652035    2579 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zt7mm"
Jun 02 15:46:59.059217 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:59.059174    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6lvc8"
Jun 02 15:46:59.059447 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:59.059334    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6lvc8" podUID="eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca"
Jun 02 15:46:59.268760 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:59.268485    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zt7mm"
Jun 02 15:46:59.269336 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:59.268968    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zt7mm"
Jun 02 15:46:59.775085 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:46:59.775037    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32e4470f-53cd-4fda-b78b-404f9e5d45c3-original-pull-secret\") pod \"global-pull-secret-syncer-vh7pj\" (UID: \"32e4470f-53cd-4fda-b78b-404f9e5d45c3\") " pod="kube-system/global-pull-secret-syncer-vh7pj"
Jun 02 15:46:59.775254 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:59.775159    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 02 15:46:59.775254 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:46:59.775237    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32e4470f-53cd-4fda-b78b-404f9e5d45c3-original-pull-secret podName:32e4470f-53cd-4fda-b78b-404f9e5d45c3 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:15.775222494 +0000 UTC m=+44.301130916 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32e4470f-53cd-4fda-b78b-404f9e5d45c3-original-pull-secret") pod "global-pull-secret-syncer-vh7pj" (UID: "32e4470f-53cd-4fda-b78b-404f9e5d45c3") : object "kube-system"/"original-pull-secret" not registered
Jun 02 15:47:00.059674 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:00.059645    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzt45"
Jun 02 15:47:00.059816 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:00.059794    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzt45" podUID="43370289-88af-4e83-b409-4e8a98e09857"
Jun 02 15:47:00.059873 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:00.059841    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vh7pj"
Jun 02 15:47:00.059972 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:00.059951    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vh7pj" podUID="32e4470f-53cd-4fda-b78b-404f9e5d45c3"
Jun 02 15:47:00.273175 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:00.272990    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lqx9k_6127e33c-aa77-437e-b443-e956037afedb/ovn-acl-logging/0.log"
Jun 02 15:47:00.273683 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:00.273478    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k" event={"ID":"6127e33c-aa77-437e-b443-e956037afedb","Type":"ContainerStarted","Data":"675ba11f061be3d72033894c8bb7178d020162121b4c32ec02d92631e27b9dff"}
Jun 02 15:47:00.273973 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:00.273948    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:47:00.274063 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:00.273986    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:47:00.274110 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:00.274094    2579 scope.go:117] "RemoveContainer" containerID="d11204c4c8fb42a26b5a6c7ab488e97d6a7050f2cf44f919646d46d187aee116"
Jun 02 15:47:00.290421 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:00.290380    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:47:01.059983 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:01.059947    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6lvc8"
Jun 02 15:47:01.060190 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:01.060092    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6lvc8" podUID="eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca"
Jun 02 15:47:01.278152 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:01.278115    2579 generic.go:358] "Generic (PLEG): container finished" podID="810b8e3e-5836-4214-a141-c340d2220639" containerID="a55f2a07e5cd9aed1917ef74902648161edfb401d4a6877c129e67da40d6c68b" exitCode=0
Jun 02 15:47:01.278611 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:01.278203    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dc6wj" event={"ID":"810b8e3e-5836-4214-a141-c340d2220639","Type":"ContainerDied","Data":"a55f2a07e5cd9aed1917ef74902648161edfb401d4a6877c129e67da40d6c68b"}
Jun 02 15:47:01.282247 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:01.282229    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lqx9k_6127e33c-aa77-437e-b443-e956037afedb/ovn-acl-logging/0.log"
Jun 02 15:47:01.282747 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:01.282716    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k" event={"ID":"6127e33c-aa77-437e-b443-e956037afedb","Type":"ContainerStarted","Data":"905314822eb84bc75e214f6d61645d7232edc485dd17c1962380d9b86bbf88a5"}
Jun 02 15:47:01.283002 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:01.282982    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:47:01.303054 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:01.303020    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:47:01.341993 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:01.341927    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k" podStartSLOduration=10.224811803 podStartE2EDuration="29.341907578s" podCreationTimestamp="2026-06-02 15:46:32 +0000 UTC" firstStartedPulling="2026-06-02 15:46:34.559824764 +0000 UTC m=+3.085733193" lastFinishedPulling="2026-06-02 15:46:53.676920545 +0000 UTC m=+22.202828968" observedRunningTime="2026-06-02 15:47:01.340842363 +0000 UTC m=+29.866750794" watchObservedRunningTime="2026-06-02 15:47:01.341907578 +0000 UTC m=+29.867816009"
Jun 02 15:47:02.062419 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:02.062379    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzt45"
Jun 02 15:47:02.062558 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:02.062379    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vh7pj"
Jun 02 15:47:02.062558 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:02.062495    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzt45" podUID="43370289-88af-4e83-b409-4e8a98e09857"
Jun 02 15:47:02.062647 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:02.062554    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vh7pj" podUID="32e4470f-53cd-4fda-b78b-404f9e5d45c3"
Jun 02 15:47:02.263955 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:02.263781    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6lvc8"]
Jun 02 15:47:02.264069 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:02.264058    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6lvc8"
Jun 02 15:47:02.264169 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:02.264153    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6lvc8" podUID="eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca"
Jun 02 15:47:02.265861 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:02.265840    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vh7pj"]
Jun 02 15:47:02.268994 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:02.268973    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zzt45"]
Jun 02 15:47:02.287663 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:02.287568    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dc6wj" event={"ID":"810b8e3e-5836-4214-a141-c340d2220639","Type":"ContainerStarted","Data":"eed69f178f212f220135a3cf660edd7900e2e9bbab1054be660cf3e253861758"}
Jun 02 15:47:02.287663 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:02.287618    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vh7pj"
Jun 02 15:47:02.287663 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:02.287619    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzt45"
Jun 02 15:47:02.288386 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:02.288362    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vh7pj" podUID="32e4470f-53cd-4fda-b78b-404f9e5d45c3"
Jun 02 15:47:02.288620 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:02.288597    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzt45" podUID="43370289-88af-4e83-b409-4e8a98e09857"
Jun 02 15:47:03.291508 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:03.291475    2579 generic.go:358] "Generic (PLEG): container finished" podID="810b8e3e-5836-4214-a141-c340d2220639" containerID="eed69f178f212f220135a3cf660edd7900e2e9bbab1054be660cf3e253861758" exitCode=0
Jun 02 15:47:03.292071 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:03.291597    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dc6wj" event={"ID":"810b8e3e-5836-4214-a141-c340d2220639","Type":"ContainerDied","Data":"eed69f178f212f220135a3cf660edd7900e2e9bbab1054be660cf3e253861758"}
Jun 02 15:47:04.059474 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:04.059438    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vh7pj"
Jun 02 15:47:04.059474 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:04.059471    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6lvc8"
Jun 02 15:47:04.059683 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:04.059561    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vh7pj" podUID="32e4470f-53cd-4fda-b78b-404f9e5d45c3"
Jun 02 15:47:04.059683 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:04.059605    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzt45"
Jun 02 15:47:04.059683 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:04.059604    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6lvc8" podUID="eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca"
Jun 02 15:47:04.059683 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:04.059671    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzt45" podUID="43370289-88af-4e83-b409-4e8a98e09857"
Jun 02 15:47:05.298225 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:05.298188    2579 generic.go:358] "Generic (PLEG): container finished" podID="810b8e3e-5836-4214-a141-c340d2220639" containerID="7454e20e169ac3f39329cee4cdc085b4e645c7f2b8c353a1481c5ebac1c8dd4d" exitCode=0
Jun 02 15:47:05.298755 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:05.298246    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dc6wj" event={"ID":"810b8e3e-5836-4214-a141-c340d2220639","Type":"ContainerDied","Data":"7454e20e169ac3f39329cee4cdc085b4e645c7f2b8c353a1481c5ebac1c8dd4d"}
Jun 02 15:47:05.722353 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:05.722262    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca-metrics-certs\") pod \"network-metrics-daemon-6lvc8\" (UID: \"eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca\") " pod="openshift-multus/network-metrics-daemon-6lvc8"
Jun 02 15:47:05.722508 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:05.722381    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:47:05.722508 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:05.722464    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca-metrics-certs podName:eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca nodeName:}" failed. No retries permitted until 2026-06-02 15:47:37.7224447 +0000 UTC m=+66.248353131 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca-metrics-certs") pod "network-metrics-daemon-6lvc8" (UID: "eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:47:05.823004 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:05.822972    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d87j5\" (UniqueName: \"kubernetes.io/projected/43370289-88af-4e83-b409-4e8a98e09857-kube-api-access-d87j5\") pod \"network-check-target-zzt45\" (UID: \"43370289-88af-4e83-b409-4e8a98e09857\") " pod="openshift-network-diagnostics/network-check-target-zzt45"
Jun 02 15:47:05.823147 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:05.823127    2579 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 02 15:47:05.823147 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:05.823146    2579 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 02 15:47:05.823227 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:05.823156    2579 projected.go:194] Error preparing data for projected volume kube-api-access-d87j5 for pod openshift-network-diagnostics/network-check-target-zzt45: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:47:05.823227 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:05.823215    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43370289-88af-4e83-b409-4e8a98e09857-kube-api-access-d87j5 podName:43370289-88af-4e83-b409-4e8a98e09857 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:37.823200252 +0000 UTC m=+66.349108662 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-d87j5" (UniqueName: "kubernetes.io/projected/43370289-88af-4e83-b409-4e8a98e09857-kube-api-access-d87j5") pod "network-check-target-zzt45" (UID: "43370289-88af-4e83-b409-4e8a98e09857") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:47:06.063520 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:06.063491    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vh7pj"
Jun 02 15:47:06.063705 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:06.063532    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzt45"
Jun 02 15:47:06.063705 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:06.063557    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6lvc8"
Jun 02 15:47:06.063705 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:06.063640    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6lvc8" podUID="eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca"
Jun 02 15:47:06.063965 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:06.063934    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vh7pj" podUID="32e4470f-53cd-4fda-b78b-404f9e5d45c3"
Jun 02 15:47:06.064057 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:06.064036    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzt45" podUID="43370289-88af-4e83-b409-4e8a98e09857"
Jun 02 15:47:06.323188 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:06.323118    2579 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-156.ec2.internal" event="NodeReady"
Jun 02 15:47:06.323671 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:06.323272    2579 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
Jun 02 15:47:06.385284 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:06.384657    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-cqh44"]
Jun 02 15:47:06.406390 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:06.406360    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bgsrq"]
Jun 02 15:47:06.406614 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:06.406543    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cqh44"
Jun 02 15:47:06.409286 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:06.408841    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jun 02 15:47:06.409286 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:06.408903    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-cw7rp\""
Jun 02 15:47:06.409286 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:06.408994    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jun 02 15:47:06.409286 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:06.409103    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jun 02 15:47:06.427159 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:06.427121    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9w4c\" (UniqueName: \"kubernetes.io/projected/5a9d0209-8692-4052-bb84-c3b8bb9d383e-kube-api-access-m9w4c\") pod \"ingress-canary-cqh44\" (UID: \"5a9d0209-8692-4052-bb84-c3b8bb9d383e\") " pod="openshift-ingress-canary/ingress-canary-cqh44"
Jun 02 15:47:06.427293 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:06.427222    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a9d0209-8692-4052-bb84-c3b8bb9d383e-cert\") pod \"ingress-canary-cqh44\" (UID: \"5a9d0209-8692-4052-bb84-c3b8bb9d383e\") " pod="openshift-ingress-canary/ingress-canary-cqh44"
Jun 02 15:47:06.427487 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:06.427467    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cqh44"]
Jun 02 15:47:06.427538 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:06.427498    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bgsrq"]
Jun 02 15:47:06.427680 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:06.427661    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bgsrq"
Jun 02 15:47:06.430165 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:06.430007    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jun 02 15:47:06.430165 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:06.430037    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jun 02 15:47:06.430323 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:06.430272    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ht87l\""
Jun 02 15:47:06.528142 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:06.528105    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxbs7\" (UniqueName: \"kubernetes.io/projected/666f2157-8d74-421e-af17-78b99e918795-kube-api-access-mxbs7\") pod \"dns-default-bgsrq\" (UID: \"666f2157-8d74-421e-af17-78b99e918795\") " pod="openshift-dns/dns-default-bgsrq"
Jun 02 15:47:06.528303 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:06.528257    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/666f2157-8d74-421e-af17-78b99e918795-config-volume\") pod \"dns-default-bgsrq\" (UID: \"666f2157-8d74-421e-af17-78b99e918795\") " pod="openshift-dns/dns-default-bgsrq"
Jun 02 15:47:06.528415 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:06.528386    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-m9w4c\" (UniqueName: \"kubernetes.io/projected/5a9d0209-8692-4052-bb84-c3b8bb9d383e-kube-api-access-m9w4c\") pod \"ingress-canary-cqh44\" (UID: \"5a9d0209-8692-4052-bb84-c3b8bb9d383e\") " pod="openshift-ingress-canary/ingress-canary-cqh44"
Jun 02 15:47:06.528547 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:06.528477    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a9d0209-8692-4052-bb84-c3b8bb9d383e-cert\") pod \"ingress-canary-cqh44\" (UID: \"5a9d0209-8692-4052-bb84-c3b8bb9d383e\") " pod="openshift-ingress-canary/ingress-canary-cqh44"
Jun 02 15:47:06.528547 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:06.528505    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/666f2157-8d74-421e-af17-78b99e918795-metrics-tls\") pod \"dns-default-bgsrq\" (UID: \"666f2157-8d74-421e-af17-78b99e918795\") " pod="openshift-dns/dns-default-bgsrq"
Jun 02 15:47:06.528547 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:06.528534    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/666f2157-8d74-421e-af17-78b99e918795-tmp-dir\") pod \"dns-default-bgsrq\" (UID: \"666f2157-8d74-421e-af17-78b99e918795\") " pod="openshift-dns/dns-default-bgsrq"
Jun 02 15:47:06.528684 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:06.528573    2579 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:47:06.528684 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:06.528644    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a9d0209-8692-4052-bb84-c3b8bb9d383e-cert podName:5a9d0209-8692-4052-bb84-c3b8bb9d383e nodeName:}" failed. No retries permitted until 2026-06-02 15:47:07.028624445 +0000 UTC m=+35.554532871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5a9d0209-8692-4052-bb84-c3b8bb9d383e-cert") pod "ingress-canary-cqh44" (UID: "5a9d0209-8692-4052-bb84-c3b8bb9d383e") : secret "canary-serving-cert" not found
Jun 02 15:47:06.542983 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:06.542952    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9w4c\" (UniqueName: \"kubernetes.io/projected/5a9d0209-8692-4052-bb84-c3b8bb9d383e-kube-api-access-m9w4c\") pod \"ingress-canary-cqh44\" (UID: \"5a9d0209-8692-4052-bb84-c3b8bb9d383e\") " pod="openshift-ingress-canary/ingress-canary-cqh44"
Jun 02 15:47:06.628942 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:06.628855    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/666f2157-8d74-421e-af17-78b99e918795-config-volume\") pod \"dns-default-bgsrq\" (UID: \"666f2157-8d74-421e-af17-78b99e918795\") " pod="openshift-dns/dns-default-bgsrq"
Jun 02 15:47:06.629117 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:06.628956    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/666f2157-8d74-421e-af17-78b99e918795-metrics-tls\") pod \"dns-default-bgsrq\" (UID: \"666f2157-8d74-421e-af17-78b99e918795\") " pod="openshift-dns/dns-default-bgsrq"
Jun 02 15:47:06.629117 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:06.628998    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/666f2157-8d74-421e-af17-78b99e918795-tmp-dir\") pod \"dns-default-bgsrq\" (UID: \"666f2157-8d74-421e-af17-78b99e918795\") " pod="openshift-dns/dns-default-bgsrq"
Jun 02 15:47:06.629117 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:06.629032    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mxbs7\" (UniqueName: \"kubernetes.io/projected/666f2157-8d74-421e-af17-78b99e918795-kube-api-access-mxbs7\") pod \"dns-default-bgsrq\" (UID: \"666f2157-8d74-421e-af17-78b99e918795\") " pod="openshift-dns/dns-default-bgsrq"
Jun 02 15:47:06.629284 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:06.629123    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:47:06.629284 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:06.629186    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/666f2157-8d74-421e-af17-78b99e918795-metrics-tls podName:666f2157-8d74-421e-af17-78b99e918795 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:07.129168477 +0000 UTC m=+35.655076900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/666f2157-8d74-421e-af17-78b99e918795-metrics-tls") pod "dns-default-bgsrq" (UID: "666f2157-8d74-421e-af17-78b99e918795") : secret "dns-default-metrics-tls" not found
Jun 02 15:47:06.629452 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:06.629390    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/666f2157-8d74-421e-af17-78b99e918795-tmp-dir\") pod \"dns-default-bgsrq\" (UID: \"666f2157-8d74-421e-af17-78b99e918795\") " pod="openshift-dns/dns-default-bgsrq"
Jun 02 15:47:06.629571 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:06.629550    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/666f2157-8d74-421e-af17-78b99e918795-config-volume\") pod \"dns-default-bgsrq\" (UID: \"666f2157-8d74-421e-af17-78b99e918795\") " pod="openshift-dns/dns-default-bgsrq"
Jun 02 15:47:06.643840 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:06.643808    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxbs7\" (UniqueName: \"kubernetes.io/projected/666f2157-8d74-421e-af17-78b99e918795-kube-api-access-mxbs7\") pod \"dns-default-bgsrq\" (UID: \"666f2157-8d74-421e-af17-78b99e918795\") " pod="openshift-dns/dns-default-bgsrq"
Jun 02 15:47:07.031577 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:07.031539    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a9d0209-8692-4052-bb84-c3b8bb9d383e-cert\") pod \"ingress-canary-cqh44\" (UID: \"5a9d0209-8692-4052-bb84-c3b8bb9d383e\") " pod="openshift-ingress-canary/ingress-canary-cqh44"
Jun 02 15:47:07.031832 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:07.031682    2579 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:47:07.031832 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:07.031743    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a9d0209-8692-4052-bb84-c3b8bb9d383e-cert podName:5a9d0209-8692-4052-bb84-c3b8bb9d383e nodeName:}" failed. No retries permitted until 2026-06-02 15:47:08.031724601 +0000 UTC m=+36.557633027 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5a9d0209-8692-4052-bb84-c3b8bb9d383e-cert") pod "ingress-canary-cqh44" (UID: "5a9d0209-8692-4052-bb84-c3b8bb9d383e") : secret "canary-serving-cert" not found
Jun 02 15:47:07.132730 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:07.132690    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/666f2157-8d74-421e-af17-78b99e918795-metrics-tls\") pod \"dns-default-bgsrq\" (UID: \"666f2157-8d74-421e-af17-78b99e918795\") " pod="openshift-dns/dns-default-bgsrq"
Jun 02 15:47:07.132960 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:07.132841    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:47:07.132960 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:07.132908    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/666f2157-8d74-421e-af17-78b99e918795-metrics-tls podName:666f2157-8d74-421e-af17-78b99e918795 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:08.132893193 +0000 UTC m=+36.658801606 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/666f2157-8d74-421e-af17-78b99e918795-metrics-tls") pod "dns-default-bgsrq" (UID: "666f2157-8d74-421e-af17-78b99e918795") : secret "dns-default-metrics-tls" not found
Jun 02 15:47:08.037944 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:08.037897    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a9d0209-8692-4052-bb84-c3b8bb9d383e-cert\") pod \"ingress-canary-cqh44\" (UID: \"5a9d0209-8692-4052-bb84-c3b8bb9d383e\") " pod="openshift-ingress-canary/ingress-canary-cqh44"
Jun 02 15:47:08.038375 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:08.038085    2579 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:47:08.038375 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:08.038180    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a9d0209-8692-4052-bb84-c3b8bb9d383e-cert podName:5a9d0209-8692-4052-bb84-c3b8bb9d383e nodeName:}" failed. No retries permitted until 2026-06-02 15:47:10.038149617 +0000 UTC m=+38.564058040 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5a9d0209-8692-4052-bb84-c3b8bb9d383e-cert") pod "ingress-canary-cqh44" (UID: "5a9d0209-8692-4052-bb84-c3b8bb9d383e") : secret "canary-serving-cert" not found
Jun 02 15:47:08.059643 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:08.059602    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vh7pj"
Jun 02 15:47:08.059822 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:08.059771    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6lvc8"
Jun 02 15:47:08.059900 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:08.059879    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzt45"
Jun 02 15:47:08.065091 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:08.063554    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jun 02 15:47:08.065091 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:08.063603    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mqvtg\""
Jun 02 15:47:08.065091 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:08.063612    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jun 02 15:47:08.065091 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:08.063554    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gq5xh\""
Jun 02 15:47:08.065091 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:08.063564    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jun 02 15:47:08.065091 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:08.063610    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jun 02 15:47:08.139061 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:08.139029    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/666f2157-8d74-421e-af17-78b99e918795-metrics-tls\") pod \"dns-default-bgsrq\" (UID: \"666f2157-8d74-421e-af17-78b99e918795\") " pod="openshift-dns/dns-default-bgsrq"
Jun 02 15:47:08.139258 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:08.139218    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:47:08.139305 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:08.139289    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/666f2157-8d74-421e-af17-78b99e918795-metrics-tls podName:666f2157-8d74-421e-af17-78b99e918795 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:10.139274197 +0000 UTC m=+38.665182610 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/666f2157-8d74-421e-af17-78b99e918795-metrics-tls") pod "dns-default-bgsrq" (UID: "666f2157-8d74-421e-af17-78b99e918795") : secret "dns-default-metrics-tls" not found
Jun 02 15:47:10.051968 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:10.051678    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a9d0209-8692-4052-bb84-c3b8bb9d383e-cert\") pod \"ingress-canary-cqh44\" (UID: \"5a9d0209-8692-4052-bb84-c3b8bb9d383e\") " pod="openshift-ingress-canary/ingress-canary-cqh44"
Jun 02 15:47:10.052442 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:10.051846    2579 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:47:10.052442 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:10.052129    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a9d0209-8692-4052-bb84-c3b8bb9d383e-cert podName:5a9d0209-8692-4052-bb84-c3b8bb9d383e nodeName:}" failed. No retries permitted until 2026-06-02 15:47:14.052104823 +0000 UTC m=+42.578013256 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5a9d0209-8692-4052-bb84-c3b8bb9d383e-cert") pod "ingress-canary-cqh44" (UID: "5a9d0209-8692-4052-bb84-c3b8bb9d383e") : secret "canary-serving-cert" not found
Jun 02 15:47:10.152705 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:10.152653    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/666f2157-8d74-421e-af17-78b99e918795-metrics-tls\") pod \"dns-default-bgsrq\" (UID: \"666f2157-8d74-421e-af17-78b99e918795\") " pod="openshift-dns/dns-default-bgsrq"
Jun 02 15:47:10.152889 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:10.152824    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:47:10.152937 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:10.152894    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/666f2157-8d74-421e-af17-78b99e918795-metrics-tls podName:666f2157-8d74-421e-af17-78b99e918795 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:14.152877842 +0000 UTC m=+42.678786256 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/666f2157-8d74-421e-af17-78b99e918795-metrics-tls") pod "dns-default-bgsrq" (UID: "666f2157-8d74-421e-af17-78b99e918795") : secret "dns-default-metrics-tls" not found
Jun 02 15:47:12.314368 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:12.314331    2579 generic.go:358] "Generic (PLEG): container finished" podID="810b8e3e-5836-4214-a141-c340d2220639" containerID="7e90b2c430720b4d2fdc36c5976a828d467ad5c615536d2d32eb85060836fac7" exitCode=0
Jun 02 15:47:12.314767 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:12.314395    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dc6wj" event={"ID":"810b8e3e-5836-4214-a141-c340d2220639","Type":"ContainerDied","Data":"7e90b2c430720b4d2fdc36c5976a828d467ad5c615536d2d32eb85060836fac7"}
Jun 02 15:47:13.319360 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:13.319324    2579 generic.go:358] "Generic (PLEG): container finished" podID="810b8e3e-5836-4214-a141-c340d2220639" containerID="6410ccc0aaadb24329978139409269f8ef8cea60ad90fa95c682d213bb2ad515" exitCode=0
Jun 02 15:47:13.319762 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:13.319390    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dc6wj" event={"ID":"810b8e3e-5836-4214-a141-c340d2220639","Type":"ContainerDied","Data":"6410ccc0aaadb24329978139409269f8ef8cea60ad90fa95c682d213bb2ad515"}
Jun 02 15:47:14.083002 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:14.082968    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a9d0209-8692-4052-bb84-c3b8bb9d383e-cert\") pod \"ingress-canary-cqh44\" (UID: \"5a9d0209-8692-4052-bb84-c3b8bb9d383e\") " pod="openshift-ingress-canary/ingress-canary-cqh44"
Jun 02 15:47:14.083153 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:14.083115    2579 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:47:14.083207 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:14.083180    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a9d0209-8692-4052-bb84-c3b8bb9d383e-cert podName:5a9d0209-8692-4052-bb84-c3b8bb9d383e nodeName:}" failed. No retries permitted until 2026-06-02 15:47:22.083164404 +0000 UTC m=+50.609072852 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5a9d0209-8692-4052-bb84-c3b8bb9d383e-cert") pod "ingress-canary-cqh44" (UID: "5a9d0209-8692-4052-bb84-c3b8bb9d383e") : secret "canary-serving-cert" not found
Jun 02 15:47:14.183594 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:14.183551    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/666f2157-8d74-421e-af17-78b99e918795-metrics-tls\") pod \"dns-default-bgsrq\" (UID: \"666f2157-8d74-421e-af17-78b99e918795\") " pod="openshift-dns/dns-default-bgsrq"
Jun 02 15:47:14.183738 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:14.183677    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:47:14.183738 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:14.183735    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/666f2157-8d74-421e-af17-78b99e918795-metrics-tls podName:666f2157-8d74-421e-af17-78b99e918795 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:22.183718299 +0000 UTC m=+50.709626725 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/666f2157-8d74-421e-af17-78b99e918795-metrics-tls") pod "dns-default-bgsrq" (UID: "666f2157-8d74-421e-af17-78b99e918795") : secret "dns-default-metrics-tls" not found
Jun 02 15:47:14.324025 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:14.323984    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dc6wj" event={"ID":"810b8e3e-5836-4214-a141-c340d2220639","Type":"ContainerStarted","Data":"edc44ebeca6707fc32d4b0237af331ce4e1ee69f40cbc12d09119d39f112ec04"}
Jun 02 15:47:14.362719 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:14.362620    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dc6wj" podStartSLOduration=5.489646702 podStartE2EDuration="42.362605452s" podCreationTimestamp="2026-06-02 15:46:32 +0000 UTC" firstStartedPulling="2026-06-02 15:46:34.562881607 +0000 UTC m=+3.088790020" lastFinishedPulling="2026-06-02 15:47:11.435840358 +0000 UTC m=+39.961748770" observedRunningTime="2026-06-02 15:47:14.362192079 +0000 UTC m=+42.888100510" watchObservedRunningTime="2026-06-02 15:47:14.362605452 +0000 UTC m=+42.888513928"
Jun 02 15:47:15.796880 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:15.796841    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32e4470f-53cd-4fda-b78b-404f9e5d45c3-original-pull-secret\") pod \"global-pull-secret-syncer-vh7pj\" (UID: \"32e4470f-53cd-4fda-b78b-404f9e5d45c3\") " pod="kube-system/global-pull-secret-syncer-vh7pj"
Jun 02 15:47:15.800230 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:15.800206    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32e4470f-53cd-4fda-b78b-404f9e5d45c3-original-pull-secret\") pod \"global-pull-secret-syncer-vh7pj\" (UID: \"32e4470f-53cd-4fda-b78b-404f9e5d45c3\") " pod="kube-system/global-pull-secret-syncer-vh7pj"
Jun 02 15:47:15.803299 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:15.803272    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-85479cc8c6-79tdg"]
Jun 02 15:47:15.835052 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:15.835018    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-85479cc8c6-79tdg"]
Jun 02 15:47:15.835052 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:15.835052    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b5b4bc578-7t2hk"]
Jun 02 15:47:15.835236 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:15.835176    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85479cc8c6-79tdg"
Jun 02 15:47:15.837863 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:15.837841    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jun 02 15:47:15.838689 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:15.838673    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jun 02 15:47:15.838778 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:15.838675    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jun 02 15:47:15.838954 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:15.838938    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jun 02 15:47:15.858971 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:15.858946    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b5b4bc578-7t2hk"]
Jun 02 15:47:15.859110 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:15.859060    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b5b4bc578-7t2hk"
Jun 02 15:47:15.862155 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:15.862130    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jun 02 15:47:15.862665 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:15.862645    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jun 02 15:47:15.862785 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:15.862652    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jun 02 15:47:15.862785 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:15.862749    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jun 02 15:47:15.873053 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:15.873034    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vh7pj"
Jun 02 15:47:15.897592 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:15.897562    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmp94\" (UniqueName: \"kubernetes.io/projected/6506fad5-cbfe-4561-8260-4f63526bbc42-kube-api-access-vmp94\") pod \"cluster-proxy-proxy-agent-7b5b4bc578-7t2hk\" (UID: \"6506fad5-cbfe-4561-8260-4f63526bbc42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b5b4bc578-7t2hk"
Jun 02 15:47:15.897753 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:15.897601    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6fdc7974-b9aa-4b56-a19a-1741e5bcc5c3-tmp\") pod \"klusterlet-addon-workmgr-85479cc8c6-79tdg\" (UID: \"6fdc7974-b9aa-4b56-a19a-1741e5bcc5c3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85479cc8c6-79tdg"
Jun 02 15:47:15.897753 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:15.897621    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6fdc7974-b9aa-4b56-a19a-1741e5bcc5c3-klusterlet-config\") pod \"klusterlet-addon-workmgr-85479cc8c6-79tdg\" (UID: \"6fdc7974-b9aa-4b56-a19a-1741e5bcc5c3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85479cc8c6-79tdg"
Jun 02 15:47:15.897753 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:15.897657    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6506fad5-cbfe-4561-8260-4f63526bbc42-ca\") pod \"cluster-proxy-proxy-agent-7b5b4bc578-7t2hk\" (UID: \"6506fad5-cbfe-4561-8260-4f63526bbc42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b5b4bc578-7t2hk"
Jun 02 15:47:15.897753 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:15.897683    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6506fad5-cbfe-4561-8260-4f63526bbc42-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7b5b4bc578-7t2hk\" (UID: \"6506fad5-cbfe-4561-8260-4f63526bbc42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b5b4bc578-7t2hk"
Jun 02 15:47:15.897753 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:15.897704    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtn2j\" (UniqueName: \"kubernetes.io/projected/6fdc7974-b9aa-4b56-a19a-1741e5bcc5c3-kube-api-access-mtn2j\") pod \"klusterlet-addon-workmgr-85479cc8c6-79tdg\" (UID: \"6fdc7974-b9aa-4b56-a19a-1741e5bcc5c3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85479cc8c6-79tdg"
Jun 02 15:47:15.898006 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:15.897801    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6506fad5-cbfe-4561-8260-4f63526bbc42-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7b5b4bc578-7t2hk\" (UID: \"6506fad5-cbfe-4561-8260-4f63526bbc42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b5b4bc578-7t2hk"
Jun 02 15:47:15.898006 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:15.897867    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6506fad5-cbfe-4561-8260-4f63526bbc42-hub\") pod \"cluster-proxy-proxy-agent-7b5b4bc578-7t2hk\" (UID: \"6506fad5-cbfe-4561-8260-4f63526bbc42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b5b4bc578-7t2hk"
Jun 02 15:47:15.898006 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:15.897957    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/6506fad5-cbfe-4561-8260-4f63526bbc42-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-7b5b4bc578-7t2hk\" (UID: \"6506fad5-cbfe-4561-8260-4f63526bbc42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b5b4bc578-7t2hk"
Jun 02 15:47:15.999074 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:15.999044    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6506fad5-cbfe-4561-8260-4f63526bbc42-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7b5b4bc578-7t2hk\" (UID: \"6506fad5-cbfe-4561-8260-4f63526bbc42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b5b4bc578-7t2hk"
Jun 02 15:47:15.999238 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:15.999087    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6506fad5-cbfe-4561-8260-4f63526bbc42-hub\") pod \"cluster-proxy-proxy-agent-7b5b4bc578-7t2hk\" (UID: \"6506fad5-cbfe-4561-8260-4f63526bbc42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b5b4bc578-7t2hk"
Jun 02 15:47:15.999238 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:15.999145    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/6506fad5-cbfe-4561-8260-4f63526bbc42-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-7b5b4bc578-7t2hk\" (UID: \"6506fad5-cbfe-4561-8260-4f63526bbc42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b5b4bc578-7t2hk"
Jun 02 15:47:15.999238 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:15.999168    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vmp94\" (UniqueName: \"kubernetes.io/projected/6506fad5-cbfe-4561-8260-4f63526bbc42-kube-api-access-vmp94\") pod \"cluster-proxy-proxy-agent-7b5b4bc578-7t2hk\" (UID: \"6506fad5-cbfe-4561-8260-4f63526bbc42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b5b4bc578-7t2hk"
Jun 02 15:47:15.999238 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:15.999193    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6fdc7974-b9aa-4b56-a19a-1741e5bcc5c3-tmp\") pod \"klusterlet-addon-workmgr-85479cc8c6-79tdg\" (UID: \"6fdc7974-b9aa-4b56-a19a-1741e5bcc5c3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85479cc8c6-79tdg"
Jun 02 15:47:15.999238 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:15.999217    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6fdc7974-b9aa-4b56-a19a-1741e5bcc5c3-klusterlet-config\") pod \"klusterlet-addon-workmgr-85479cc8c6-79tdg\" (UID: \"6fdc7974-b9aa-4b56-a19a-1741e5bcc5c3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85479cc8c6-79tdg"
Jun 02 15:47:15.999531 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:15.999243    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6506fad5-cbfe-4561-8260-4f63526bbc42-ca\") pod \"cluster-proxy-proxy-agent-7b5b4bc578-7t2hk\" (UID: \"6506fad5-cbfe-4561-8260-4f63526bbc42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b5b4bc578-7t2hk"
Jun 02 15:47:15.999531 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:15.999271    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6506fad5-cbfe-4561-8260-4f63526bbc42-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7b5b4bc578-7t2hk\" (UID: \"6506fad5-cbfe-4561-8260-4f63526bbc42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b5b4bc578-7t2hk"
Jun 02 15:47:15.999531 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:15.999296    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mtn2j\" (UniqueName: \"kubernetes.io/projected/6fdc7974-b9aa-4b56-a19a-1741e5bcc5c3-kube-api-access-mtn2j\") pod \"klusterlet-addon-workmgr-85479cc8c6-79tdg\" (UID: \"6fdc7974-b9aa-4b56-a19a-1741e5bcc5c3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85479cc8c6-79tdg"
Jun 02 15:47:16.000850 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:16.000198    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/6506fad5-cbfe-4561-8260-4f63526bbc42-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-7b5b4bc578-7t2hk\" (UID: \"6506fad5-cbfe-4561-8260-4f63526bbc42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b5b4bc578-7t2hk"
Jun 02 15:47:16.000850 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:16.000558    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6fdc7974-b9aa-4b56-a19a-1741e5bcc5c3-tmp\") pod \"klusterlet-addon-workmgr-85479cc8c6-79tdg\" (UID: \"6fdc7974-b9aa-4b56-a19a-1741e5bcc5c3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85479cc8c6-79tdg"
Jun 02 15:47:16.002539 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:16.002512    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6506fad5-cbfe-4561-8260-4f63526bbc42-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7b5b4bc578-7t2hk\" (UID: \"6506fad5-cbfe-4561-8260-4f63526bbc42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b5b4bc578-7t2hk"
Jun 02 15:47:16.002799 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:16.002782    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6506fad5-cbfe-4561-8260-4f63526bbc42-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7b5b4bc578-7t2hk\" (UID: \"6506fad5-cbfe-4561-8260-4f63526bbc42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b5b4bc578-7t2hk"
Jun 02 15:47:16.002886 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:16.002798    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6506fad5-cbfe-4561-8260-4f63526bbc42-ca\") pod \"cluster-proxy-proxy-agent-7b5b4bc578-7t2hk\" (UID: \"6506fad5-cbfe-4561-8260-4f63526bbc42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b5b4bc578-7t2hk"
Jun 02 15:47:16.002933 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:16.002898    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6506fad5-cbfe-4561-8260-4f63526bbc42-hub\") pod \"cluster-proxy-proxy-agent-7b5b4bc578-7t2hk\" (UID: \"6506fad5-cbfe-4561-8260-4f63526bbc42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b5b4bc578-7t2hk"
Jun 02 15:47:16.003311 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:16.003294    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6fdc7974-b9aa-4b56-a19a-1741e5bcc5c3-klusterlet-config\") pod \"klusterlet-addon-workmgr-85479cc8c6-79tdg\" (UID: \"6fdc7974-b9aa-4b56-a19a-1741e5bcc5c3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85479cc8c6-79tdg"
Jun 02 15:47:16.008833 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:16.008809    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtn2j\" (UniqueName: \"kubernetes.io/projected/6fdc7974-b9aa-4b56-a19a-1741e5bcc5c3-kube-api-access-mtn2j\") pod \"klusterlet-addon-workmgr-85479cc8c6-79tdg\" (UID: \"6fdc7974-b9aa-4b56-a19a-1741e5bcc5c3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85479cc8c6-79tdg"
Jun 02 15:47:16.008966 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:16.008945    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmp94\" (UniqueName: \"kubernetes.io/projected/6506fad5-cbfe-4561-8260-4f63526bbc42-kube-api-access-vmp94\") pod \"cluster-proxy-proxy-agent-7b5b4bc578-7t2hk\" (UID: \"6506fad5-cbfe-4561-8260-4f63526bbc42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b5b4bc578-7t2hk"
Jun 02 15:47:16.034598 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:16.034569    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vh7pj"]
Jun 02 15:47:16.038222 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:47:16.038189    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32e4470f_53cd_4fda_b78b_404f9e5d45c3.slice/crio-84c044e695c2642c42e74be9ab3f60eb83a272265fb58abee1b11b05b8f3b42c WatchSource:0}: Error finding container 84c044e695c2642c42e74be9ab3f60eb83a272265fb58abee1b11b05b8f3b42c: Status 404 returned error can't find the container with id 84c044e695c2642c42e74be9ab3f60eb83a272265fb58abee1b11b05b8f3b42c
Jun 02 15:47:16.145133 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:16.145029    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85479cc8c6-79tdg"
Jun 02 15:47:16.181969 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:16.181934    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b5b4bc578-7t2hk"
Jun 02 15:47:16.295954 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:16.295915    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-85479cc8c6-79tdg"]
Jun 02 15:47:16.305636 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:47:16.305586    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fdc7974_b9aa_4b56_a19a_1741e5bcc5c3.slice/crio-d4e884da2f4b45a2715b258c2293d71086eefa44918327f534189576cde6df38 WatchSource:0}: Error finding container d4e884da2f4b45a2715b258c2293d71086eefa44918327f534189576cde6df38: Status 404 returned error can't find the container with id d4e884da2f4b45a2715b258c2293d71086eefa44918327f534189576cde6df38
Jun 02 15:47:16.328547 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:16.328512    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85479cc8c6-79tdg" event={"ID":"6fdc7974-b9aa-4b56-a19a-1741e5bcc5c3","Type":"ContainerStarted","Data":"d4e884da2f4b45a2715b258c2293d71086eefa44918327f534189576cde6df38"}
Jun 02 15:47:16.329645 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:16.329609    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vh7pj" event={"ID":"32e4470f-53cd-4fda-b78b-404f9e5d45c3","Type":"ContainerStarted","Data":"84c044e695c2642c42e74be9ab3f60eb83a272265fb58abee1b11b05b8f3b42c"}
Jun 02 15:47:16.355979 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:16.355946    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b5b4bc578-7t2hk"]
Jun 02 15:47:16.359267 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:47:16.359238    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6506fad5_cbfe_4561_8260_4f63526bbc42.slice/crio-f67e589674517687d1dfecc4a9d9ae5159bafc40457ee79738a288e1314c2eb1 WatchSource:0}: Error finding container f67e589674517687d1dfecc4a9d9ae5159bafc40457ee79738a288e1314c2eb1: Status 404 returned error can't find the container with id f67e589674517687d1dfecc4a9d9ae5159bafc40457ee79738a288e1314c2eb1
Jun 02 15:47:17.334682 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:17.334622    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b5b4bc578-7t2hk" event={"ID":"6506fad5-cbfe-4561-8260-4f63526bbc42","Type":"ContainerStarted","Data":"f67e589674517687d1dfecc4a9d9ae5159bafc40457ee79738a288e1314c2eb1"}
Jun 02 15:47:22.156733 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:22.156691    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a9d0209-8692-4052-bb84-c3b8bb9d383e-cert\") pod \"ingress-canary-cqh44\" (UID: \"5a9d0209-8692-4052-bb84-c3b8bb9d383e\") " pod="openshift-ingress-canary/ingress-canary-cqh44"
Jun 02 15:47:22.157147 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:22.156847    2579 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:47:22.157147 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:22.156922    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a9d0209-8692-4052-bb84-c3b8bb9d383e-cert podName:5a9d0209-8692-4052-bb84-c3b8bb9d383e nodeName:}" failed. No retries permitted until 2026-06-02 15:47:38.15690553 +0000 UTC m=+66.682813939 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5a9d0209-8692-4052-bb84-c3b8bb9d383e-cert") pod "ingress-canary-cqh44" (UID: "5a9d0209-8692-4052-bb84-c3b8bb9d383e") : secret "canary-serving-cert" not found
Jun 02 15:47:22.257534 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:22.257494    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/666f2157-8d74-421e-af17-78b99e918795-metrics-tls\") pod \"dns-default-bgsrq\" (UID: \"666f2157-8d74-421e-af17-78b99e918795\") " pod="openshift-dns/dns-default-bgsrq"
Jun 02 15:47:22.257702 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:22.257656    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:47:22.257745 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:22.257720    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/666f2157-8d74-421e-af17-78b99e918795-metrics-tls podName:666f2157-8d74-421e-af17-78b99e918795 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:38.257704148 +0000 UTC m=+66.783612562 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/666f2157-8d74-421e-af17-78b99e918795-metrics-tls") pod "dns-default-bgsrq" (UID: "666f2157-8d74-421e-af17-78b99e918795") : secret "dns-default-metrics-tls" not found
Jun 02 15:47:22.346842 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:22.346805    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vh7pj" event={"ID":"32e4470f-53cd-4fda-b78b-404f9e5d45c3","Type":"ContainerStarted","Data":"b0709483cf7b09f8dacfc663dbe2101dad24374d9ea0957f33a82c26cad77591"}
Jun 02 15:47:22.348208 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:22.348179    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b5b4bc578-7t2hk" event={"ID":"6506fad5-cbfe-4561-8260-4f63526bbc42","Type":"ContainerStarted","Data":"67b3432677492a41bb9909f8f7292410f2eeb43eb8c212e52ed6ae5a9842d516"}
Jun 02 15:47:33.303757 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:33.303725    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lqx9k"
Jun 02 15:47:33.344531 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:33.344450    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vh7pj" podStartSLOduration=44.709046262 podStartE2EDuration="50.344434483s" podCreationTimestamp="2026-06-02 15:46:43 +0000 UTC" firstStartedPulling="2026-06-02 15:47:16.040381778 +0000 UTC m=+44.566290190" lastFinishedPulling="2026-06-02 15:47:21.675769997 +0000 UTC m=+50.201678411" observedRunningTime="2026-06-02 15:47:22.377189767 +0000 UTC m=+50.903098199" watchObservedRunningTime="2026-06-02 15:47:33.344434483 +0000 UTC m=+61.870342913"
Jun 02 15:47:37.779158 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:37.779120    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca-metrics-certs\") pod \"network-metrics-daemon-6lvc8\" (UID: \"eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca\") " pod="openshift-multus/network-metrics-daemon-6lvc8"
Jun 02 15:47:37.782271 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:37.782252    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jun 02 15:47:37.789752 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:37.789725    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jun 02 15:47:37.789879 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:37.789815    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca-metrics-certs podName:eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca nodeName:}" failed. No retries permitted until 2026-06-02 15:48:41.789795774 +0000 UTC m=+130.315704187 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca-metrics-certs") pod "network-metrics-daemon-6lvc8" (UID: "eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca") : secret "metrics-daemon-secret" not found
Jun 02 15:47:37.879694 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:37.879652    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d87j5\" (UniqueName: \"kubernetes.io/projected/43370289-88af-4e83-b409-4e8a98e09857-kube-api-access-d87j5\") pod \"network-check-target-zzt45\" (UID: \"43370289-88af-4e83-b409-4e8a98e09857\") " pod="openshift-network-diagnostics/network-check-target-zzt45"
Jun 02 15:47:37.882549 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:37.882531    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jun 02 15:47:37.893436 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:37.893414    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jun 02 15:47:37.904324 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:37.904293    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d87j5\" (UniqueName: \"kubernetes.io/projected/43370289-88af-4e83-b409-4e8a98e09857-kube-api-access-d87j5\") pod \"network-check-target-zzt45\" (UID: \"43370289-88af-4e83-b409-4e8a98e09857\") " pod="openshift-network-diagnostics/network-check-target-zzt45"
Jun 02 15:47:38.090250 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:38.090158    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gq5xh\""
Jun 02 15:47:38.097557 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:38.097533    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzt45"
Jun 02 15:47:38.181827 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:38.181798    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a9d0209-8692-4052-bb84-c3b8bb9d383e-cert\") pod \"ingress-canary-cqh44\" (UID: \"5a9d0209-8692-4052-bb84-c3b8bb9d383e\") " pod="openshift-ingress-canary/ingress-canary-cqh44"
Jun 02 15:47:38.182006 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:38.181984    2579 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:47:38.182078 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:38.182066    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a9d0209-8692-4052-bb84-c3b8bb9d383e-cert podName:5a9d0209-8692-4052-bb84-c3b8bb9d383e nodeName:}" failed. No retries permitted until 2026-06-02 15:48:10.182045547 +0000 UTC m=+98.707953957 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5a9d0209-8692-4052-bb84-c3b8bb9d383e-cert") pod "ingress-canary-cqh44" (UID: "5a9d0209-8692-4052-bb84-c3b8bb9d383e") : secret "canary-serving-cert" not found
Jun 02 15:47:38.231897 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:38.231859    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zzt45"]
Jun 02 15:47:38.234766 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:47:38.234739    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43370289_88af_4e83_b409_4e8a98e09857.slice/crio-03af87da2d8e76ca845e34ff23bed9c708ee626f4279a0a758d6035cd9c0f483 WatchSource:0}: Error finding container 03af87da2d8e76ca845e34ff23bed9c708ee626f4279a0a758d6035cd9c0f483: Status 404 returned error can't find the container with id 03af87da2d8e76ca845e34ff23bed9c708ee626f4279a0a758d6035cd9c0f483
Jun 02 15:47:38.282568 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:38.282531    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/666f2157-8d74-421e-af17-78b99e918795-metrics-tls\") pod \"dns-default-bgsrq\" (UID: \"666f2157-8d74-421e-af17-78b99e918795\") " pod="openshift-dns/dns-default-bgsrq"
Jun 02 15:47:38.282776 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:38.282689    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:47:38.282776 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:47:38.282773    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/666f2157-8d74-421e-af17-78b99e918795-metrics-tls podName:666f2157-8d74-421e-af17-78b99e918795 nodeName:}" failed. No retries permitted until 2026-06-02 15:48:10.282749597 +0000 UTC m=+98.808658024 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/666f2157-8d74-421e-af17-78b99e918795-metrics-tls") pod "dns-default-bgsrq" (UID: "666f2157-8d74-421e-af17-78b99e918795") : secret "dns-default-metrics-tls" not found
Jun 02 15:47:38.381728 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:38.381629    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zzt45" event={"ID":"43370289-88af-4e83-b409-4e8a98e09857","Type":"ContainerStarted","Data":"03af87da2d8e76ca845e34ff23bed9c708ee626f4279a0a758d6035cd9c0f483"}
Jun 02 15:47:41.389855 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:41.389811    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zzt45" event={"ID":"43370289-88af-4e83-b409-4e8a98e09857","Type":"ContainerStarted","Data":"b9bbef4de206023dbee55113342004f1310e1fa07d075875f99e8b4db2a3d3ed"}
Jun 02 15:47:41.390300 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:41.389992    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zzt45"
Jun 02 15:47:41.412263 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:41.412207    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zzt45" podStartSLOduration=66.378374554 podStartE2EDuration="1m9.412186885s" podCreationTimestamp="2026-06-02 15:46:32 +0000 UTC" firstStartedPulling="2026-06-02 15:47:38.236575823 +0000 UTC m=+66.762484245" lastFinishedPulling="2026-06-02 15:47:41.270388164 +0000 UTC m=+69.796296576" observedRunningTime="2026-06-02 15:47:41.409448693 +0000 UTC m=+69.935357125" watchObservedRunningTime="2026-06-02 15:47:41.412186885 +0000 UTC m=+69.938095313"
Jun 02 15:47:42.394148 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:42.394108    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b5b4bc578-7t2hk" event={"ID":"6506fad5-cbfe-4561-8260-4f63526bbc42","Type":"ContainerStarted","Data":"2aaf6f6a6675c8fb260b264d713e60ecb60ceb2081bd20b0d073a68edcd1b447"}
Jun 02 15:47:42.394148 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:42.394148    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b5b4bc578-7t2hk" event={"ID":"6506fad5-cbfe-4561-8260-4f63526bbc42","Type":"ContainerStarted","Data":"bedc83d791be3058b03b6dfc55dc12cc43f16bf6760862eba9452c798dc34e2d"}
Jun 02 15:47:42.431010 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:42.430962    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b5b4bc578-7t2hk" podStartSLOduration=1.899594013 podStartE2EDuration="27.430945993s" podCreationTimestamp="2026-06-02 15:47:15 +0000 UTC" firstStartedPulling="2026-06-02 15:47:16.360995757 +0000 UTC m=+44.886904166" lastFinishedPulling="2026-06-02 15:47:41.892347735 +0000 UTC m=+70.418256146" observedRunningTime="2026-06-02 15:47:42.428699895 +0000 UTC m=+70.954608325" watchObservedRunningTime="2026-06-02 15:47:42.430945993 +0000 UTC m=+70.956854423"
Jun 02 15:47:44.399930 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:44.399836    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85479cc8c6-79tdg" event={"ID":"6fdc7974-b9aa-4b56-a19a-1741e5bcc5c3","Type":"ContainerStarted","Data":"abacf411fb2b1638a95798ed6a848107ddab7d2549402800e064deb360cfd291"}
Jun 02 15:47:44.400346 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:44.400037    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85479cc8c6-79tdg"
Jun 02 15:47:44.401590 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:44.401571    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85479cc8c6-79tdg"
Jun 02 15:47:44.426191 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:47:44.426138    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85479cc8c6-79tdg" podStartSLOduration=1.717957076 podStartE2EDuration="29.426125634s" podCreationTimestamp="2026-06-02 15:47:15 +0000 UTC" firstStartedPulling="2026-06-02 15:47:16.307503393 +0000 UTC m=+44.833411804" lastFinishedPulling="2026-06-02 15:47:44.015671952 +0000 UTC m=+72.541580362" observedRunningTime="2026-06-02 15:47:44.424479964 +0000 UTC m=+72.950388394" watchObservedRunningTime="2026-06-02 15:47:44.426125634 +0000 UTC m=+72.952034064"
Jun 02 15:48:10.215868 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:48:10.215821    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a9d0209-8692-4052-bb84-c3b8bb9d383e-cert\") pod \"ingress-canary-cqh44\" (UID: \"5a9d0209-8692-4052-bb84-c3b8bb9d383e\") " pod="openshift-ingress-canary/ingress-canary-cqh44"
Jun 02 15:48:10.216357 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:48:10.215954    2579 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:48:10.216357 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:48:10.216023    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a9d0209-8692-4052-bb84-c3b8bb9d383e-cert podName:5a9d0209-8692-4052-bb84-c3b8bb9d383e nodeName:}" failed. No retries permitted until 2026-06-02 15:49:14.216006678 +0000 UTC m=+162.741915087 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5a9d0209-8692-4052-bb84-c3b8bb9d383e-cert") pod "ingress-canary-cqh44" (UID: "5a9d0209-8692-4052-bb84-c3b8bb9d383e") : secret "canary-serving-cert" not found
Jun 02 15:48:10.317099 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:48:10.317061    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/666f2157-8d74-421e-af17-78b99e918795-metrics-tls\") pod \"dns-default-bgsrq\" (UID: \"666f2157-8d74-421e-af17-78b99e918795\") " pod="openshift-dns/dns-default-bgsrq"
Jun 02 15:48:10.317278 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:48:10.317235    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:48:10.317328 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:48:10.317317    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/666f2157-8d74-421e-af17-78b99e918795-metrics-tls podName:666f2157-8d74-421e-af17-78b99e918795 nodeName:}" failed. No retries permitted until 2026-06-02 15:49:14.317296045 +0000 UTC m=+162.843204455 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/666f2157-8d74-421e-af17-78b99e918795-metrics-tls") pod "dns-default-bgsrq" (UID: "666f2157-8d74-421e-af17-78b99e918795") : secret "dns-default-metrics-tls" not found
Jun 02 15:48:12.396424 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:48:12.396374    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zzt45"
Jun 02 15:48:41.847376 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:48:41.847321    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca-metrics-certs\") pod \"network-metrics-daemon-6lvc8\" (UID: \"eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca\") " pod="openshift-multus/network-metrics-daemon-6lvc8"
Jun 02 15:48:41.847888 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:48:41.847498    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jun 02 15:48:41.847888 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:48:41.847572    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca-metrics-certs podName:eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca nodeName:}" failed. No retries permitted until 2026-06-02 15:50:43.847554969 +0000 UTC m=+252.373463391 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca-metrics-certs") pod "network-metrics-daemon-6lvc8" (UID: "eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca") : secret "metrics-daemon-secret" not found
Jun 02 15:48:45.259462 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:48:45.259431    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-th978_4290cf74-a174-4c3f-88b4-66b05bbe4750/dns-node-resolver/0.log"
Jun 02 15:48:45.659932 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:48:45.659836    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cwjmt_3bee98d4-d583-4ecb-b07b-9f422a9810da/node-ca/0.log"
Jun 02 15:49:06.183635 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:06.183586    2579 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b5b4bc578-7t2hk" podUID="6506fad5-cbfe-4561-8260-4f63526bbc42" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jun 02 15:49:09.418150 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:49:09.418089    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-cqh44" podUID="5a9d0209-8692-4052-bb84-c3b8bb9d383e"
Jun 02 15:49:09.439315 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:49:09.439274    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-bgsrq" podUID="666f2157-8d74-421e-af17-78b99e918795"
Jun 02 15:49:09.605353 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:09.605319    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cqh44"
Jun 02 15:49:11.081038 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:49:11.080990    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-6lvc8" podUID="eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca"
Jun 02 15:49:14.289519 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:14.289459    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a9d0209-8692-4052-bb84-c3b8bb9d383e-cert\") pod \"ingress-canary-cqh44\" (UID: \"5a9d0209-8692-4052-bb84-c3b8bb9d383e\") " pod="openshift-ingress-canary/ingress-canary-cqh44"
Jun 02 15:49:14.292028 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:14.292002    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a9d0209-8692-4052-bb84-c3b8bb9d383e-cert\") pod \"ingress-canary-cqh44\" (UID: \"5a9d0209-8692-4052-bb84-c3b8bb9d383e\") " pod="openshift-ingress-canary/ingress-canary-cqh44"
Jun 02 15:49:14.390097 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:14.390045    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/666f2157-8d74-421e-af17-78b99e918795-metrics-tls\") pod \"dns-default-bgsrq\" (UID: \"666f2157-8d74-421e-af17-78b99e918795\") " pod="openshift-dns/dns-default-bgsrq"
Jun 02 15:49:14.392555 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:14.392534    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/666f2157-8d74-421e-af17-78b99e918795-metrics-tls\") pod \"dns-default-bgsrq\" (UID: \"666f2157-8d74-421e-af17-78b99e918795\") " pod="openshift-dns/dns-default-bgsrq"
Jun 02 15:49:14.410764 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:14.410737    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-cw7rp\""
Jun 02 15:49:14.416318 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:14.416304    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cqh44"
Jun 02 15:49:14.552581 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:14.552486    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cqh44"]
Jun 02 15:49:14.555291 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:49:14.555260    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a9d0209_8692_4052_bb84_c3b8bb9d383e.slice/crio-71fef1676bea5a9c49b9dfdb38bfa76f764fe8558dfc10af049c02d57f91c721 WatchSource:0}: Error finding container 71fef1676bea5a9c49b9dfdb38bfa76f764fe8558dfc10af049c02d57f91c721: Status 404 returned error can't find the container with id 71fef1676bea5a9c49b9dfdb38bfa76f764fe8558dfc10af049c02d57f91c721
Jun 02 15:49:14.618031 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:14.617997    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cqh44" event={"ID":"5a9d0209-8692-4052-bb84-c3b8bb9d383e","Type":"ContainerStarted","Data":"71fef1676bea5a9c49b9dfdb38bfa76f764fe8558dfc10af049c02d57f91c721"}
Jun 02 15:49:16.183748 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:16.183699    2579 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b5b4bc578-7t2hk" podUID="6506fad5-cbfe-4561-8260-4f63526bbc42" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jun 02 15:49:16.624273 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:16.624229    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cqh44" event={"ID":"5a9d0209-8692-4052-bb84-c3b8bb9d383e","Type":"ContainerStarted","Data":"f7591e653373c68ec3d6cf42a8dff61f699fec61f1235b78c5190b7217b1a585"}
Jun 02 15:49:16.656332 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:16.656281    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-cqh44" podStartSLOduration=129.109279371 podStartE2EDuration="2m10.65626621s" podCreationTimestamp="2026-06-02 15:47:06 +0000 UTC" firstStartedPulling="2026-06-02 15:49:14.557091297 +0000 UTC m=+163.082999719" lastFinishedPulling="2026-06-02 15:49:16.10407814 +0000 UTC m=+164.629986558" observedRunningTime="2026-06-02 15:49:16.655312215 +0000 UTC m=+165.181220646" watchObservedRunningTime="2026-06-02 15:49:16.65626621 +0000 UTC m=+165.182174642"
Jun 02 15:49:17.442886 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.442856    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-82hzh"]
Jun 02 15:49:17.446057 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.446035    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-82hzh"
Jun 02 15:49:17.468778 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.468746    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jun 02 15:49:17.469287 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.469258    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jun 02 15:49:17.469287 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.469259    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jun 02 15:49:17.469536 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.469440    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cr5dk\""
Jun 02 15:49:17.477302 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.477284    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jun 02 15:49:17.497756 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.497720    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-82hzh"]
Jun 02 15:49:17.518072 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.518040    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b959e92c-6548-4cb4-945c-e229516c5e9c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-82hzh\" (UID: \"b959e92c-6548-4cb4-945c-e229516c5e9c\") " pod="openshift-insights/insights-runtime-extractor-82hzh"
Jun 02 15:49:17.518214 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.518085    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b959e92c-6548-4cb4-945c-e229516c5e9c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-82hzh\" (UID: \"b959e92c-6548-4cb4-945c-e229516c5e9c\") " pod="openshift-insights/insights-runtime-extractor-82hzh"
Jun 02 15:49:17.518214 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.518109    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2bp4\" (UniqueName: \"kubernetes.io/projected/b959e92c-6548-4cb4-945c-e229516c5e9c-kube-api-access-w2bp4\") pod \"insights-runtime-extractor-82hzh\" (UID: \"b959e92c-6548-4cb4-945c-e229516c5e9c\") " pod="openshift-insights/insights-runtime-extractor-82hzh"
Jun 02 15:49:17.518312 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.518220    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b959e92c-6548-4cb4-945c-e229516c5e9c-crio-socket\") pod \"insights-runtime-extractor-82hzh\" (UID: \"b959e92c-6548-4cb4-945c-e229516c5e9c\") " pod="openshift-insights/insights-runtime-extractor-82hzh"
Jun 02 15:49:17.518312 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.518264    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b959e92c-6548-4cb4-945c-e229516c5e9c-data-volume\") pod \"insights-runtime-extractor-82hzh\" (UID: \"b959e92c-6548-4cb4-945c-e229516c5e9c\") " pod="openshift-insights/insights-runtime-extractor-82hzh"
Jun 02 15:49:17.619596 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.619564    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b959e92c-6548-4cb4-945c-e229516c5e9c-crio-socket\") pod \"insights-runtime-extractor-82hzh\" (UID: \"b959e92c-6548-4cb4-945c-e229516c5e9c\") " pod="openshift-insights/insights-runtime-extractor-82hzh"
Jun 02 15:49:17.619596 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.619600    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b959e92c-6548-4cb4-945c-e229516c5e9c-data-volume\") pod \"insights-runtime-extractor-82hzh\" (UID: \"b959e92c-6548-4cb4-945c-e229516c5e9c\") " pod="openshift-insights/insights-runtime-extractor-82hzh"
Jun 02 15:49:17.619796 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.619642    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b959e92c-6548-4cb4-945c-e229516c5e9c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-82hzh\" (UID: \"b959e92c-6548-4cb4-945c-e229516c5e9c\") " pod="openshift-insights/insights-runtime-extractor-82hzh"
Jun 02 15:49:17.619796 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.619661    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b959e92c-6548-4cb4-945c-e229516c5e9c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-82hzh\" (UID: \"b959e92c-6548-4cb4-945c-e229516c5e9c\") " pod="openshift-insights/insights-runtime-extractor-82hzh"
Jun 02 15:49:17.619796 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.619678    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w2bp4\" (UniqueName: \"kubernetes.io/projected/b959e92c-6548-4cb4-945c-e229516c5e9c-kube-api-access-w2bp4\") pod \"insights-runtime-extractor-82hzh\" (UID: \"b959e92c-6548-4cb4-945c-e229516c5e9c\") " pod="openshift-insights/insights-runtime-extractor-82hzh"
Jun 02 15:49:17.619796 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.619686    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b959e92c-6548-4cb4-945c-e229516c5e9c-crio-socket\") pod \"insights-runtime-extractor-82hzh\" (UID: \"b959e92c-6548-4cb4-945c-e229516c5e9c\") " pod="openshift-insights/insights-runtime-extractor-82hzh"
Jun 02 15:49:17.620082 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.620063    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b959e92c-6548-4cb4-945c-e229516c5e9c-data-volume\") pod \"insights-runtime-extractor-82hzh\" (UID: \"b959e92c-6548-4cb4-945c-e229516c5e9c\") " pod="openshift-insights/insights-runtime-extractor-82hzh"
Jun 02 15:49:17.620254 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.620236    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b959e92c-6548-4cb4-945c-e229516c5e9c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-82hzh\" (UID: \"b959e92c-6548-4cb4-945c-e229516c5e9c\") " pod="openshift-insights/insights-runtime-extractor-82hzh"
Jun 02 15:49:17.622191 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.622175    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b959e92c-6548-4cb4-945c-e229516c5e9c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-82hzh\" (UID: \"b959e92c-6548-4cb4-945c-e229516c5e9c\") " pod="openshift-insights/insights-runtime-extractor-82hzh"
Jun 02 15:49:17.657728 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.657701    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2bp4\" (UniqueName: \"kubernetes.io/projected/b959e92c-6548-4cb4-945c-e229516c5e9c-kube-api-access-w2bp4\") pod \"insights-runtime-extractor-82hzh\" (UID: \"b959e92c-6548-4cb4-945c-e229516c5e9c\") " pod="openshift-insights/insights-runtime-extractor-82hzh"
Jun 02 15:49:17.724985 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.724906    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-77b6f9cc7b-d8g9c"]
Jun 02 15:49:17.727972 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.727956    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77b6f9cc7b-d8g9c"
Jun 02 15:49:17.744488 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.744463    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jun 02 15:49:17.754142 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.754112    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-82hzh"
Jun 02 15:49:17.754287 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.754206    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jun 02 15:49:17.754287 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.754254    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jun 02 15:49:17.754392 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.754347    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nsswx\""
Jun 02 15:49:17.755099 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.754758    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jun 02 15:49:17.775249 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.775219    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77b6f9cc7b-d8g9c"]
Jun 02 15:49:17.821803 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.821768    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00f0d738-668c-4e32-99d0-137724dca252-trusted-ca\") pod \"image-registry-77b6f9cc7b-d8g9c\" (UID: \"00f0d738-668c-4e32-99d0-137724dca252\") " pod="openshift-image-registry/image-registry-77b6f9cc7b-d8g9c"
Jun 02 15:49:17.821977 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.821816    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwfll\" (UniqueName: \"kubernetes.io/projected/00f0d738-668c-4e32-99d0-137724dca252-kube-api-access-fwfll\") pod \"image-registry-77b6f9cc7b-d8g9c\" (UID: \"00f0d738-668c-4e32-99d0-137724dca252\") " pod="openshift-image-registry/image-registry-77b6f9cc7b-d8g9c"
Jun 02 15:49:17.821977 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.821860    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/00f0d738-668c-4e32-99d0-137724dca252-registry-certificates\") pod \"image-registry-77b6f9cc7b-d8g9c\" (UID: \"00f0d738-668c-4e32-99d0-137724dca252\") " pod="openshift-image-registry/image-registry-77b6f9cc7b-d8g9c"
Jun 02 15:49:17.821977 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.821886    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/00f0d738-668c-4e32-99d0-137724dca252-image-registry-private-configuration\") pod \"image-registry-77b6f9cc7b-d8g9c\" (UID: \"00f0d738-668c-4e32-99d0-137724dca252\") " pod="openshift-image-registry/image-registry-77b6f9cc7b-d8g9c"
Jun 02 15:49:17.821977 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.821917    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/00f0d738-668c-4e32-99d0-137724dca252-installation-pull-secrets\") pod \"image-registry-77b6f9cc7b-d8g9c\" (UID: \"00f0d738-668c-4e32-99d0-137724dca252\") " pod="openshift-image-registry/image-registry-77b6f9cc7b-d8g9c"
Jun 02 15:49:17.821977 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.821943    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/00f0d738-668c-4e32-99d0-137724dca252-ca-trust-extracted\") pod \"image-registry-77b6f9cc7b-d8g9c\" (UID: \"00f0d738-668c-4e32-99d0-137724dca252\") " pod="openshift-image-registry/image-registry-77b6f9cc7b-d8g9c"
Jun 02 15:49:17.821977 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.821970    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00f0d738-668c-4e32-99d0-137724dca252-registry-tls\") pod \"image-registry-77b6f9cc7b-d8g9c\" (UID: \"00f0d738-668c-4e32-99d0-137724dca252\") " pod="openshift-image-registry/image-registry-77b6f9cc7b-d8g9c"
Jun 02 15:49:17.822237 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.822011    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/00f0d738-668c-4e32-99d0-137724dca252-bound-sa-token\") pod \"image-registry-77b6f9cc7b-d8g9c\" (UID: \"00f0d738-668c-4e32-99d0-137724dca252\") " pod="openshift-image-registry/image-registry-77b6f9cc7b-d8g9c"
Jun 02 15:49:17.922939 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.922907    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/00f0d738-668c-4e32-99d0-137724dca252-ca-trust-extracted\") pod \"image-registry-77b6f9cc7b-d8g9c\" (UID: \"00f0d738-668c-4e32-99d0-137724dca252\") " pod="openshift-image-registry/image-registry-77b6f9cc7b-d8g9c"
Jun 02 15:49:17.922939 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.922946    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00f0d738-668c-4e32-99d0-137724dca252-registry-tls\") pod \"image-registry-77b6f9cc7b-d8g9c\" (UID: \"00f0d738-668c-4e32-99d0-137724dca252\") " pod="openshift-image-registry/image-registry-77b6f9cc7b-d8g9c"
Jun 02 15:49:17.923191 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.922979    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/00f0d738-668c-4e32-99d0-137724dca252-bound-sa-token\") pod \"image-registry-77b6f9cc7b-d8g9c\" (UID: \"00f0d738-668c-4e32-99d0-137724dca252\") " pod="openshift-image-registry/image-registry-77b6f9cc7b-d8g9c"
Jun 02 15:49:17.923191 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.923011    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00f0d738-668c-4e32-99d0-137724dca252-trusted-ca\") pod \"image-registry-77b6f9cc7b-d8g9c\" (UID: \"00f0d738-668c-4e32-99d0-137724dca252\") " pod="openshift-image-registry/image-registry-77b6f9cc7b-d8g9c"
Jun 02 15:49:17.923191 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.923027    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fwfll\" (UniqueName: \"kubernetes.io/projected/00f0d738-668c-4e32-99d0-137724dca252-kube-api-access-fwfll\") pod \"image-registry-77b6f9cc7b-d8g9c\" (UID: \"00f0d738-668c-4e32-99d0-137724dca252\") " pod="openshift-image-registry/image-registry-77b6f9cc7b-d8g9c"
Jun 02 15:49:17.923191 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.923044    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/00f0d738-668c-4e32-99d0-137724dca252-registry-certificates\") pod \"image-registry-77b6f9cc7b-d8g9c\" (UID: \"00f0d738-668c-4e32-99d0-137724dca252\") " pod="openshift-image-registry/image-registry-77b6f9cc7b-d8g9c"
Jun 02 15:49:17.923191 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.923063    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/00f0d738-668c-4e32-99d0-137724dca252-image-registry-private-configuration\") pod \"image-registry-77b6f9cc7b-d8g9c\" (UID: \"00f0d738-668c-4e32-99d0-137724dca252\") " pod="openshift-image-registry/image-registry-77b6f9cc7b-d8g9c"
Jun 02 15:49:17.923191 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.923085    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/00f0d738-668c-4e32-99d0-137724dca252-installation-pull-secrets\") pod \"image-registry-77b6f9cc7b-d8g9c\" (UID: \"00f0d738-668c-4e32-99d0-137724dca252\") " pod="openshift-image-registry/image-registry-77b6f9cc7b-d8g9c"
Jun 02 15:49:17.923503 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.923440    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/00f0d738-668c-4e32-99d0-137724dca252-ca-trust-extracted\") pod \"image-registry-77b6f9cc7b-d8g9c\" (UID: \"00f0d738-668c-4e32-99d0-137724dca252\") " pod="openshift-image-registry/image-registry-77b6f9cc7b-d8g9c"
Jun 02 15:49:17.924073 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.924046    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/00f0d738-668c-4e32-99d0-137724dca252-registry-certificates\") pod \"image-registry-77b6f9cc7b-d8g9c\" (UID: \"00f0d738-668c-4e32-99d0-137724dca252\") " pod="openshift-image-registry/image-registry-77b6f9cc7b-d8g9c"
Jun 02 15:49:17.924334 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.924315    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00f0d738-668c-4e32-99d0-137724dca252-trusted-ca\") pod \"image-registry-77b6f9cc7b-d8g9c\" (UID: \"00f0d738-668c-4e32-99d0-137724dca252\") " pod="openshift-image-registry/image-registry-77b6f9cc7b-d8g9c"
Jun 02 15:49:17.925680 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.925649    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/00f0d738-668c-4e32-99d0-137724dca252-image-registry-private-configuration\") pod \"image-registry-77b6f9cc7b-d8g9c\" (UID: \"00f0d738-668c-4e32-99d0-137724dca252\") " pod="openshift-image-registry/image-registry-77b6f9cc7b-d8g9c"
Jun 02 15:49:17.925680 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.925669    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00f0d738-668c-4e32-99d0-137724dca252-registry-tls\") pod \"image-registry-77b6f9cc7b-d8g9c\" (UID: \"00f0d738-668c-4e32-99d0-137724dca252\") " pod="openshift-image-registry/image-registry-77b6f9cc7b-d8g9c"
Jun 02 15:49:17.925820 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.925804    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/00f0d738-668c-4e32-99d0-137724dca252-installation-pull-secrets\") pod \"image-registry-77b6f9cc7b-d8g9c\" (UID: \"00f0d738-668c-4e32-99d0-137724dca252\") " pod="openshift-image-registry/image-registry-77b6f9cc7b-d8g9c"
Jun 02 15:49:17.933234 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:49:17.933210    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb959e92c_6548_4cb4_945c_e229516c5e9c.slice/crio-36c22b96923b9403fd6329b6df5f5438be211ead082cf9226b0a137560e762c7 WatchSource:0}: Error finding container 36c22b96923b9403fd6329b6df5f5438be211ead082cf9226b0a137560e762c7: Status 404 returned error can't find the container with id 36c22b96923b9403fd6329b6df5f5438be211ead082cf9226b0a137560e762c7
Jun 02 15:49:17.934012 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.933990    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-82hzh"]
Jun 02 15:49:17.945722 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.945702    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/00f0d738-668c-4e32-99d0-137724dca252-bound-sa-token\") pod \"image-registry-77b6f9cc7b-d8g9c\" (UID: \"00f0d738-668c-4e32-99d0-137724dca252\") " pod="openshift-image-registry/image-registry-77b6f9cc7b-d8g9c"
Jun 02 15:49:17.953997 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:17.953974    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwfll\" (UniqueName: \"kubernetes.io/projected/00f0d738-668c-4e32-99d0-137724dca252-kube-api-access-fwfll\") pod \"image-registry-77b6f9cc7b-d8g9c\" (UID: \"00f0d738-668c-4e32-99d0-137724dca252\") " pod="openshift-image-registry/image-registry-77b6f9cc7b-d8g9c"
Jun 02 15:49:18.036995 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:18.036952    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77b6f9cc7b-d8g9c"
Jun 02 15:49:18.190796 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:18.190609    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77b6f9cc7b-d8g9c"]
Jun 02 15:49:18.193256 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:49:18.193225    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00f0d738_668c_4e32_99d0_137724dca252.slice/crio-b3e88efedc2e8a8a765a526204944424513de541e2e52081706a542ad8d64ffd WatchSource:0}: Error finding container b3e88efedc2e8a8a765a526204944424513de541e2e52081706a542ad8d64ffd: Status 404 returned error can't find the container with id b3e88efedc2e8a8a765a526204944424513de541e2e52081706a542ad8d64ffd
Jun 02 15:49:18.630794 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:18.630760    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77b6f9cc7b-d8g9c" event={"ID":"00f0d738-668c-4e32-99d0-137724dca252","Type":"ContainerStarted","Data":"9c1caee38015babfd4cbc4e12301192c1aa8c5ff733aad292421721ca0223bae"}
Jun 02 15:49:18.630794 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:18.630806    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77b6f9cc7b-d8g9c" event={"ID":"00f0d738-668c-4e32-99d0-137724dca252","Type":"ContainerStarted","Data":"b3e88efedc2e8a8a765a526204944424513de541e2e52081706a542ad8d64ffd"}
Jun 02 15:49:18.631294 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:18.630846    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-77b6f9cc7b-d8g9c"
Jun 02 15:49:18.632116 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:18.632070    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-82hzh" event={"ID":"b959e92c-6548-4cb4-945c-e229516c5e9c","Type":"ContainerStarted","Data":"872ca56b8aa212777cdb2b1d1143c2d1f37726a9e0a4ecb8b09e7ae71119b396"}
Jun 02 15:49:18.632116 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:18.632099    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-82hzh" event={"ID":"b959e92c-6548-4cb4-945c-e229516c5e9c","Type":"ContainerStarted","Data":"36c22b96923b9403fd6329b6df5f5438be211ead082cf9226b0a137560e762c7"}
Jun 02 15:49:19.636999 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:19.636912    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-82hzh" event={"ID":"b959e92c-6548-4cb4-945c-e229516c5e9c","Type":"ContainerStarted","Data":"491207ef6694836fe866422d2859fc0359f6333351cffd0b76e30746beb6a80c"}
Jun 02 15:49:20.642261 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:20.642223    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-82hzh" event={"ID":"b959e92c-6548-4cb4-945c-e229516c5e9c","Type":"ContainerStarted","Data":"1506ba3730658e90815225ad360921284ff72c7d74508a9ec59029390d2df472"}
Jun 02 15:49:20.675018 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:20.674963    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-77b6f9cc7b-d8g9c" podStartSLOduration=3.674947795 podStartE2EDuration="3.674947795s" podCreationTimestamp="2026-06-02 15:49:17 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 15:49:18.674829459 +0000 UTC m=+167.200737884" watchObservedRunningTime="2026-06-02 15:49:20.674947795 +0000 UTC m=+169.200856225"
Jun 02 15:49:20.675740 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:20.675712    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-82hzh" podStartSLOduration=1.5258503509999999 podStartE2EDuration="3.675702593s" podCreationTimestamp="2026-06-02 15:49:17 +0000 UTC" firstStartedPulling="2026-06-02 15:49:17.986607807 +0000 UTC m=+166.512516216" lastFinishedPulling="2026-06-02 15:49:20.136460043 +0000 UTC m=+168.662368458" observedRunningTime="2026-06-02 15:49:20.674526808 +0000 UTC m=+169.200435242" watchObservedRunningTime="2026-06-02 15:49:20.675702593 +0000 UTC m=+169.201611024"
Jun 02 15:49:23.059362 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:23.059308    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6lvc8"
Jun 02 15:49:24.059380 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:24.059338    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bgsrq"
Jun 02 15:49:24.063547 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:24.063524    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ht87l\""
Jun 02 15:49:24.070336 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:24.070320    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bgsrq"
Jun 02 15:49:24.204158 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:24.204127    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bgsrq"]
Jun 02 15:49:24.207796 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:49:24.207764    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod666f2157_8d74_421e_af17_78b99e918795.slice/crio-a758b042d62dad4cf947c7807d6da486c9812b8dca86eb7f018e430a015dd782 WatchSource:0}: Error finding container a758b042d62dad4cf947c7807d6da486c9812b8dca86eb7f018e430a015dd782: Status 404 returned error can't find the container with id a758b042d62dad4cf947c7807d6da486c9812b8dca86eb7f018e430a015dd782
Jun 02 15:49:24.654374 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:24.654339    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bgsrq" event={"ID":"666f2157-8d74-421e-af17-78b99e918795","Type":"ContainerStarted","Data":"a758b042d62dad4cf947c7807d6da486c9812b8dca86eb7f018e430a015dd782"}
Jun 02 15:49:25.284768 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:25.284735    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6rlgc"]
Jun 02 15:49:25.288022 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:25.287999    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6rlgc"
Jun 02 15:49:25.291044 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:25.291016    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jun 02 15:49:25.291761 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:25.291744    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jun 02 15:49:25.292169 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:25.292156    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jun 02 15:49:25.292483 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:25.292464    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jun 02 15:49:25.292583 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:25.292552    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jun 02 15:49:25.293056 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:25.293037    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jun 02 15:49:25.293137 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:25.293067    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mnlvm\""
Jun 02 15:49:25.381881 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:25.381840    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/29f69b2e-fd61-45ea-a3a9-3680485eb5fa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6rlgc\" (UID: \"29f69b2e-fd61-45ea-a3a9-3680485eb5fa\") " pod="openshift-monitoring/node-exporter-6rlgc"
Jun 02 15:49:25.381881 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:25.381886    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/29f69b2e-fd61-45ea-a3a9-3680485eb5fa-node-exporter-accelerators-collector-config\") pod \"node-exporter-6rlgc\" (UID: \"29f69b2e-fd61-45ea-a3a9-3680485eb5fa\") " pod="openshift-monitoring/node-exporter-6rlgc"
Jun 02 15:49:25.382125 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:25.381986    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/29f69b2e-fd61-45ea-a3a9-3680485eb5fa-root\") pod \"node-exporter-6rlgc\" (UID: \"29f69b2e-fd61-45ea-a3a9-3680485eb5fa\") " pod="openshift-monitoring/node-exporter-6rlgc"
Jun 02 15:49:25.382125 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:25.382029    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/29f69b2e-fd61-45ea-a3a9-3680485eb5fa-node-exporter-tls\") pod \"node-exporter-6rlgc\" (UID: \"29f69b2e-fd61-45ea-a3a9-3680485eb5fa\") " pod="openshift-monitoring/node-exporter-6rlgc"
Jun 02 15:49:25.382125 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:25.382100    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/29f69b2e-fd61-45ea-a3a9-3680485eb5fa-node-exporter-wtmp\") pod \"node-exporter-6rlgc\" (UID: \"29f69b2e-fd61-45ea-a3a9-3680485eb5fa\") " pod="openshift-monitoring/node-exporter-6rlgc"
Jun 02 15:49:25.382239 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:25.382127    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29f69b2e-fd61-45ea-a3a9-3680485eb5fa-sys\") pod \"node-exporter-6rlgc\" (UID: \"29f69b2e-fd61-45ea-a3a9-3680485eb5fa\") " pod="openshift-monitoring/node-exporter-6rlgc"
Jun 02 15:49:25.382239 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:25.382151    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww9xr\" (UniqueName: \"kubernetes.io/projected/29f69b2e-fd61-45ea-a3a9-3680485eb5fa-kube-api-access-ww9xr\") pod \"node-exporter-6rlgc\" (UID: \"29f69b2e-fd61-45ea-a3a9-3680485eb5fa\") " pod="openshift-monitoring/node-exporter-6rlgc"
Jun 02 15:49:25.382239 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:25.382176    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29f69b2e-fd61-45ea-a3a9-3680485eb5fa-metrics-client-ca\") pod \"node-exporter-6rlgc\" (UID: \"29f69b2e-fd61-45ea-a3a9-3680485eb5fa\") " pod="openshift-monitoring/node-exporter-6rlgc"
Jun 02 15:49:25.382239 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:25.382204    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/29f69b2e-fd61-45ea-a3a9-3680485eb5fa-node-exporter-textfile\") pod \"node-exporter-6rlgc\" (UID: \"29f69b2e-fd61-45ea-a3a9-3680485eb5fa\") " pod="openshift-monitoring/node-exporter-6rlgc"
Jun 02 15:49:25.483017 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:25.482976    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/29f69b2e-fd61-45ea-a3a9-3680485eb5fa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6rlgc\" (UID: \"29f69b2e-fd61-45ea-a3a9-3680485eb5fa\") " pod="openshift-monitoring/node-exporter-6rlgc"
Jun 02 15:49:25.483192 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:25.483030    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/29f69b2e-fd61-45ea-a3a9-3680485eb5fa-node-exporter-accelerators-collector-config\") pod \"node-exporter-6rlgc\" (UID: \"29f69b2e-fd61-45ea-a3a9-3680485eb5fa\") " pod="openshift-monitoring/node-exporter-6rlgc"
Jun 02 15:49:25.483192 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:25.483081    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/29f69b2e-fd61-45ea-a3a9-3680485eb5fa-root\") pod \"node-exporter-6rlgc\" (UID: \"29f69b2e-fd61-45ea-a3a9-3680485eb5fa\") " pod="openshift-monitoring/node-exporter-6rlgc"
Jun 02 15:49:25.483192 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:25.483154    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/29f69b2e-fd61-45ea-a3a9-3680485eb5fa-root\") pod \"node-exporter-6rlgc\" (UID: \"29f69b2e-fd61-45ea-a3a9-3680485eb5fa\") " pod="openshift-monitoring/node-exporter-6rlgc"
Jun 02 15:49:25.483305 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:25.483213    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/29f69b2e-fd61-45ea-a3a9-3680485eb5fa-node-exporter-tls\") pod \"node-exporter-6rlgc\" (UID: \"29f69b2e-fd61-45ea-a3a9-3680485eb5fa\") " pod="openshift-monitoring/node-exporter-6rlgc"
Jun 02 15:49:25.483305 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:25.483270    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/29f69b2e-fd61-45ea-a3a9-3680485eb5fa-node-exporter-wtmp\") pod \"node-exporter-6rlgc\" (UID: \"29f69b2e-fd61-45ea-a3a9-3680485eb5fa\") " pod="openshift-monitoring/node-exporter-6rlgc"
Jun 02 15:49:25.483393 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:25.483303    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29f69b2e-fd61-45ea-a3a9-3680485eb5fa-sys\") pod \"node-exporter-6rlgc\" (UID: \"29f69b2e-fd61-45ea-a3a9-3680485eb5fa\") " pod="openshift-monitoring/node-exporter-6rlgc"
Jun 02 15:49:25.483393 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:25.483329    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ww9xr\" (UniqueName: \"kubernetes.io/projected/29f69b2e-fd61-45ea-a3a9-3680485eb5fa-kube-api-access-ww9xr\") pod \"node-exporter-6rlgc\" (UID: \"29f69b2e-fd61-45ea-a3a9-3680485eb5fa\") " pod="openshift-monitoring/node-exporter-6rlgc"
Jun 02 15:49:25.483393 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:49:25.483344    2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jun 02 15:49:25.483548 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:25.483427    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29f69b2e-fd61-45ea-a3a9-3680485eb5fa-sys\") pod \"node-exporter-6rlgc\" (UID: \"29f69b2e-fd61-45ea-a3a9-3680485eb5fa\") " pod="openshift-monitoring/node-exporter-6rlgc"
Jun 02 15:49:25.483548 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:49:25.483444    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29f69b2e-fd61-45ea-a3a9-3680485eb5fa-node-exporter-tls podName:29f69b2e-fd61-45ea-a3a9-3680485eb5fa nodeName:}" failed. No retries permitted until 2026-06-02 15:49:25.983420925 +0000 UTC m=+174.509329350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/29f69b2e-fd61-45ea-a3a9-3680485eb5fa-node-exporter-tls") pod "node-exporter-6rlgc" (UID: "29f69b2e-fd61-45ea-a3a9-3680485eb5fa") : secret "node-exporter-tls" not found
Jun 02 15:49:25.483548 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:25.483482    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/29f69b2e-fd61-45ea-a3a9-3680485eb5fa-node-exporter-wtmp\") pod \"node-exporter-6rlgc\" (UID: \"29f69b2e-fd61-45ea-a3a9-3680485eb5fa\") " pod="openshift-monitoring/node-exporter-6rlgc"
Jun 02 15:49:25.483548 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:25.483524    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29f69b2e-fd61-45ea-a3a9-3680485eb5fa-metrics-client-ca\") pod \"node-exporter-6rlgc\" (UID: \"29f69b2e-fd61-45ea-a3a9-3680485eb5fa\") " pod="openshift-monitoring/node-exporter-6rlgc"
Jun 02 15:49:25.483747 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:25.483561    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/29f69b2e-fd61-45ea-a3a9-3680485eb5fa-node-exporter-textfile\") pod \"node-exporter-6rlgc\" (UID: \"29f69b2e-fd61-45ea-a3a9-3680485eb5fa\") " pod="openshift-monitoring/node-exporter-6rlgc"
Jun 02 15:49:25.483747 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:25.483716    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/29f69b2e-fd61-45ea-a3a9-3680485eb5fa-node-exporter-accelerators-collector-config\") pod \"node-exporter-6rlgc\" (UID: \"29f69b2e-fd61-45ea-a3a9-3680485eb5fa\") " pod="openshift-monitoring/node-exporter-6rlgc"
Jun 02 15:49:25.483887 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:25.483868    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/29f69b2e-fd61-45ea-a3a9-3680485eb5fa-node-exporter-textfile\") pod \"node-exporter-6rlgc\" (UID: \"29f69b2e-fd61-45ea-a3a9-3680485eb5fa\") " pod="openshift-monitoring/node-exporter-6rlgc"
Jun 02 15:49:25.484216 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:25.484185    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29f69b2e-fd61-45ea-a3a9-3680485eb5fa-metrics-client-ca\") pod \"node-exporter-6rlgc\" (UID: \"29f69b2e-fd61-45ea-a3a9-3680485eb5fa\") " pod="openshift-monitoring/node-exporter-6rlgc"
Jun 02 15:49:25.485860 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:25.485840    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/29f69b2e-fd61-45ea-a3a9-3680485eb5fa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6rlgc\" (UID: \"29f69b2e-fd61-45ea-a3a9-3680485eb5fa\") " pod="openshift-monitoring/node-exporter-6rlgc"
Jun 02 15:49:25.494608 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:25.494587    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww9xr\" (UniqueName: \"kubernetes.io/projected/29f69b2e-fd61-45ea-a3a9-3680485eb5fa-kube-api-access-ww9xr\") pod \"node-exporter-6rlgc\" (UID: \"29f69b2e-fd61-45ea-a3a9-3680485eb5fa\") " pod="openshift-monitoring/node-exporter-6rlgc"
Jun 02 15:49:25.989715 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:25.989679    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/29f69b2e-fd61-45ea-a3a9-3680485eb5fa-node-exporter-tls\") pod \"node-exporter-6rlgc\" (UID: \"29f69b2e-fd61-45ea-a3a9-3680485eb5fa\") " pod="openshift-monitoring/node-exporter-6rlgc"
Jun 02 15:49:25.992599 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:25.992574    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/29f69b2e-fd61-45ea-a3a9-3680485eb5fa-node-exporter-tls\") pod \"node-exporter-6rlgc\" (UID: \"29f69b2e-fd61-45ea-a3a9-3680485eb5fa\") " pod="openshift-monitoring/node-exporter-6rlgc"
Jun 02 15:49:26.183680 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:26.183585    2579 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b5b4bc578-7t2hk" podUID="6506fad5-cbfe-4561-8260-4f63526bbc42" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jun 02 15:49:26.183680 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:26.183667    2579 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b5b4bc578-7t2hk"
Jun 02 15:49:26.184196 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:26.184165    2579 kuberuntime_manager.go:1107] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"2aaf6f6a6675c8fb260b264d713e60ecb60ceb2081bd20b0d073a68edcd1b447"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b5b4bc578-7t2hk" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jun 02 15:49:26.184269 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:26.184228    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b5b4bc578-7t2hk" podUID="6506fad5-cbfe-4561-8260-4f63526bbc42" containerName="service-proxy" containerID="cri-o://2aaf6f6a6675c8fb260b264d713e60ecb60ceb2081bd20b0d073a68edcd1b447" gracePeriod=30
Jun 02 15:49:26.198916 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:26.198889    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6rlgc"
Jun 02 15:49:26.207197 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:49:26.207171    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29f69b2e_fd61_45ea_a3a9_3680485eb5fa.slice/crio-033cb8f2313dc26190977b938cb27285eb9fa77aae5c770d0a6c864c25ac4587 WatchSource:0}: Error finding container 033cb8f2313dc26190977b938cb27285eb9fa77aae5c770d0a6c864c25ac4587: Status 404 returned error can't find the container with id 033cb8f2313dc26190977b938cb27285eb9fa77aae5c770d0a6c864c25ac4587
Jun 02 15:49:26.663727 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:26.663688    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bgsrq" event={"ID":"666f2157-8d74-421e-af17-78b99e918795","Type":"ContainerStarted","Data":"0f9a74fcc7b5616bc76e10a32bb366651b9d1d1ad4872bd62ec99bbc459f7e66"}
Jun 02 15:49:26.663727 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:26.663724    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bgsrq" event={"ID":"666f2157-8d74-421e-af17-78b99e918795","Type":"ContainerStarted","Data":"3bfe076acc2487731eddfa45511b2ac360fcbf988517863487b90a22c94c264b"}
Jun 02 15:49:26.664239 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:26.663797    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bgsrq"
Jun 02 15:49:26.664772 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:26.664750    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6rlgc" event={"ID":"29f69b2e-fd61-45ea-a3a9-3680485eb5fa","Type":"ContainerStarted","Data":"033cb8f2313dc26190977b938cb27285eb9fa77aae5c770d0a6c864c25ac4587"}
Jun 02 15:49:26.666630 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:26.666607    2579 generic.go:358] "Generic (PLEG): container finished" podID="6506fad5-cbfe-4561-8260-4f63526bbc42" containerID="2aaf6f6a6675c8fb260b264d713e60ecb60ceb2081bd20b0d073a68edcd1b447" exitCode=2
Jun 02 15:49:26.666738 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:26.666636    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b5b4bc578-7t2hk" event={"ID":"6506fad5-cbfe-4561-8260-4f63526bbc42","Type":"ContainerDied","Data":"2aaf6f6a6675c8fb260b264d713e60ecb60ceb2081bd20b0d073a68edcd1b447"}
Jun 02 15:49:26.666738 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:26.666652    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b5b4bc578-7t2hk" event={"ID":"6506fad5-cbfe-4561-8260-4f63526bbc42","Type":"ContainerStarted","Data":"7c2f38b17b7663434a30f11ee9e87afcc237b994e690bd05d82b818447bf0098"}
Jun 02 15:49:26.692557 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:26.692497    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-bgsrq" podStartSLOduration=139.032127061 podStartE2EDuration="2m20.692481236s" podCreationTimestamp="2026-06-02 15:47:06 +0000 UTC" firstStartedPulling="2026-06-02 15:49:24.209870778 +0000 UTC m=+172.735779189" lastFinishedPulling="2026-06-02 15:49:25.870224955 +0000 UTC m=+174.396133364" observedRunningTime="2026-06-02 15:49:26.692048925 +0000 UTC m=+175.217957356" watchObservedRunningTime="2026-06-02 15:49:26.692481236 +0000 UTC m=+175.218389711"
Jun 02 15:49:27.671332 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:27.671303    2579 generic.go:358] "Generic (PLEG): container finished" podID="29f69b2e-fd61-45ea-a3a9-3680485eb5fa" containerID="61fd98c97a4058ee174db46f8ebd6cf62359aba76f2432032368a40d89181bf7" exitCode=0
Jun 02 15:49:27.671766 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:27.671417    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6rlgc" event={"ID":"29f69b2e-fd61-45ea-a3a9-3680485eb5fa","Type":"ContainerDied","Data":"61fd98c97a4058ee174db46f8ebd6cf62359aba76f2432032368a40d89181bf7"}
Jun 02 15:49:28.675993 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:28.675955    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6rlgc" event={"ID":"29f69b2e-fd61-45ea-a3a9-3680485eb5fa","Type":"ContainerStarted","Data":"6049af72b4fe7f8a244789582718f2dda8c0b09a3960711423c9ac998f08eb23"}
Jun 02 15:49:28.675993 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:28.675996    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6rlgc" event={"ID":"29f69b2e-fd61-45ea-a3a9-3680485eb5fa","Type":"ContainerStarted","Data":"26f20f8fbfd2ae204e4242b88e0289b88a5f7f4115c7538fa73bf0dfb08de103"}
Jun 02 15:49:28.706944 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:28.706888    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6rlgc" podStartSLOduration=2.830875564 podStartE2EDuration="3.706871815s" podCreationTimestamp="2026-06-02 15:49:25 +0000 UTC" firstStartedPulling="2026-06-02 15:49:26.289141103 +0000 UTC m=+174.815049516" lastFinishedPulling="2026-06-02 15:49:27.165137358 +0000 UTC m=+175.691045767" observedRunningTime="2026-06-02 15:49:28.705233393 +0000 UTC m=+177.231141823" watchObservedRunningTime="2026-06-02 15:49:28.706871815 +0000 UTC m=+177.232780272"
Jun 02 15:49:36.674423 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:36.674374    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bgsrq"
Jun 02 15:49:38.041370 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:38.041325    2579 patch_prober.go:28] interesting pod/image-registry-77b6f9cc7b-d8g9c container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jun 02 15:49:38.041777 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:38.041389    2579 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-77b6f9cc7b-d8g9c" podUID="00f0d738-668c-4e32-99d0-137724dca252" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 15:49:39.641643 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:39.641608    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-77b6f9cc7b-d8g9c"
Jun 02 15:49:44.401136 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:44.401096    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85479cc8c6-79tdg" podUID="6fdc7974-b9aa-4b56-a19a-1741e5bcc5c3" containerName="acm-agent" probeResult="failure" output="Get \"http://10.133.0.8:8000/readyz\": dial tcp 10.133.0.8:8000: connect: connection refused"
Jun 02 15:49:44.718450 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:44.718335    2579 generic.go:358] "Generic (PLEG): container finished" podID="6fdc7974-b9aa-4b56-a19a-1741e5bcc5c3" containerID="abacf411fb2b1638a95798ed6a848107ddab7d2549402800e064deb360cfd291" exitCode=1
Jun 02 15:49:44.718450 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:44.718388    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85479cc8c6-79tdg" event={"ID":"6fdc7974-b9aa-4b56-a19a-1741e5bcc5c3","Type":"ContainerDied","Data":"abacf411fb2b1638a95798ed6a848107ddab7d2549402800e064deb360cfd291"}
Jun 02 15:49:44.718822 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:44.718804    2579 scope.go:117] "RemoveContainer" containerID="abacf411fb2b1638a95798ed6a848107ddab7d2549402800e064deb360cfd291"
Jun 02 15:49:45.726006 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:45.725964    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85479cc8c6-79tdg" event={"ID":"6fdc7974-b9aa-4b56-a19a-1741e5bcc5c3","Type":"ContainerStarted","Data":"bfd068deb46ac9c4c56381c975468203c244ac86701108320a7881a87f58e4b4"}
Jun 02 15:49:45.726436 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:45.726321    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85479cc8c6-79tdg"
Jun 02 15:49:45.727035 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:49:45.727014    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85479cc8c6-79tdg"
Jun 02 15:50:20.554818 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:50:20.554788    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6rlgc_29f69b2e-fd61-45ea-a3a9-3680485eb5fa/init-textfile/0.log"
Jun 02 15:50:20.725146 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:50:20.725118    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6rlgc_29f69b2e-fd61-45ea-a3a9-3680485eb5fa/node-exporter/0.log"
Jun 02 15:50:20.921074 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:50:20.920994    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6rlgc_29f69b2e-fd61-45ea-a3a9-3680485eb5fa/kube-rbac-proxy/0.log"
Jun 02 15:50:25.321509 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:50:25.321481    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cqh44_5a9d0209-8692-4052-bb84-c3b8bb9d383e/serve-healthcheck-canary/0.log"
Jun 02 15:50:43.882890 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:50:43.882850    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca-metrics-certs\") pod \"network-metrics-daemon-6lvc8\" (UID: \"eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca\") " pod="openshift-multus/network-metrics-daemon-6lvc8"
Jun 02 15:50:43.885432 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:50:43.885391    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca-metrics-certs\") pod \"network-metrics-daemon-6lvc8\" (UID: \"eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca\") " pod="openshift-multus/network-metrics-daemon-6lvc8"
Jun 02 15:50:44.062694 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:50:44.062656    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mqvtg\""
Jun 02 15:50:44.070131 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:50:44.070113    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6lvc8"
Jun 02 15:50:44.211448 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:50:44.211396    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6lvc8"]
Jun 02 15:50:44.215166 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:50:44.215134    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb58a64a_e8f6_4492_9aa2_9c9b6bf926ca.slice/crio-65feac8a1bb215099d80f808d59d8230ba0a7da8a4458ee925841bb274000806 WatchSource:0}: Error finding container 65feac8a1bb215099d80f808d59d8230ba0a7da8a4458ee925841bb274000806: Status 404 returned error can't find the container with id 65feac8a1bb215099d80f808d59d8230ba0a7da8a4458ee925841bb274000806
Jun 02 15:50:44.876382 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:50:44.876349    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6lvc8" event={"ID":"eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca","Type":"ContainerStarted","Data":"65feac8a1bb215099d80f808d59d8230ba0a7da8a4458ee925841bb274000806"}
Jun 02 15:50:45.881330 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:50:45.881293    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6lvc8" event={"ID":"eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca","Type":"ContainerStarted","Data":"b5e4d287b7b5925e26f3fda17d1614e5745b0b594a41ced5accd2c451f0705ce"}
Jun 02 15:50:45.881330 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:50:45.881330    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6lvc8" event={"ID":"eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca","Type":"ContainerStarted","Data":"f25dd3111c68907972c168d6a61dc9778b7a15d2f28bcaf66fc8f568534133ab"}
Jun 02 15:50:45.914160 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:50:45.914107    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6lvc8" podStartSLOduration=252.855118405 podStartE2EDuration="4m13.914091101s" podCreationTimestamp="2026-06-02 15:46:32 +0000 UTC" firstStartedPulling="2026-06-02 15:50:44.216981056 +0000 UTC m=+252.742889464" lastFinishedPulling="2026-06-02 15:50:45.275953748 +0000 UTC m=+253.801862160" observedRunningTime="2026-06-02 15:50:45.913539025 +0000 UTC m=+254.439447470" watchObservedRunningTime="2026-06-02 15:50:45.914091101 +0000 UTC m=+254.439999532"
Jun 02 15:51:31.932416 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:51:31.932374    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lqx9k_6127e33c-aa77-437e-b443-e956037afedb/ovn-acl-logging/0.log"
Jun 02 15:51:31.933971 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:51:31.933946    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lqx9k_6127e33c-aa77-437e-b443-e956037afedb/ovn-acl-logging/0.log"
Jun 02 15:51:31.936373 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:51:31.936354    2579 kubelet.go:1628] "Image garbage collection succeeded"
Jun 02 15:56:31.952189 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:56:31.952159    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lqx9k_6127e33c-aa77-437e-b443-e956037afedb/ovn-acl-logging/0.log"
Jun 02 15:56:31.952788 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:56:31.952770    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lqx9k_6127e33c-aa77-437e-b443-e956037afedb/ovn-acl-logging/0.log"
Jun 02 15:59:26.907661 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:26.907624    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-djgq7"]
Jun 02 15:59:26.910571 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:26.910555    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-djgq7"
Jun 02 15:59:26.915250 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:26.915215    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jun 02 15:59:26.915436 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:26.915350    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jun 02 15:59:26.915436 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:26.915350    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jun 02 15:59:26.915562 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:26.915470    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jun 02 15:59:26.916237 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:26.916217    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jun 02 15:59:26.916340 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:26.916250    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-5rhjx\""
Jun 02 15:59:26.930741 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:26.930711    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-djgq7"]
Jun 02 15:59:26.991458 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:26.991420    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/f7c6e309-ff75-42a0-8c15-57d82cced0cb-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-djgq7\" (UID: \"f7c6e309-ff75-42a0-8c15-57d82cced0cb\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-djgq7"
Jun 02 15:59:26.991458 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:26.991462    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl2h8\" (UniqueName: \"kubernetes.io/projected/f7c6e309-ff75-42a0-8c15-57d82cced0cb-kube-api-access-pl2h8\") pod \"keda-metrics-apiserver-7c9f485588-djgq7\" (UID: \"f7c6e309-ff75-42a0-8c15-57d82cced0cb\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-djgq7"
Jun 02 15:59:26.991674 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:26.991559    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f7c6e309-ff75-42a0-8c15-57d82cced0cb-certificates\") pod \"keda-metrics-apiserver-7c9f485588-djgq7\" (UID: \"f7c6e309-ff75-42a0-8c15-57d82cced0cb\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-djgq7"
Jun 02 15:59:27.092520 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:27.092489    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f7c6e309-ff75-42a0-8c15-57d82cced0cb-certificates\") pod \"keda-metrics-apiserver-7c9f485588-djgq7\" (UID: \"f7c6e309-ff75-42a0-8c15-57d82cced0cb\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-djgq7"
Jun 02 15:59:27.092697 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:27.092539    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/f7c6e309-ff75-42a0-8c15-57d82cced0cb-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-djgq7\" (UID: \"f7c6e309-ff75-42a0-8c15-57d82cced0cb\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-djgq7"
Jun 02 15:59:27.092697 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:27.092565    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pl2h8\" (UniqueName: \"kubernetes.io/projected/f7c6e309-ff75-42a0-8c15-57d82cced0cb-kube-api-access-pl2h8\") pod \"keda-metrics-apiserver-7c9f485588-djgq7\" (UID: \"f7c6e309-ff75-42a0-8c15-57d82cced0cb\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-djgq7"
Jun 02 15:59:27.092697 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:59:27.092646    2579 secret.go:281] references non-existent secret key: tls.crt
Jun 02 15:59:27.092697 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:59:27.092671    2579 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Jun 02 15:59:27.092697 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:59:27.092693    2579 projected.go:264] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Jun 02 15:59:27.092875 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:59:27.092717    2579 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-djgq7: [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Jun 02 15:59:27.092875 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:59:27.092777    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7c6e309-ff75-42a0-8c15-57d82cced0cb-certificates podName:f7c6e309-ff75-42a0-8c15-57d82cced0cb nodeName:}" failed. No retries permitted until 2026-06-02 15:59:27.59275563 +0000 UTC m=+776.118664040 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/f7c6e309-ff75-42a0-8c15-57d82cced0cb-certificates") pod "keda-metrics-apiserver-7c9f485588-djgq7" (UID: "f7c6e309-ff75-42a0-8c15-57d82cced0cb") : [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Jun 02 15:59:27.093001 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:27.092978    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/f7c6e309-ff75-42a0-8c15-57d82cced0cb-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-djgq7\" (UID: \"f7c6e309-ff75-42a0-8c15-57d82cced0cb\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-djgq7"
Jun 02 15:59:27.115637 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:27.115612    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl2h8\" (UniqueName: \"kubernetes.io/projected/f7c6e309-ff75-42a0-8c15-57d82cced0cb-kube-api-access-pl2h8\") pod \"keda-metrics-apiserver-7c9f485588-djgq7\" (UID: \"f7c6e309-ff75-42a0-8c15-57d82cced0cb\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-djgq7"
Jun 02 15:59:27.251514 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:27.251472    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-cf49989db-z9hbj"]
Jun 02 15:59:27.255420 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:27.255379    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-cf49989db-z9hbj"
Jun 02 15:59:27.258966 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:27.258928    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jun 02 15:59:27.266542 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:27.266518    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-cf49989db-z9hbj"]
Jun 02 15:59:27.293632 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:27.293595    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2e2b5f84-d5d2-496c-8e3c-79ea79f771a9-certificates\") pod \"keda-admission-cf49989db-z9hbj\" (UID: \"2e2b5f84-d5d2-496c-8e3c-79ea79f771a9\") " pod="openshift-keda/keda-admission-cf49989db-z9hbj"
Jun 02 15:59:27.293814 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:27.293657    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwrtz\" (UniqueName: \"kubernetes.io/projected/2e2b5f84-d5d2-496c-8e3c-79ea79f771a9-kube-api-access-wwrtz\") pod \"keda-admission-cf49989db-z9hbj\" (UID: \"2e2b5f84-d5d2-496c-8e3c-79ea79f771a9\") " pod="openshift-keda/keda-admission-cf49989db-z9hbj"
Jun 02 15:59:27.394183 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:27.394138    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wwrtz\" (UniqueName: \"kubernetes.io/projected/2e2b5f84-d5d2-496c-8e3c-79ea79f771a9-kube-api-access-wwrtz\") pod \"keda-admission-cf49989db-z9hbj\" (UID: \"2e2b5f84-d5d2-496c-8e3c-79ea79f771a9\") " pod="openshift-keda/keda-admission-cf49989db-z9hbj"
Jun 02 15:59:27.394348 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:27.394214    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2e2b5f84-d5d2-496c-8e3c-79ea79f771a9-certificates\") pod \"keda-admission-cf49989db-z9hbj\" (UID: \"2e2b5f84-d5d2-496c-8e3c-79ea79f771a9\") " pod="openshift-keda/keda-admission-cf49989db-z9hbj"
Jun 02 15:59:27.394348 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:59:27.394313    2579 projected.go:264] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Jun 02 15:59:27.394348 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:59:27.394334    2579 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-cf49989db-z9hbj: secret "keda-admission-webhooks-certs" not found
Jun 02 15:59:27.394480 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:59:27.394421    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e2b5f84-d5d2-496c-8e3c-79ea79f771a9-certificates podName:2e2b5f84-d5d2-496c-8e3c-79ea79f771a9 nodeName:}" failed. No retries permitted until 2026-06-02 15:59:27.8943784 +0000 UTC m=+776.420286809 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/2e2b5f84-d5d2-496c-8e3c-79ea79f771a9-certificates") pod "keda-admission-cf49989db-z9hbj" (UID: "2e2b5f84-d5d2-496c-8e3c-79ea79f771a9") : secret "keda-admission-webhooks-certs" not found
Jun 02 15:59:27.409899 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:27.409869    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwrtz\" (UniqueName: \"kubernetes.io/projected/2e2b5f84-d5d2-496c-8e3c-79ea79f771a9-kube-api-access-wwrtz\") pod \"keda-admission-cf49989db-z9hbj\" (UID: \"2e2b5f84-d5d2-496c-8e3c-79ea79f771a9\") " pod="openshift-keda/keda-admission-cf49989db-z9hbj"
Jun 02 15:59:27.595576 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:27.595457    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f7c6e309-ff75-42a0-8c15-57d82cced0cb-certificates\") pod \"keda-metrics-apiserver-7c9f485588-djgq7\" (UID: \"f7c6e309-ff75-42a0-8c15-57d82cced0cb\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-djgq7"
Jun 02 15:59:27.595777 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:59:27.595624    2579 secret.go:281] references non-existent secret key: tls.crt
Jun 02 15:59:27.595777 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:59:27.595649    2579 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Jun 02 15:59:27.595777 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:59:27.595671    2579 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-djgq7: references non-existent secret key: tls.crt
Jun 02 15:59:27.595777 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:59:27.595739    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7c6e309-ff75-42a0-8c15-57d82cced0cb-certificates podName:f7c6e309-ff75-42a0-8c15-57d82cced0cb nodeName:}" failed. No retries permitted until 2026-06-02 15:59:28.595720423 +0000 UTC m=+777.121628831 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/f7c6e309-ff75-42a0-8c15-57d82cced0cb-certificates") pod "keda-metrics-apiserver-7c9f485588-djgq7" (UID: "f7c6e309-ff75-42a0-8c15-57d82cced0cb") : references non-existent secret key: tls.crt
Jun 02 15:59:27.898415 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:27.898302    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2e2b5f84-d5d2-496c-8e3c-79ea79f771a9-certificates\") pod \"keda-admission-cf49989db-z9hbj\" (UID: \"2e2b5f84-d5d2-496c-8e3c-79ea79f771a9\") " pod="openshift-keda/keda-admission-cf49989db-z9hbj"
Jun 02 15:59:27.900904 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:27.900882    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2e2b5f84-d5d2-496c-8e3c-79ea79f771a9-certificates\") pod \"keda-admission-cf49989db-z9hbj\" (UID: \"2e2b5f84-d5d2-496c-8e3c-79ea79f771a9\") " pod="openshift-keda/keda-admission-cf49989db-z9hbj"
Jun 02 15:59:28.166997 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:28.166901    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-cf49989db-z9hbj"
Jun 02 15:59:28.295528 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:28.295504    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-cf49989db-z9hbj"]
Jun 02 15:59:28.298014 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:59:28.297982    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e2b5f84_d5d2_496c_8e3c_79ea79f771a9.slice/crio-b3b8c6da8dcdeedb7692e44f85c6f49e949ef1521fa6d456d90feb009b9e8a84 WatchSource:0}: Error finding container b3b8c6da8dcdeedb7692e44f85c6f49e949ef1521fa6d456d90feb009b9e8a84: Status 404 returned error can't find the container with id b3b8c6da8dcdeedb7692e44f85c6f49e949ef1521fa6d456d90feb009b9e8a84
Jun 02 15:59:28.299467 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:28.299448    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 15:59:28.604810 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:28.604768    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f7c6e309-ff75-42a0-8c15-57d82cced0cb-certificates\") pod \"keda-metrics-apiserver-7c9f485588-djgq7\" (UID: \"f7c6e309-ff75-42a0-8c15-57d82cced0cb\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-djgq7"
Jun 02 15:59:28.604998 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:59:28.604916    2579 secret.go:281] references non-existent secret key: tls.crt
Jun 02 15:59:28.604998 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:59:28.604937    2579 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Jun 02 15:59:28.604998 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:59:28.604957    2579 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-djgq7: references non-existent secret key: tls.crt
Jun 02 15:59:28.605107 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:59:28.605023    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7c6e309-ff75-42a0-8c15-57d82cced0cb-certificates podName:f7c6e309-ff75-42a0-8c15-57d82cced0cb nodeName:}" failed. No retries permitted until 2026-06-02 15:59:30.605006945 +0000 UTC m=+779.130915354 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/f7c6e309-ff75-42a0-8c15-57d82cced0cb-certificates") pod "keda-metrics-apiserver-7c9f485588-djgq7" (UID: "f7c6e309-ff75-42a0-8c15-57d82cced0cb") : references non-existent secret key: tls.crt
Jun 02 15:59:29.238542 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:29.238505    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-cf49989db-z9hbj" event={"ID":"2e2b5f84-d5d2-496c-8e3c-79ea79f771a9","Type":"ContainerStarted","Data":"b3b8c6da8dcdeedb7692e44f85c6f49e949ef1521fa6d456d90feb009b9e8a84"}
Jun 02 15:59:30.621447 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:30.621381    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f7c6e309-ff75-42a0-8c15-57d82cced0cb-certificates\") pod \"keda-metrics-apiserver-7c9f485588-djgq7\" (UID: \"f7c6e309-ff75-42a0-8c15-57d82cced0cb\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-djgq7"
Jun 02 15:59:30.621923 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:59:30.621563    2579 secret.go:281] references non-existent secret key: tls.crt
Jun 02 15:59:30.621923 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:59:30.621581    2579 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Jun 02 15:59:30.621923 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:59:30.621605    2579 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-djgq7: references non-existent secret key: tls.crt
Jun 02 15:59:30.621923 ip-10-0-129-156 kubenswrapper[2579]: E0602 15:59:30.621663    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7c6e309-ff75-42a0-8c15-57d82cced0cb-certificates podName:f7c6e309-ff75-42a0-8c15-57d82cced0cb nodeName:}" failed. No retries permitted until 2026-06-02 15:59:34.621645085 +0000 UTC m=+783.147553512 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/f7c6e309-ff75-42a0-8c15-57d82cced0cb-certificates") pod "keda-metrics-apiserver-7c9f485588-djgq7" (UID: "f7c6e309-ff75-42a0-8c15-57d82cced0cb") : references non-existent secret key: tls.crt
Jun 02 15:59:34.252579 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:34.252539    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-cf49989db-z9hbj" event={"ID":"2e2b5f84-d5d2-496c-8e3c-79ea79f771a9","Type":"ContainerStarted","Data":"74c8026be7f54a015c654041b43972555ae9c73a79639f5abe9af42b405274b3"}
Jun 02 15:59:34.253023 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:34.252745    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-cf49989db-z9hbj"
Jun 02 15:59:34.285391 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:34.285339    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-admission-cf49989db-z9hbj" podStartSLOduration=2.123844139 podStartE2EDuration="7.285324593s" podCreationTimestamp="2026-06-02 15:59:27 +0000 UTC" firstStartedPulling="2026-06-02 15:59:28.299674738 +0000 UTC m=+776.825583149" lastFinishedPulling="2026-06-02 15:59:33.461155194 +0000 UTC m=+781.987063603" observedRunningTime="2026-06-02 15:59:34.285057737 +0000 UTC m=+782.810966171" watchObservedRunningTime="2026-06-02 15:59:34.285324593 +0000 UTC m=+782.811233067"
Jun 02 15:59:34.654380 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:34.654297    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f7c6e309-ff75-42a0-8c15-57d82cced0cb-certificates\") pod \"keda-metrics-apiserver-7c9f485588-djgq7\" (UID: \"f7c6e309-ff75-42a0-8c15-57d82cced0cb\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-djgq7"
Jun 02 15:59:34.656923 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:34.656904    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f7c6e309-ff75-42a0-8c15-57d82cced0cb-certificates\") pod \"keda-metrics-apiserver-7c9f485588-djgq7\" (UID: \"f7c6e309-ff75-42a0-8c15-57d82cced0cb\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-djgq7"
Jun 02 15:59:34.720809 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:34.720771    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-djgq7"
Jun 02 15:59:34.903039 ip-10-0-129-156 kubenswrapper[2579]: W0602 15:59:34.903006    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7c6e309_ff75_42a0_8c15_57d82cced0cb.slice/crio-05cc1089bfced5a5b0293f34022a9997e02243024354f20c28531313c2d5cd55 WatchSource:0}: Error finding container 05cc1089bfced5a5b0293f34022a9997e02243024354f20c28531313c2d5cd55: Status 404 returned error can't find the container with id 05cc1089bfced5a5b0293f34022a9997e02243024354f20c28531313c2d5cd55
Jun 02 15:59:34.905448 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:34.905421    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-djgq7"]
Jun 02 15:59:35.255859 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:35.255821    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-djgq7" event={"ID":"f7c6e309-ff75-42a0-8c15-57d82cced0cb","Type":"ContainerStarted","Data":"05cc1089bfced5a5b0293f34022a9997e02243024354f20c28531313c2d5cd55"}
Jun 02 15:59:42.276198 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:42.276157    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-djgq7" event={"ID":"f7c6e309-ff75-42a0-8c15-57d82cced0cb","Type":"ContainerStarted","Data":"129de58cc9a10e8bd53f06c5b24b82f201cd42a1cbc1ad1d64eae1ce0057f872"}
Jun 02 15:59:42.276616 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:42.276261    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-djgq7"
Jun 02 15:59:42.295723 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:42.295676    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-djgq7" podStartSLOduration=9.450288122 podStartE2EDuration="16.295659577s" podCreationTimestamp="2026-06-02 15:59:26 +0000 UTC" firstStartedPulling="2026-06-02 15:59:34.904560597 +0000 UTC m=+783.430469026" lastFinishedPulling="2026-06-02 15:59:41.749932059 +0000 UTC m=+790.275840481" observedRunningTime="2026-06-02 15:59:42.294715108 +0000 UTC m=+790.820623537" watchObservedRunningTime="2026-06-02 15:59:42.295659577 +0000 UTC m=+790.821568007"
Jun 02 15:59:53.285465 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:53.285382    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-djgq7"
Jun 02 15:59:55.258681 ip-10-0-129-156 kubenswrapper[2579]: I0602 15:59:55.258643    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-cf49989db-z9hbj"
Jun 02 16:00:35.885945 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:00:35.885904    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-79f457656b-r48bb"]
Jun 02 16:00:35.889119 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:00:35.889102    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-79f457656b-r48bb"
Jun 02 16:00:35.892639 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:00:35.892618    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jun 02 16:00:35.892766 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:00:35.892686    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-lqfnk\""
Jun 02 16:00:35.893530 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:00:35.893508    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jun 02 16:00:35.893530 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:00:35.893520    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jun 02 16:00:35.903543 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:00:35.903518    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-79f457656b-r48bb"]
Jun 02 16:00:35.924708 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:00:35.924675    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f3c112f5-381e-4196-85fc-fbeda63c6b43-data\") pod \"seaweedfs-79f457656b-r48bb\" (UID: \"f3c112f5-381e-4196-85fc-fbeda63c6b43\") " pod="kserve/seaweedfs-79f457656b-r48bb"
Jun 02 16:00:35.924849 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:00:35.924725    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkx28\" (UniqueName: \"kubernetes.io/projected/f3c112f5-381e-4196-85fc-fbeda63c6b43-kube-api-access-fkx28\") pod \"seaweedfs-79f457656b-r48bb\" (UID: \"f3c112f5-381e-4196-85fc-fbeda63c6b43\") " pod="kserve/seaweedfs-79f457656b-r48bb"
Jun 02 16:00:36.025575 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:00:36.025535    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f3c112f5-381e-4196-85fc-fbeda63c6b43-data\") pod \"seaweedfs-79f457656b-r48bb\" (UID: \"f3c112f5-381e-4196-85fc-fbeda63c6b43\") " pod="kserve/seaweedfs-79f457656b-r48bb"
Jun 02 16:00:36.025575 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:00:36.025583    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fkx28\" (UniqueName: \"kubernetes.io/projected/f3c112f5-381e-4196-85fc-fbeda63c6b43-kube-api-access-fkx28\") pod \"seaweedfs-79f457656b-r48bb\" (UID: \"f3c112f5-381e-4196-85fc-fbeda63c6b43\") " pod="kserve/seaweedfs-79f457656b-r48bb"
Jun 02 16:00:36.025946 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:00:36.025923    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f3c112f5-381e-4196-85fc-fbeda63c6b43-data\") pod \"seaweedfs-79f457656b-r48bb\" (UID: \"f3c112f5-381e-4196-85fc-fbeda63c6b43\") " pod="kserve/seaweedfs-79f457656b-r48bb"
Jun 02 16:00:36.035363 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:00:36.035328    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkx28\" (UniqueName: \"kubernetes.io/projected/f3c112f5-381e-4196-85fc-fbeda63c6b43-kube-api-access-fkx28\") pod \"seaweedfs-79f457656b-r48bb\" (UID: \"f3c112f5-381e-4196-85fc-fbeda63c6b43\") " pod="kserve/seaweedfs-79f457656b-r48bb"
Jun 02 16:00:36.197976 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:00:36.197902    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-79f457656b-r48bb"
Jun 02 16:00:36.331026 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:00:36.330991    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-79f457656b-r48bb"]
Jun 02 16:00:36.334530 ip-10-0-129-156 kubenswrapper[2579]: W0602 16:00:36.334496    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3c112f5_381e_4196_85fc_fbeda63c6b43.slice/crio-19188b5dc110027bf5445b1979a708703034e84daf9c852a71a317565e98759e WatchSource:0}: Error finding container 19188b5dc110027bf5445b1979a708703034e84daf9c852a71a317565e98759e: Status 404 returned error can't find the container with id 19188b5dc110027bf5445b1979a708703034e84daf9c852a71a317565e98759e
Jun 02 16:00:36.417350 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:00:36.417322    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-79f457656b-r48bb" event={"ID":"f3c112f5-381e-4196-85fc-fbeda63c6b43","Type":"ContainerStarted","Data":"19188b5dc110027bf5445b1979a708703034e84daf9c852a71a317565e98759e"}
Jun 02 16:00:40.430263 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:00:40.430219    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-79f457656b-r48bb" event={"ID":"f3c112f5-381e-4196-85fc-fbeda63c6b43","Type":"ContainerStarted","Data":"a651866a1e23a0b198ac858fd14f7e0d0840f4495be425902e48fcd545e38d27"}
Jun 02 16:00:40.430702 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:00:40.430396    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-79f457656b-r48bb"
Jun 02 16:00:40.450049 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:00:40.450001    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/seaweedfs-79f457656b-r48bb" podStartSLOduration=2.266592547 podStartE2EDuration="5.449985214s" podCreationTimestamp="2026-06-02 16:00:35 +0000 UTC" firstStartedPulling="2026-06-02 16:00:36.336086837 +0000 UTC m=+844.861995246" lastFinishedPulling="2026-06-02 16:00:39.519479505 +0000 UTC m=+848.045387913" observedRunningTime="2026-06-02 16:00:40.449073382 +0000 UTC m=+848.974981813" watchObservedRunningTime="2026-06-02 16:00:40.449985214 +0000 UTC m=+848.975893644"
Jun 02 16:00:46.437312 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:00:46.437280    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-79f457656b-r48bb"
Jun 02 16:01:31.969459 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:01:31.969360    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lqx9k_6127e33c-aa77-437e-b443-e956037afedb/ovn-acl-logging/0.log"
Jun 02 16:01:31.970482 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:01:31.970459    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lqx9k_6127e33c-aa77-437e-b443-e956037afedb/ovn-acl-logging/0.log"
Jun 02 16:01:46.265091 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:01:46.265045    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-696fc77849-hj7x7"]
Jun 02 16:01:46.268285 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:01:46.268266    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-hj7x7"
Jun 02 16:01:46.271591 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:01:46.271566    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-rsdvw\""
Jun 02 16:01:46.271591 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:01:46.271570    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jun 02 16:01:46.276318 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:01:46.276294    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-hj7x7"]
Jun 02 16:01:46.362432 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:01:46.362364    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwthr\" (UniqueName: \"kubernetes.io/projected/4c713f6d-8f54-42fa-83c4-0fba73803c43-kube-api-access-nwthr\") pod \"odh-model-controller-696fc77849-hj7x7\" (UID: \"4c713f6d-8f54-42fa-83c4-0fba73803c43\") " pod="kserve/odh-model-controller-696fc77849-hj7x7"
Jun 02 16:01:46.362624 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:01:46.362442    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4c713f6d-8f54-42fa-83c4-0fba73803c43-cert\") pod \"odh-model-controller-696fc77849-hj7x7\" (UID: \"4c713f6d-8f54-42fa-83c4-0fba73803c43\") " pod="kserve/odh-model-controller-696fc77849-hj7x7"
Jun 02 16:01:46.462841 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:01:46.462804    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4c713f6d-8f54-42fa-83c4-0fba73803c43-cert\") pod \"odh-model-controller-696fc77849-hj7x7\" (UID: \"4c713f6d-8f54-42fa-83c4-0fba73803c43\") " pod="kserve/odh-model-controller-696fc77849-hj7x7"
Jun 02 16:01:46.462997 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:01:46.462875    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nwthr\" (UniqueName: \"kubernetes.io/projected/4c713f6d-8f54-42fa-83c4-0fba73803c43-kube-api-access-nwthr\") pod \"odh-model-controller-696fc77849-hj7x7\" (UID: \"4c713f6d-8f54-42fa-83c4-0fba73803c43\") " pod="kserve/odh-model-controller-696fc77849-hj7x7"
Jun 02 16:01:46.465456 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:01:46.465426    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4c713f6d-8f54-42fa-83c4-0fba73803c43-cert\") pod \"odh-model-controller-696fc77849-hj7x7\" (UID: \"4c713f6d-8f54-42fa-83c4-0fba73803c43\") " pod="kserve/odh-model-controller-696fc77849-hj7x7"
Jun 02 16:01:46.478835 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:01:46.478811    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwthr\" (UniqueName: \"kubernetes.io/projected/4c713f6d-8f54-42fa-83c4-0fba73803c43-kube-api-access-nwthr\") pod \"odh-model-controller-696fc77849-hj7x7\" (UID: \"4c713f6d-8f54-42fa-83c4-0fba73803c43\") " pod="kserve/odh-model-controller-696fc77849-hj7x7"
Jun 02 16:01:46.579251 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:01:46.579159    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-hj7x7"
Jun 02 16:01:46.731271 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:01:46.731238    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-hj7x7"]
Jun 02 16:01:46.735109 ip-10-0-129-156 kubenswrapper[2579]: W0602 16:01:46.735079    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c713f6d_8f54_42fa_83c4_0fba73803c43.slice/crio-ff234b9c455a9096b3cbc1b4684cf2bd78b2b8ed84ad0a15b9c5ad09932f91e8 WatchSource:0}: Error finding container ff234b9c455a9096b3cbc1b4684cf2bd78b2b8ed84ad0a15b9c5ad09932f91e8: Status 404 returned error can't find the container with id ff234b9c455a9096b3cbc1b4684cf2bd78b2b8ed84ad0a15b9c5ad09932f91e8
Jun 02 16:01:47.607854 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:01:47.607818    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-hj7x7" event={"ID":"4c713f6d-8f54-42fa-83c4-0fba73803c43","Type":"ContainerStarted","Data":"ff234b9c455a9096b3cbc1b4684cf2bd78b2b8ed84ad0a15b9c5ad09932f91e8"}
Jun 02 16:01:49.614710 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:01:49.614676    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-hj7x7" event={"ID":"4c713f6d-8f54-42fa-83c4-0fba73803c43","Type":"ContainerStarted","Data":"8a2801e468131d5dd8be30d4a3b57b26a7781d5604a1f7d7f0133f254db4237c"}
Jun 02 16:01:49.615096 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:01:49.614826    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-696fc77849-hj7x7"
Jun 02 16:01:49.632371 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:01:49.632321    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/odh-model-controller-696fc77849-hj7x7" podStartSLOduration=0.956415957 podStartE2EDuration="3.632306217s" podCreationTimestamp="2026-06-02 16:01:46 +0000 UTC" firstStartedPulling="2026-06-02 16:01:46.7367391 +0000 UTC m=+915.262647512" lastFinishedPulling="2026-06-02 16:01:49.41262936 +0000 UTC m=+917.938537772" observedRunningTime="2026-06-02 16:01:49.631197313 +0000 UTC m=+918.157105745" watchObservedRunningTime="2026-06-02 16:01:49.632306217 +0000 UTC m=+918.158214648"
Jun 02 16:02:00.620834 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:02:00.620806    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-696fc77849-hj7x7"
Jun 02 16:02:13.356428 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:02:13.356365    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-ddd4dbfd-pzzml"]
Jun 02 16:02:13.359479 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:02:13.359447    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-pzzml"
Jun 02 16:02:13.362287 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:02:13.362248    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jun 02 16:02:13.363574 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:02:13.363545    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-ddd4dbfd-pzzml"]
Jun 02 16:02:13.474157 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:02:13.474120    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c2ec5015-1243-46e1-9bf9-7fa333b7b432-data\") pod \"seaweedfs-tls-custom-ddd4dbfd-pzzml\" (UID: \"c2ec5015-1243-46e1-9bf9-7fa333b7b432\") " pod="kserve/seaweedfs-tls-custom-ddd4dbfd-pzzml"
Jun 02 16:02:13.474157 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:02:13.474160    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzqfq\" (UniqueName: \"kubernetes.io/projected/c2ec5015-1243-46e1-9bf9-7fa333b7b432-kube-api-access-tzqfq\") pod \"seaweedfs-tls-custom-ddd4dbfd-pzzml\" (UID: \"c2ec5015-1243-46e1-9bf9-7fa333b7b432\") " pod="kserve/seaweedfs-tls-custom-ddd4dbfd-pzzml"
Jun 02 16:02:13.575264 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:02:13.575225    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c2ec5015-1243-46e1-9bf9-7fa333b7b432-data\") pod \"seaweedfs-tls-custom-ddd4dbfd-pzzml\" (UID: \"c2ec5015-1243-46e1-9bf9-7fa333b7b432\") " pod="kserve/seaweedfs-tls-custom-ddd4dbfd-pzzml"
Jun 02 16:02:13.575264 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:02:13.575265    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tzqfq\" (UniqueName: \"kubernetes.io/projected/c2ec5015-1243-46e1-9bf9-7fa333b7b432-kube-api-access-tzqfq\") pod \"seaweedfs-tls-custom-ddd4dbfd-pzzml\" (UID: \"c2ec5015-1243-46e1-9bf9-7fa333b7b432\") " pod="kserve/seaweedfs-tls-custom-ddd4dbfd-pzzml"
Jun 02 16:02:13.575737 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:02:13.575712    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c2ec5015-1243-46e1-9bf9-7fa333b7b432-data\") pod \"seaweedfs-tls-custom-ddd4dbfd-pzzml\" (UID: \"c2ec5015-1243-46e1-9bf9-7fa333b7b432\") " pod="kserve/seaweedfs-tls-custom-ddd4dbfd-pzzml"
Jun 02 16:02:13.585206 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:02:13.585181    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzqfq\" (UniqueName: \"kubernetes.io/projected/c2ec5015-1243-46e1-9bf9-7fa333b7b432-kube-api-access-tzqfq\") pod \"seaweedfs-tls-custom-ddd4dbfd-pzzml\" (UID: \"c2ec5015-1243-46e1-9bf9-7fa333b7b432\") " pod="kserve/seaweedfs-tls-custom-ddd4dbfd-pzzml"
Jun 02 16:02:13.669488 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:02:13.669345    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-pzzml"
Jun 02 16:02:13.799799 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:02:13.799685    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-ddd4dbfd-pzzml"]
Jun 02 16:02:13.802802 ip-10-0-129-156 kubenswrapper[2579]: W0602 16:02:13.802772    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2ec5015_1243_46e1_9bf9_7fa333b7b432.slice/crio-5fae9b30a4a0017835939dd068f3163c26d4652555cbae97e970c95a8e9e8f49 WatchSource:0}: Error finding container 5fae9b30a4a0017835939dd068f3163c26d4652555cbae97e970c95a8e9e8f49: Status 404 returned error can't find the container with id 5fae9b30a4a0017835939dd068f3163c26d4652555cbae97e970c95a8e9e8f49
Jun 02 16:02:14.685299 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:02:14.685259    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-pzzml" event={"ID":"c2ec5015-1243-46e1-9bf9-7fa333b7b432","Type":"ContainerStarted","Data":"c355a5927e782057b16a62cbf9e0f01b4fe58203ac4c0bf08dadc29e578bf5e0"}
Jun 02 16:02:14.685299 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:02:14.685304    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-pzzml" event={"ID":"c2ec5015-1243-46e1-9bf9-7fa333b7b432","Type":"ContainerStarted","Data":"5fae9b30a4a0017835939dd068f3163c26d4652555cbae97e970c95a8e9e8f49"}
Jun 02 16:02:14.704460 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:02:14.704388    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-pzzml" podStartSLOduration=1.436344636 podStartE2EDuration="1.704371394s" podCreationTimestamp="2026-06-02 16:02:13 +0000 UTC" firstStartedPulling="2026-06-02 16:02:13.804297233 +0000 UTC m=+942.330205641" lastFinishedPulling="2026-06-02 16:02:14.072323989 +0000 UTC m=+942.598232399" observedRunningTime="2026-06-02 16:02:14.703374292 +0000 UTC m=+943.229282726" watchObservedRunningTime="2026-06-02 16:02:14.704371394 +0000 UTC m=+943.230279824"
Jun 02 16:02:15.771627 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:02:15.771587    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-ddd4dbfd-pzzml"]
Jun 02 16:02:16.691125 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:02:16.691081    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-pzzml" podUID="c2ec5015-1243-46e1-9bf9-7fa333b7b432" containerName="seaweedfs-tls-custom" containerID="cri-o://c355a5927e782057b16a62cbf9e0f01b4fe58203ac4c0bf08dadc29e578bf5e0" gracePeriod=30
Jun 02 16:02:17.937522 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:02:17.937499    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-pzzml"
Jun 02 16:02:18.010252 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:02:18.010212    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c2ec5015-1243-46e1-9bf9-7fa333b7b432-data\") pod \"c2ec5015-1243-46e1-9bf9-7fa333b7b432\" (UID: \"c2ec5015-1243-46e1-9bf9-7fa333b7b432\") "
Jun 02 16:02:18.010463 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:02:18.010301    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzqfq\" (UniqueName: \"kubernetes.io/projected/c2ec5015-1243-46e1-9bf9-7fa333b7b432-kube-api-access-tzqfq\") pod \"c2ec5015-1243-46e1-9bf9-7fa333b7b432\" (UID: \"c2ec5015-1243-46e1-9bf9-7fa333b7b432\") "
Jun 02 16:02:18.011453 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:02:18.011426    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2ec5015-1243-46e1-9bf9-7fa333b7b432-data" (OuterVolumeSpecName: "data") pod "c2ec5015-1243-46e1-9bf9-7fa333b7b432" (UID: "c2ec5015-1243-46e1-9bf9-7fa333b7b432"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:02:18.012615 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:02:18.012592    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2ec5015-1243-46e1-9bf9-7fa333b7b432-kube-api-access-tzqfq" (OuterVolumeSpecName: "kube-api-access-tzqfq") pod "c2ec5015-1243-46e1-9bf9-7fa333b7b432" (UID: "c2ec5015-1243-46e1-9bf9-7fa333b7b432"). InnerVolumeSpecName "kube-api-access-tzqfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:02:18.111223 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:02:18.111185    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzqfq\" (UniqueName: \"kubernetes.io/projected/c2ec5015-1243-46e1-9bf9-7fa333b7b432-kube-api-access-tzqfq\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:02:18.111223 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:02:18.111216    2579 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c2ec5015-1243-46e1-9bf9-7fa333b7b432-data\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:02:18.697844 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:02:18.697807    2579 generic.go:358] "Generic (PLEG): container finished" podID="c2ec5015-1243-46e1-9bf9-7fa333b7b432" containerID="c355a5927e782057b16a62cbf9e0f01b4fe58203ac4c0bf08dadc29e578bf5e0" exitCode=0
Jun 02 16:02:18.698114 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:02:18.697865    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-pzzml"
Jun 02 16:02:18.698114 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:02:18.697890    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-pzzml" event={"ID":"c2ec5015-1243-46e1-9bf9-7fa333b7b432","Type":"ContainerDied","Data":"c355a5927e782057b16a62cbf9e0f01b4fe58203ac4c0bf08dadc29e578bf5e0"}
Jun 02 16:02:18.698114 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:02:18.697927    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-pzzml" event={"ID":"c2ec5015-1243-46e1-9bf9-7fa333b7b432","Type":"ContainerDied","Data":"5fae9b30a4a0017835939dd068f3163c26d4652555cbae97e970c95a8e9e8f49"}
Jun 02 16:02:18.698114 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:02:18.697943    2579 scope.go:117] "RemoveContainer" containerID="c355a5927e782057b16a62cbf9e0f01b4fe58203ac4c0bf08dadc29e578bf5e0"
Jun 02 16:02:18.707627 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:02:18.707606    2579 scope.go:117] "RemoveContainer" containerID="c355a5927e782057b16a62cbf9e0f01b4fe58203ac4c0bf08dadc29e578bf5e0"
Jun 02 16:02:18.707938 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:02:18.707917    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c355a5927e782057b16a62cbf9e0f01b4fe58203ac4c0bf08dadc29e578bf5e0\": container with ID starting with c355a5927e782057b16a62cbf9e0f01b4fe58203ac4c0bf08dadc29e578bf5e0 not found: ID does not exist" containerID="c355a5927e782057b16a62cbf9e0f01b4fe58203ac4c0bf08dadc29e578bf5e0"
Jun 02 16:02:18.707997 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:02:18.707948    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c355a5927e782057b16a62cbf9e0f01b4fe58203ac4c0bf08dadc29e578bf5e0"} err="failed to get container status \"c355a5927e782057b16a62cbf9e0f01b4fe58203ac4c0bf08dadc29e578bf5e0\": rpc error: code = NotFound desc = could not find container \"c355a5927e782057b16a62cbf9e0f01b4fe58203ac4c0bf08dadc29e578bf5e0\": container with ID starting with c355a5927e782057b16a62cbf9e0f01b4fe58203ac4c0bf08dadc29e578bf5e0 not found: ID does not exist"
Jun 02 16:02:18.722917 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:02:18.722886    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-ddd4dbfd-pzzml"]
Jun 02 16:02:18.730055 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:02:18.730032    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-ddd4dbfd-pzzml"]
Jun 02 16:02:20.063670 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:02:20.063619    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2ec5015-1243-46e1-9bf9-7fa333b7b432" path="/var/lib/kubelet/pods/c2ec5015-1243-46e1-9bf9-7fa333b7b432/volumes"
Jun 02 16:06:31.989355 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:06:31.989326    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lqx9k_6127e33c-aa77-437e-b443-e956037afedb/ovn-acl-logging/0.log"
Jun 02 16:06:31.990773 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:06:31.990746    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lqx9k_6127e33c-aa77-437e-b443-e956037afedb/ovn-acl-logging/0.log"
Jun 02 16:11:32.006481 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:11:32.006453    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lqx9k_6127e33c-aa77-437e-b443-e956037afedb/ovn-acl-logging/0.log"
Jun 02 16:11:32.008060 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:11:32.008042    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lqx9k_6127e33c-aa77-437e-b443-e956037afedb/ovn-acl-logging/0.log"
Jun 02 16:16:32.024520 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:16:32.024483    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lqx9k_6127e33c-aa77-437e-b443-e956037afedb/ovn-acl-logging/0.log"
Jun 02 16:16:32.026945 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:16:32.026925    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lqx9k_6127e33c-aa77-437e-b443-e956037afedb/ovn-acl-logging/0.log"
Jun 02 16:17:00.144841 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:17:00.144806    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk"]
Jun 02 16:17:00.145321 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:17:00.145060    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2ec5015-1243-46e1-9bf9-7fa333b7b432" containerName="seaweedfs-tls-custom"
Jun 02 16:17:00.145321 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:17:00.145072    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2ec5015-1243-46e1-9bf9-7fa333b7b432" containerName="seaweedfs-tls-custom"
Jun 02 16:17:00.145321 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:17:00.145122    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="c2ec5015-1243-46e1-9bf9-7fa333b7b432" containerName="seaweedfs-tls-custom"
Jun 02 16:17:00.149242 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:17:00.149209    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk"
Jun 02 16:17:00.151878 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:17:00.151853    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Jun 02 16:17:00.152040 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:17:00.151858    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ztrcr\""
Jun 02 16:17:00.152040 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:17:00.151899    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Jun 02 16:17:00.152665 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:17:00.152649    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jun 02 16:17:00.152720 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:17:00.152653    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jun 02 16:17:00.159001 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:17:00.158976    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk"]
Jun 02 16:17:00.237510 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:17:00.237466    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7-proxy-tls\") pod \"isvc-paddle-predictor-6b8b7cfb4b-nprsk\" (UID: \"2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk"
Jun 02 16:17:00.237700 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:17:00.237528    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96xj2\" (UniqueName: \"kubernetes.io/projected/2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7-kube-api-access-96xj2\") pod \"isvc-paddle-predictor-6b8b7cfb4b-nprsk\" (UID: \"2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk"
Jun 02 16:17:00.237700 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:17:00.237592    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7-kserve-provision-location\") pod \"isvc-paddle-predictor-6b8b7cfb4b-nprsk\" (UID: \"2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk"
Jun 02 16:17:00.237700 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:17:00.237640    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7-isvc-paddle-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6b8b7cfb4b-nprsk\" (UID: \"2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk"
Jun 02 16:17:00.338198 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:17:00.338160    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7-isvc-paddle-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6b8b7cfb4b-nprsk\" (UID: \"2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk"
Jun 02 16:17:00.338392 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:17:00.338213    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7-proxy-tls\") pod \"isvc-paddle-predictor-6b8b7cfb4b-nprsk\" (UID: \"2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk"
Jun 02 16:17:00.338392 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:17:00.338245    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-96xj2\" (UniqueName: \"kubernetes.io/projected/2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7-kube-api-access-96xj2\") pod \"isvc-paddle-predictor-6b8b7cfb4b-nprsk\" (UID: \"2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk"
Jun 02 16:17:00.338392 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:17:00.338267    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7-kserve-provision-location\") pod \"isvc-paddle-predictor-6b8b7cfb4b-nprsk\" (UID: \"2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk"
Jun 02 16:17:00.338605 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:17:00.338417    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-predictor-serving-cert: secret "isvc-paddle-predictor-serving-cert" not found
Jun 02 16:17:00.338605 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:17:00.338515    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7-proxy-tls podName:2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7 nodeName:}" failed. No retries permitted until 2026-06-02 16:17:00.838491231 +0000 UTC m=+1829.364399657 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7-proxy-tls") pod "isvc-paddle-predictor-6b8b7cfb4b-nprsk" (UID: "2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7") : secret "isvc-paddle-predictor-serving-cert" not found
Jun 02 16:17:00.338718 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:17:00.338664    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7-kserve-provision-location\") pod \"isvc-paddle-predictor-6b8b7cfb4b-nprsk\" (UID: \"2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk"
Jun 02 16:17:00.338881 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:17:00.338859    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7-isvc-paddle-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6b8b7cfb4b-nprsk\" (UID: \"2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk"
Jun 02 16:17:00.346771 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:17:00.346743    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96xj2\" (UniqueName: \"kubernetes.io/projected/2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7-kube-api-access-96xj2\") pod \"isvc-paddle-predictor-6b8b7cfb4b-nprsk\" (UID: \"2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk"
Jun 02 16:17:00.843204 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:17:00.843145    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7-proxy-tls\") pod \"isvc-paddle-predictor-6b8b7cfb4b-nprsk\" (UID: \"2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk"
Jun 02 16:17:00.845748 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:17:00.845716    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7-proxy-tls\") pod \"isvc-paddle-predictor-6b8b7cfb4b-nprsk\" (UID: \"2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk"
Jun 02 16:17:01.060540 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:17:01.060501    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk"
Jun 02 16:17:01.193755 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:17:01.193731    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk"]
Jun 02 16:17:01.196168 ip-10-0-129-156 kubenswrapper[2579]: W0602 16:17:01.196141    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a2ddf3d_ccb8_4311_8705_cf1505dfd5f7.slice/crio-6b32b611c22edf7674e7c831171763365855bed57ae1d3ffaf25dd9af902f22c WatchSource:0}: Error finding container 6b32b611c22edf7674e7c831171763365855bed57ae1d3ffaf25dd9af902f22c: Status 404 returned error can't find the container with id 6b32b611c22edf7674e7c831171763365855bed57ae1d3ffaf25dd9af902f22c
Jun 02 16:17:01.198345 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:17:01.198323    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 16:17:02.104734 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:17:02.104692    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk" event={"ID":"2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7","Type":"ContainerStarted","Data":"6b32b611c22edf7674e7c831171763365855bed57ae1d3ffaf25dd9af902f22c"}
Jun 02 16:17:06.117462 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:17:06.117427    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk" event={"ID":"2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7","Type":"ContainerStarted","Data":"2eb6378454bc0f57630cd3bc9389319412982c3bb62e91f5db4daa56e91f7636"}
Jun 02 16:17:10.130909 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:17:10.130874    2579 generic.go:358] "Generic (PLEG): container finished" podID="2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7" containerID="2eb6378454bc0f57630cd3bc9389319412982c3bb62e91f5db4daa56e91f7636" exitCode=0
Jun 02 16:17:10.131294 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:17:10.130931    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk" event={"ID":"2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7","Type":"ContainerDied","Data":"2eb6378454bc0f57630cd3bc9389319412982c3bb62e91f5db4daa56e91f7636"}
Jun 02 16:17:25.178779 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:17:25.178737    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk" event={"ID":"2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7","Type":"ContainerStarted","Data":"2b962d0a37a45296254e5a59df29e2406d46e5ca35e24f851378d62f01c827f1"}
Jun 02 16:17:28.190174 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:17:28.190138    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk" event={"ID":"2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7","Type":"ContainerStarted","Data":"5f65331bce2c73902521b20aaaa2c5319f6ae2afcaac5425c941e138159c5dab"}
Jun 02 16:17:28.190648 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:17:28.190444    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk"
Jun 02 16:17:28.210323 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:17:28.210258    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk" podStartSLOduration=2.130710224 podStartE2EDuration="28.210240293s" podCreationTimestamp="2026-06-02 16:17:00 +0000 UTC" firstStartedPulling="2026-06-02 16:17:01.198521195 +0000 UTC m=+1829.724429625" lastFinishedPulling="2026-06-02 16:17:27.278051275 +0000 UTC m=+1855.803959694" observedRunningTime="2026-06-02 16:17:28.20933032 +0000 UTC m=+1856.735238750" watchObservedRunningTime="2026-06-02 16:17:28.210240293 +0000 UTC m=+1856.736148725"
Jun 02 16:17:29.192966 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:17:29.192930    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk"
Jun 02 16:17:29.194038 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:17:29.194012    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk" podUID="2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Jun 02 16:17:30.195473 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:17:30.195435    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk" podUID="2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Jun 02 16:17:35.200172 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:17:35.200142    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk"
Jun 02 16:17:35.200644 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:17:35.200617    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk" podUID="2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Jun 02 16:17:45.201495 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:17:45.201456    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk" podUID="2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Jun 02 16:17:55.200892 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:17:55.200805    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk" podUID="2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Jun 02 16:18:05.201484 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:18:05.201442    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk" podUID="2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Jun 02 16:18:15.202243 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:18:15.202210    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk"
Jun 02 16:18:21.655851 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:18:21.655808    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk"]
Jun 02 16:18:21.656368 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:18:21.656174    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk" podUID="2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7" containerName="kserve-container" containerID="cri-o://2b962d0a37a45296254e5a59df29e2406d46e5ca35e24f851378d62f01c827f1" gracePeriod=30
Jun 02 16:18:21.656368 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:18:21.656195    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk" podUID="2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7" containerName="kube-rbac-proxy" containerID="cri-o://5f65331bce2c73902521b20aaaa2c5319f6ae2afcaac5425c941e138159c5dab" gracePeriod=30
Jun 02 16:18:22.337902 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:18:22.337863    2579 generic.go:358] "Generic (PLEG): container finished" podID="2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7" containerID="5f65331bce2c73902521b20aaaa2c5319f6ae2afcaac5425c941e138159c5dab" exitCode=2
Jun 02 16:18:22.338104 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:18:22.337914    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk" event={"ID":"2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7","Type":"ContainerDied","Data":"5f65331bce2c73902521b20aaaa2c5319f6ae2afcaac5425c941e138159c5dab"}
Jun 02 16:18:24.606884 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:18:24.606859    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk"
Jun 02 16:18:24.716347 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:18:24.716261    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7-proxy-tls\") pod \"2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7\" (UID: \"2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7\") "
Jun 02 16:18:24.716347 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:18:24.716309    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7-isvc-paddle-kube-rbac-proxy-sar-config\") pod \"2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7\" (UID: \"2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7\") "
Jun 02 16:18:24.716607 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:18:24.716360    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96xj2\" (UniqueName: \"kubernetes.io/projected/2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7-kube-api-access-96xj2\") pod \"2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7\" (UID: \"2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7\") "
Jun 02 16:18:24.716607 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:18:24.716388    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7-kserve-provision-location\") pod \"2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7\" (UID: \"2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7\") "
Jun 02 16:18:24.716779 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:18:24.716736    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7-isvc-paddle-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-paddle-kube-rbac-proxy-sar-config") pod "2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7" (UID: "2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7"). InnerVolumeSpecName "isvc-paddle-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:18:24.718770 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:18:24.718742    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7-kube-api-access-96xj2" (OuterVolumeSpecName: "kube-api-access-96xj2") pod "2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7" (UID: "2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7"). InnerVolumeSpecName "kube-api-access-96xj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:18:24.718880 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:18:24.718792    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7" (UID: "2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:18:24.727931 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:18:24.727895    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7" (UID: "2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:18:24.817859 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:18:24.817812    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7-kserve-provision-location\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:18:24.817859 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:18:24.817849    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7-proxy-tls\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:18:24.817859 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:18:24.817859    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7-isvc-paddle-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:18:24.817859 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:18:24.817869    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96xj2\" (UniqueName: \"kubernetes.io/projected/2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7-kube-api-access-96xj2\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:18:25.348414 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:18:25.348367    2579 generic.go:358] "Generic (PLEG): container finished" podID="2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7" containerID="2b962d0a37a45296254e5a59df29e2406d46e5ca35e24f851378d62f01c827f1" exitCode=0
Jun 02 16:18:25.348593 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:18:25.348440    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk" event={"ID":"2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7","Type":"ContainerDied","Data":"2b962d0a37a45296254e5a59df29e2406d46e5ca35e24f851378d62f01c827f1"}
Jun 02 16:18:25.348593 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:18:25.348468    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk"
Jun 02 16:18:25.348593 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:18:25.348480    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk" event={"ID":"2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7","Type":"ContainerDied","Data":"6b32b611c22edf7674e7c831171763365855bed57ae1d3ffaf25dd9af902f22c"}
Jun 02 16:18:25.348593 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:18:25.348496    2579 scope.go:117] "RemoveContainer" containerID="5f65331bce2c73902521b20aaaa2c5319f6ae2afcaac5425c941e138159c5dab"
Jun 02 16:18:25.356883 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:18:25.356869    2579 scope.go:117] "RemoveContainer" containerID="2b962d0a37a45296254e5a59df29e2406d46e5ca35e24f851378d62f01c827f1"
Jun 02 16:18:25.363926 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:18:25.363910    2579 scope.go:117] "RemoveContainer" containerID="2eb6378454bc0f57630cd3bc9389319412982c3bb62e91f5db4daa56e91f7636"
Jun 02 16:18:25.370805 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:18:25.370774    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk"]
Jun 02 16:18:25.371279 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:18:25.371259    2579 scope.go:117] "RemoveContainer" containerID="5f65331bce2c73902521b20aaaa2c5319f6ae2afcaac5425c941e138159c5dab"
Jun 02 16:18:25.371536 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:18:25.371515    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f65331bce2c73902521b20aaaa2c5319f6ae2afcaac5425c941e138159c5dab\": container with ID starting with 5f65331bce2c73902521b20aaaa2c5319f6ae2afcaac5425c941e138159c5dab not found: ID does not exist" containerID="5f65331bce2c73902521b20aaaa2c5319f6ae2afcaac5425c941e138159c5dab"
Jun 02 16:18:25.371594 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:18:25.371558    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f65331bce2c73902521b20aaaa2c5319f6ae2afcaac5425c941e138159c5dab"} err="failed to get container status \"5f65331bce2c73902521b20aaaa2c5319f6ae2afcaac5425c941e138159c5dab\": rpc error: code = NotFound desc = could not find container \"5f65331bce2c73902521b20aaaa2c5319f6ae2afcaac5425c941e138159c5dab\": container with ID starting with 5f65331bce2c73902521b20aaaa2c5319f6ae2afcaac5425c941e138159c5dab not found: ID does not exist"
Jun 02 16:18:25.371594 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:18:25.371578    2579 scope.go:117] "RemoveContainer" containerID="2b962d0a37a45296254e5a59df29e2406d46e5ca35e24f851378d62f01c827f1"
Jun 02 16:18:25.371807 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:18:25.371785    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b962d0a37a45296254e5a59df29e2406d46e5ca35e24f851378d62f01c827f1\": container with ID starting with 2b962d0a37a45296254e5a59df29e2406d46e5ca35e24f851378d62f01c827f1 not found: ID does not exist" containerID="2b962d0a37a45296254e5a59df29e2406d46e5ca35e24f851378d62f01c827f1"
Jun 02 16:18:25.371847 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:18:25.371811    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b962d0a37a45296254e5a59df29e2406d46e5ca35e24f851378d62f01c827f1"} err="failed to get container status \"2b962d0a37a45296254e5a59df29e2406d46e5ca35e24f851378d62f01c827f1\": rpc error: code = NotFound desc = could not find container \"2b962d0a37a45296254e5a59df29e2406d46e5ca35e24f851378d62f01c827f1\": container with ID starting with 2b962d0a37a45296254e5a59df29e2406d46e5ca35e24f851378d62f01c827f1 not found: ID does not exist"
Jun 02 16:18:25.371847 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:18:25.371825    2579 scope.go:117] "RemoveContainer" containerID="2eb6378454bc0f57630cd3bc9389319412982c3bb62e91f5db4daa56e91f7636"
Jun 02 16:18:25.372019 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:18:25.372001    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2eb6378454bc0f57630cd3bc9389319412982c3bb62e91f5db4daa56e91f7636\": container with ID starting with 2eb6378454bc0f57630cd3bc9389319412982c3bb62e91f5db4daa56e91f7636 not found: ID does not exist" containerID="2eb6378454bc0f57630cd3bc9389319412982c3bb62e91f5db4daa56e91f7636"
Jun 02 16:18:25.372062 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:18:25.372025    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eb6378454bc0f57630cd3bc9389319412982c3bb62e91f5db4daa56e91f7636"} err="failed to get container status \"2eb6378454bc0f57630cd3bc9389319412982c3bb62e91f5db4daa56e91f7636\": rpc error: code = NotFound desc = could not find container \"2eb6378454bc0f57630cd3bc9389319412982c3bb62e91f5db4daa56e91f7636\": container with ID starting with 2eb6378454bc0f57630cd3bc9389319412982c3bb62e91f5db4daa56e91f7636 not found: ID does not exist"
Jun 02 16:18:25.377183 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:18:25.377162    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nprsk"]
Jun 02 16:18:26.068979 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:18:26.068935    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7" path="/var/lib/kubelet/pods/2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7/volumes"
Jun 02 16:19:33.416141 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:33.416096    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9"]
Jun 02 16:19:33.418285 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:33.416372    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7" containerName="kube-rbac-proxy"
Jun 02 16:19:33.418285 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:33.416384    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7" containerName="kube-rbac-proxy"
Jun 02 16:19:33.418285 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:33.416393    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7" containerName="kserve-container"
Jun 02 16:19:33.418285 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:33.416414    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7" containerName="kserve-container"
Jun 02 16:19:33.418285 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:33.416456    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7" containerName="storage-initializer"
Jun 02 16:19:33.418285 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:33.416463    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7" containerName="storage-initializer"
Jun 02 16:19:33.418285 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:33.416505    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7" containerName="kserve-container"
Jun 02 16:19:33.418285 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:33.416519    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="2a2ddf3d-ccb8-4311-8705-cf1505dfd5f7" containerName="kube-rbac-proxy"
Jun 02 16:19:33.419162 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:33.419146    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9"
Jun 02 16:19:33.424772 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:33.424742    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Jun 02 16:19:33.424929 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:33.424747    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Jun 02 16:19:33.424929 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:33.424899    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jun 02 16:19:33.425860 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:33.425805    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jun 02 16:19:33.425964 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:33.425897    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ztrcr\""
Jun 02 16:19:33.439530 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:33.439500    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9"]
Jun 02 16:19:33.537509 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:33.537473    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38a3c9b9-db00-46dc-9524-69e0c0b40366-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9\" (UID: \"38a3c9b9-db00-46dc-9524-69e0c0b40366\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9"
Jun 02 16:19:33.537509 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:33.537509    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgsfp\" (UniqueName: \"kubernetes.io/projected/38a3c9b9-db00-46dc-9524-69e0c0b40366-kube-api-access-wgsfp\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9\" (UID: \"38a3c9b9-db00-46dc-9524-69e0c0b40366\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9"
Jun 02 16:19:33.537745 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:33.537533    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38a3c9b9-db00-46dc-9524-69e0c0b40366-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9\" (UID: \"38a3c9b9-db00-46dc-9524-69e0c0b40366\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9"
Jun 02 16:19:33.537745 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:33.537664    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38a3c9b9-db00-46dc-9524-69e0c0b40366-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9\" (UID: \"38a3c9b9-db00-46dc-9524-69e0c0b40366\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9"
Jun 02 16:19:33.638776 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:33.638730    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38a3c9b9-db00-46dc-9524-69e0c0b40366-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9\" (UID: \"38a3c9b9-db00-46dc-9524-69e0c0b40366\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9"
Jun 02 16:19:33.638981 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:33.638814    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38a3c9b9-db00-46dc-9524-69e0c0b40366-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9\" (UID: \"38a3c9b9-db00-46dc-9524-69e0c0b40366\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9"
Jun 02 16:19:33.638981 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:33.638843    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wgsfp\" (UniqueName: \"kubernetes.io/projected/38a3c9b9-db00-46dc-9524-69e0c0b40366-kube-api-access-wgsfp\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9\" (UID: \"38a3c9b9-db00-46dc-9524-69e0c0b40366\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9"
Jun 02 16:19:33.638981 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:33.638872    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38a3c9b9-db00-46dc-9524-69e0c0b40366-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9\" (UID: \"38a3c9b9-db00-46dc-9524-69e0c0b40366\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9"
Jun 02 16:19:33.638981 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:19:33.638890    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-serving-cert: secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Jun 02 16:19:33.638981 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:19:33.638978    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38a3c9b9-db00-46dc-9524-69e0c0b40366-proxy-tls podName:38a3c9b9-db00-46dc-9524-69e0c0b40366 nodeName:}" failed. No retries permitted until 2026-06-02 16:19:34.138954909 +0000 UTC m=+1982.664863318 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/38a3c9b9-db00-46dc-9524-69e0c0b40366-proxy-tls") pod "isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9" (UID: "38a3c9b9-db00-46dc-9524-69e0c0b40366") : secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Jun 02 16:19:33.639315 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:33.639292    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38a3c9b9-db00-46dc-9524-69e0c0b40366-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9\" (UID: \"38a3c9b9-db00-46dc-9524-69e0c0b40366\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9"
Jun 02 16:19:33.639548 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:33.639526    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38a3c9b9-db00-46dc-9524-69e0c0b40366-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9\" (UID: \"38a3c9b9-db00-46dc-9524-69e0c0b40366\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9"
Jun 02 16:19:33.664157 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:33.664121    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgsfp\" (UniqueName: \"kubernetes.io/projected/38a3c9b9-db00-46dc-9524-69e0c0b40366-kube-api-access-wgsfp\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9\" (UID: \"38a3c9b9-db00-46dc-9524-69e0c0b40366\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9"
Jun 02 16:19:34.143649 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:34.143605    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38a3c9b9-db00-46dc-9524-69e0c0b40366-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9\" (UID: \"38a3c9b9-db00-46dc-9524-69e0c0b40366\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9"
Jun 02 16:19:34.146302 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:34.146277    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38a3c9b9-db00-46dc-9524-69e0c0b40366-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9\" (UID: \"38a3c9b9-db00-46dc-9524-69e0c0b40366\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9"
Jun 02 16:19:34.329465 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:34.329389    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9"
Jun 02 16:19:34.465128 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:34.465086    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9"]
Jun 02 16:19:34.469605 ip-10-0-129-156 kubenswrapper[2579]: W0602 16:19:34.469564    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38a3c9b9_db00_46dc_9524_69e0c0b40366.slice/crio-78469dd05740430c16f5a52058ccd5289720eaa624fd11ae1975a8ca3063710c WatchSource:0}: Error finding container 78469dd05740430c16f5a52058ccd5289720eaa624fd11ae1975a8ca3063710c: Status 404 returned error can't find the container with id 78469dd05740430c16f5a52058ccd5289720eaa624fd11ae1975a8ca3063710c
Jun 02 16:19:34.536578 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:34.536549    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9" event={"ID":"38a3c9b9-db00-46dc-9524-69e0c0b40366","Type":"ContainerStarted","Data":"9fedaacde54ee12170bb4c90023276857a7a0bb1b06505bb1fe00f78f6efb0d3"}
Jun 02 16:19:34.536705 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:34.536584    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9" event={"ID":"38a3c9b9-db00-46dc-9524-69e0c0b40366","Type":"ContainerStarted","Data":"78469dd05740430c16f5a52058ccd5289720eaa624fd11ae1975a8ca3063710c"}
Jun 02 16:19:39.551583 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:39.551547    2579 generic.go:358] "Generic (PLEG): container finished" podID="38a3c9b9-db00-46dc-9524-69e0c0b40366" containerID="9fedaacde54ee12170bb4c90023276857a7a0bb1b06505bb1fe00f78f6efb0d3" exitCode=0
Jun 02 16:19:39.552008 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:39.551605    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9" event={"ID":"38a3c9b9-db00-46dc-9524-69e0c0b40366","Type":"ContainerDied","Data":"9fedaacde54ee12170bb4c90023276857a7a0bb1b06505bb1fe00f78f6efb0d3"}
Jun 02 16:19:40.555930 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:40.555895    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9" event={"ID":"38a3c9b9-db00-46dc-9524-69e0c0b40366","Type":"ContainerStarted","Data":"9eb57e5edf766476a2136e316611a97a6239c299bb400da6321c2ccb2eb1969f"}
Jun 02 16:19:40.555930 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:40.555933    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9" event={"ID":"38a3c9b9-db00-46dc-9524-69e0c0b40366","Type":"ContainerStarted","Data":"9a4a2096b342bdf4a20f5ae55e9d0d622f411b1e5124b54a99bb087d92f2ff9a"}
Jun 02 16:19:40.556356 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:40.556133    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9"
Jun 02 16:19:40.576038 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:40.575986    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9" podStartSLOduration=7.575966011 podStartE2EDuration="7.575966011s" podCreationTimestamp="2026-06-02 16:19:33 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:19:40.574225917 +0000 UTC m=+1989.100134348" watchObservedRunningTime="2026-06-02 16:19:40.575966011 +0000 UTC m=+1989.101874441"
Jun 02 16:19:41.559222 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:41.559188    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9"
Jun 02 16:19:41.560651 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:41.560622    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9" podUID="38a3c9b9-db00-46dc-9524-69e0c0b40366" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jun 02 16:19:42.561916 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:42.561872    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9" podUID="38a3c9b9-db00-46dc-9524-69e0c0b40366" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jun 02 16:19:47.566818 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:47.566785    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9"
Jun 02 16:19:47.567326 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:47.567297    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9" podUID="38a3c9b9-db00-46dc-9524-69e0c0b40366" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jun 02 16:19:57.568271 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:19:57.568232    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9" podUID="38a3c9b9-db00-46dc-9524-69e0c0b40366" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jun 02 16:20:07.568023 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:07.567974    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9" podUID="38a3c9b9-db00-46dc-9524-69e0c0b40366" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jun 02 16:20:17.567899 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:17.567854    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9" podUID="38a3c9b9-db00-46dc-9524-69e0c0b40366" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jun 02 16:20:27.568852 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:27.568816    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9"
Jun 02 16:20:35.013195 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:35.013162    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9"]
Jun 02 16:20:35.013650 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:35.013540    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9" podUID="38a3c9b9-db00-46dc-9524-69e0c0b40366" containerName="kserve-container" containerID="cri-o://9a4a2096b342bdf4a20f5ae55e9d0d622f411b1e5124b54a99bb087d92f2ff9a" gracePeriod=30
Jun 02 16:20:35.013650 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:35.013578    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9" podUID="38a3c9b9-db00-46dc-9524-69e0c0b40366" containerName="kube-rbac-proxy" containerID="cri-o://9eb57e5edf766476a2136e316611a97a6239c299bb400da6321c2ccb2eb1969f" gracePeriod=30
Jun 02 16:20:35.118166 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:35.118126    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m"]
Jun 02 16:20:35.121707 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:35.121687    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m"
Jun 02 16:20:35.124184 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:35.124160    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Jun 02 16:20:35.124332 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:35.124191    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Jun 02 16:20:35.132337 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:35.132315    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m"]
Jun 02 16:20:35.233728 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:35.233675    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1e98fa0-b8f3-4de9-bce9-793214ffd7f6-isvc-pmml-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8bb578669-p578m\" (UID: \"c1e98fa0-b8f3-4de9-bce9-793214ffd7f6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m"
Jun 02 16:20:35.233904 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:35.233749    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1e98fa0-b8f3-4de9-bce9-793214ffd7f6-kserve-provision-location\") pod \"isvc-pmml-predictor-8bb578669-p578m\" (UID: \"c1e98fa0-b8f3-4de9-bce9-793214ffd7f6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m"
Jun 02 16:20:35.233904 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:35.233781    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1e98fa0-b8f3-4de9-bce9-793214ffd7f6-proxy-tls\") pod \"isvc-pmml-predictor-8bb578669-p578m\" (UID: \"c1e98fa0-b8f3-4de9-bce9-793214ffd7f6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m"
Jun 02 16:20:35.233904 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:35.233807    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh6k7\" (UniqueName: \"kubernetes.io/projected/c1e98fa0-b8f3-4de9-bce9-793214ffd7f6-kube-api-access-hh6k7\") pod \"isvc-pmml-predictor-8bb578669-p578m\" (UID: \"c1e98fa0-b8f3-4de9-bce9-793214ffd7f6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m"
Jun 02 16:20:35.335007 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:35.334916    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1e98fa0-b8f3-4de9-bce9-793214ffd7f6-isvc-pmml-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8bb578669-p578m\" (UID: \"c1e98fa0-b8f3-4de9-bce9-793214ffd7f6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m"
Jun 02 16:20:35.335007 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:35.334971    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1e98fa0-b8f3-4de9-bce9-793214ffd7f6-kserve-provision-location\") pod \"isvc-pmml-predictor-8bb578669-p578m\" (UID: \"c1e98fa0-b8f3-4de9-bce9-793214ffd7f6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m"
Jun 02 16:20:35.335007 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:35.335006    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1e98fa0-b8f3-4de9-bce9-793214ffd7f6-proxy-tls\") pod \"isvc-pmml-predictor-8bb578669-p578m\" (UID: \"c1e98fa0-b8f3-4de9-bce9-793214ffd7f6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m"
Jun 02 16:20:35.335272 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:35.335032    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hh6k7\" (UniqueName: \"kubernetes.io/projected/c1e98fa0-b8f3-4de9-bce9-793214ffd7f6-kube-api-access-hh6k7\") pod \"isvc-pmml-predictor-8bb578669-p578m\" (UID: \"c1e98fa0-b8f3-4de9-bce9-793214ffd7f6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m"
Jun 02 16:20:35.335490 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:35.335471    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1e98fa0-b8f3-4de9-bce9-793214ffd7f6-kserve-provision-location\") pod \"isvc-pmml-predictor-8bb578669-p578m\" (UID: \"c1e98fa0-b8f3-4de9-bce9-793214ffd7f6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m"
Jun 02 16:20:35.335652 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:35.335632    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1e98fa0-b8f3-4de9-bce9-793214ffd7f6-isvc-pmml-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8bb578669-p578m\" (UID: \"c1e98fa0-b8f3-4de9-bce9-793214ffd7f6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m"
Jun 02 16:20:35.337675 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:35.337653    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1e98fa0-b8f3-4de9-bce9-793214ffd7f6-proxy-tls\") pod \"isvc-pmml-predictor-8bb578669-p578m\" (UID: \"c1e98fa0-b8f3-4de9-bce9-793214ffd7f6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m"
Jun 02 16:20:35.343429 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:35.343375    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh6k7\" (UniqueName: \"kubernetes.io/projected/c1e98fa0-b8f3-4de9-bce9-793214ffd7f6-kube-api-access-hh6k7\") pod \"isvc-pmml-predictor-8bb578669-p578m\" (UID: \"c1e98fa0-b8f3-4de9-bce9-793214ffd7f6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m"
Jun 02 16:20:35.432691 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:35.432648    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m"
Jun 02 16:20:35.561117 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:35.561083    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m"]
Jun 02 16:20:35.564957 ip-10-0-129-156 kubenswrapper[2579]: W0602 16:20:35.564926    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1e98fa0_b8f3_4de9_bce9_793214ffd7f6.slice/crio-04be41392202b10786012c83f866c33738e8308f2744ec2b059d72fd5b97c85d WatchSource:0}: Error finding container 04be41392202b10786012c83f866c33738e8308f2744ec2b059d72fd5b97c85d: Status 404 returned error can't find the container with id 04be41392202b10786012c83f866c33738e8308f2744ec2b059d72fd5b97c85d
Jun 02 16:20:35.712119 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:35.712074    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m" event={"ID":"c1e98fa0-b8f3-4de9-bce9-793214ffd7f6","Type":"ContainerStarted","Data":"decf40518491ce65ec0afd308af316db56a461a11476fe2acc973b4fd62daea2"}
Jun 02 16:20:35.712119 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:35.712128    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m" event={"ID":"c1e98fa0-b8f3-4de9-bce9-793214ffd7f6","Type":"ContainerStarted","Data":"04be41392202b10786012c83f866c33738e8308f2744ec2b059d72fd5b97c85d"}
Jun 02 16:20:35.713986 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:35.713958    2579 generic.go:358] "Generic (PLEG): container finished" podID="38a3c9b9-db00-46dc-9524-69e0c0b40366" containerID="9eb57e5edf766476a2136e316611a97a6239c299bb400da6321c2ccb2eb1969f" exitCode=2
Jun 02 16:20:35.714109 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:35.713996    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9" event={"ID":"38a3c9b9-db00-46dc-9524-69e0c0b40366","Type":"ContainerDied","Data":"9eb57e5edf766476a2136e316611a97a6239c299bb400da6321c2ccb2eb1969f"}
Jun 02 16:20:37.562879 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:37.562834    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9" podUID="38a3c9b9-db00-46dc-9524-69e0c0b40366" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.18:8643/healthz\": dial tcp 10.133.0.18:8643: connect: connection refused"
Jun 02 16:20:37.568113 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:37.568076    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9" podUID="38a3c9b9-db00-46dc-9524-69e0c0b40366" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jun 02 16:20:38.063381 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:38.063354    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9"
Jun 02 16:20:38.158529 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:38.158431    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38a3c9b9-db00-46dc-9524-69e0c0b40366-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") pod \"38a3c9b9-db00-46dc-9524-69e0c0b40366\" (UID: \"38a3c9b9-db00-46dc-9524-69e0c0b40366\") "
Jun 02 16:20:38.158683 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:38.158532    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38a3c9b9-db00-46dc-9524-69e0c0b40366-proxy-tls\") pod \"38a3c9b9-db00-46dc-9524-69e0c0b40366\" (UID: \"38a3c9b9-db00-46dc-9524-69e0c0b40366\") "
Jun 02 16:20:38.158683 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:38.158592    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38a3c9b9-db00-46dc-9524-69e0c0b40366-kserve-provision-location\") pod \"38a3c9b9-db00-46dc-9524-69e0c0b40366\" (UID: \"38a3c9b9-db00-46dc-9524-69e0c0b40366\") "
Jun 02 16:20:38.158683 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:38.158626    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wgsfp\" (UniqueName: \"kubernetes.io/projected/38a3c9b9-db00-46dc-9524-69e0c0b40366-kube-api-access-wgsfp\") pod \"38a3c9b9-db00-46dc-9524-69e0c0b40366\" (UID: \"38a3c9b9-db00-46dc-9524-69e0c0b40366\") "
Jun 02 16:20:38.158906 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:38.158876    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38a3c9b9-db00-46dc-9524-69e0c0b40366-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config") pod "38a3c9b9-db00-46dc-9524-69e0c0b40366" (UID: "38a3c9b9-db00-46dc-9524-69e0c0b40366"). InnerVolumeSpecName "isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:20:38.160869 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:38.160845    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38a3c9b9-db00-46dc-9524-69e0c0b40366-kube-api-access-wgsfp" (OuterVolumeSpecName: "kube-api-access-wgsfp") pod "38a3c9b9-db00-46dc-9524-69e0c0b40366" (UID: "38a3c9b9-db00-46dc-9524-69e0c0b40366"). InnerVolumeSpecName "kube-api-access-wgsfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:20:38.160947 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:38.160864    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38a3c9b9-db00-46dc-9524-69e0c0b40366-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "38a3c9b9-db00-46dc-9524-69e0c0b40366" (UID: "38a3c9b9-db00-46dc-9524-69e0c0b40366"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:20:38.172078 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:38.172045    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38a3c9b9-db00-46dc-9524-69e0c0b40366-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "38a3c9b9-db00-46dc-9524-69e0c0b40366" (UID: "38a3c9b9-db00-46dc-9524-69e0c0b40366"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:20:38.259159 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:38.259118    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38a3c9b9-db00-46dc-9524-69e0c0b40366-kserve-provision-location\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:20:38.259159 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:38.259151    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgsfp\" (UniqueName: \"kubernetes.io/projected/38a3c9b9-db00-46dc-9524-69e0c0b40366-kube-api-access-wgsfp\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:20:38.259159 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:38.259164    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38a3c9b9-db00-46dc-9524-69e0c0b40366-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:20:38.259396 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:38.259178    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38a3c9b9-db00-46dc-9524-69e0c0b40366-proxy-tls\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:20:38.723827 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:38.723790    2579 generic.go:358] "Generic (PLEG): container finished" podID="38a3c9b9-db00-46dc-9524-69e0c0b40366" containerID="9a4a2096b342bdf4a20f5ae55e9d0d622f411b1e5124b54a99bb087d92f2ff9a" exitCode=0
Jun 02 16:20:38.724240 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:38.723838    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9" event={"ID":"38a3c9b9-db00-46dc-9524-69e0c0b40366","Type":"ContainerDied","Data":"9a4a2096b342bdf4a20f5ae55e9d0d622f411b1e5124b54a99bb087d92f2ff9a"}
Jun 02 16:20:38.724240 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:38.723864    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9" event={"ID":"38a3c9b9-db00-46dc-9524-69e0c0b40366","Type":"ContainerDied","Data":"78469dd05740430c16f5a52058ccd5289720eaa624fd11ae1975a8ca3063710c"}
Jun 02 16:20:38.724240 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:38.723879    2579 scope.go:117] "RemoveContainer" containerID="9eb57e5edf766476a2136e316611a97a6239c299bb400da6321c2ccb2eb1969f"
Jun 02 16:20:38.724240 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:38.723895    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9"
Jun 02 16:20:38.732198 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:38.732178    2579 scope.go:117] "RemoveContainer" containerID="9a4a2096b342bdf4a20f5ae55e9d0d622f411b1e5124b54a99bb087d92f2ff9a"
Jun 02 16:20:38.739518 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:38.739487    2579 scope.go:117] "RemoveContainer" containerID="9fedaacde54ee12170bb4c90023276857a7a0bb1b06505bb1fe00f78f6efb0d3"
Jun 02 16:20:38.746213 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:38.746195    2579 scope.go:117] "RemoveContainer" containerID="9eb57e5edf766476a2136e316611a97a6239c299bb400da6321c2ccb2eb1969f"
Jun 02 16:20:38.746524 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:20:38.746498    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9eb57e5edf766476a2136e316611a97a6239c299bb400da6321c2ccb2eb1969f\": container with ID starting with 9eb57e5edf766476a2136e316611a97a6239c299bb400da6321c2ccb2eb1969f not found: ID does not exist" containerID="9eb57e5edf766476a2136e316611a97a6239c299bb400da6321c2ccb2eb1969f"
Jun 02 16:20:38.746614 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:38.746531    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eb57e5edf766476a2136e316611a97a6239c299bb400da6321c2ccb2eb1969f"} err="failed to get container status \"9eb57e5edf766476a2136e316611a97a6239c299bb400da6321c2ccb2eb1969f\": rpc error: code = NotFound desc = could not find container \"9eb57e5edf766476a2136e316611a97a6239c299bb400da6321c2ccb2eb1969f\": container with ID starting with 9eb57e5edf766476a2136e316611a97a6239c299bb400da6321c2ccb2eb1969f not found: ID does not exist"
Jun 02 16:20:38.746614 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:38.746552    2579 scope.go:117] "RemoveContainer" containerID="9a4a2096b342bdf4a20f5ae55e9d0d622f411b1e5124b54a99bb087d92f2ff9a"
Jun 02 16:20:38.746822 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:20:38.746800    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a4a2096b342bdf4a20f5ae55e9d0d622f411b1e5124b54a99bb087d92f2ff9a\": container with ID starting with 9a4a2096b342bdf4a20f5ae55e9d0d622f411b1e5124b54a99bb087d92f2ff9a not found: ID does not exist" containerID="9a4a2096b342bdf4a20f5ae55e9d0d622f411b1e5124b54a99bb087d92f2ff9a"
Jun 02 16:20:38.746895 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:38.746827    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a4a2096b342bdf4a20f5ae55e9d0d622f411b1e5124b54a99bb087d92f2ff9a"} err="failed to get container status \"9a4a2096b342bdf4a20f5ae55e9d0d622f411b1e5124b54a99bb087d92f2ff9a\": rpc error: code = NotFound desc = could not find container \"9a4a2096b342bdf4a20f5ae55e9d0d622f411b1e5124b54a99bb087d92f2ff9a\": container with ID starting with 9a4a2096b342bdf4a20f5ae55e9d0d622f411b1e5124b54a99bb087d92f2ff9a not found: ID does not exist"
Jun 02 16:20:38.746895 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:38.746844    2579 scope.go:117] "RemoveContainer" containerID="9fedaacde54ee12170bb4c90023276857a7a0bb1b06505bb1fe00f78f6efb0d3"
Jun 02 16:20:38.747081 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:20:38.747064    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fedaacde54ee12170bb4c90023276857a7a0bb1b06505bb1fe00f78f6efb0d3\": container with ID starting with 9fedaacde54ee12170bb4c90023276857a7a0bb1b06505bb1fe00f78f6efb0d3 not found: ID does not exist" containerID="9fedaacde54ee12170bb4c90023276857a7a0bb1b06505bb1fe00f78f6efb0d3"
Jun 02 16:20:38.747116 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:38.747086    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fedaacde54ee12170bb4c90023276857a7a0bb1b06505bb1fe00f78f6efb0d3"} err="failed to get container status \"9fedaacde54ee12170bb4c90023276857a7a0bb1b06505bb1fe00f78f6efb0d3\": rpc error: code = NotFound desc = could not find container \"9fedaacde54ee12170bb4c90023276857a7a0bb1b06505bb1fe00f78f6efb0d3\": container with ID starting with 9fedaacde54ee12170bb4c90023276857a7a0bb1b06505bb1fe00f78f6efb0d3 not found: ID does not exist"
Jun 02 16:20:38.753627 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:38.753606    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9"]
Jun 02 16:20:38.756982 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:38.756963    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-v2nc9"]
Jun 02 16:20:39.728942 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:39.728911    2579 generic.go:358] "Generic (PLEG): container finished" podID="c1e98fa0-b8f3-4de9-bce9-793214ffd7f6" containerID="decf40518491ce65ec0afd308af316db56a461a11476fe2acc973b4fd62daea2" exitCode=0
Jun 02 16:20:39.729384 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:39.728986    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m" event={"ID":"c1e98fa0-b8f3-4de9-bce9-793214ffd7f6","Type":"ContainerDied","Data":"decf40518491ce65ec0afd308af316db56a461a11476fe2acc973b4fd62daea2"}
Jun 02 16:20:40.064477 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:40.064365    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38a3c9b9-db00-46dc-9524-69e0c0b40366" path="/var/lib/kubelet/pods/38a3c9b9-db00-46dc-9524-69e0c0b40366/volumes"
Jun 02 16:20:47.758734 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:47.758699    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m" event={"ID":"c1e98fa0-b8f3-4de9-bce9-793214ffd7f6","Type":"ContainerStarted","Data":"1692dc5d1a7c353f3f4787b4954ee913d42293d1b66fe5b4c658ca0eeed08c96"}
Jun 02 16:20:47.759100 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:47.758745    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m" event={"ID":"c1e98fa0-b8f3-4de9-bce9-793214ffd7f6","Type":"ContainerStarted","Data":"f76c399772ddd0732713075251fa8f56be6453476c7060523c1f8aca07954965"}
Jun 02 16:20:47.759100 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:47.758956    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m"
Jun 02 16:20:47.780593 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:47.780537    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m" podStartSLOduration=5.410955874 podStartE2EDuration="12.780520399s" podCreationTimestamp="2026-06-02 16:20:35 +0000 UTC" firstStartedPulling="2026-06-02 16:20:39.730275976 +0000 UTC m=+2048.256184384" lastFinishedPulling="2026-06-02 16:20:47.099840484 +0000 UTC m=+2055.625748909" observedRunningTime="2026-06-02 16:20:47.779235448 +0000 UTC m=+2056.305143887" watchObservedRunningTime="2026-06-02 16:20:47.780520399 +0000 UTC m=+2056.306428830"
Jun 02 16:20:48.762049 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:48.762004    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m"
Jun 02 16:20:48.763151 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:48.763127    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m" podUID="c1e98fa0-b8f3-4de9-bce9-793214ffd7f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jun 02 16:20:49.765162 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:49.765071    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m" podUID="c1e98fa0-b8f3-4de9-bce9-793214ffd7f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jun 02 16:20:54.769923 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:54.769891    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m"
Jun 02 16:20:54.770525 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:20:54.770483    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m" podUID="c1e98fa0-b8f3-4de9-bce9-793214ffd7f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jun 02 16:21:04.771417 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:21:04.771357    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m" podUID="c1e98fa0-b8f3-4de9-bce9-793214ffd7f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jun 02 16:21:14.771174 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:21:14.771132    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m" podUID="c1e98fa0-b8f3-4de9-bce9-793214ffd7f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jun 02 16:21:24.771034 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:21:24.770992    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m" podUID="c1e98fa0-b8f3-4de9-bce9-793214ffd7f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jun 02 16:21:32.042949 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:21:32.042921    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lqx9k_6127e33c-aa77-437e-b443-e956037afedb/ovn-acl-logging/0.log"
Jun 02 16:21:32.045458 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:21:32.045434    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lqx9k_6127e33c-aa77-437e-b443-e956037afedb/ovn-acl-logging/0.log"
Jun 02 16:21:34.770596 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:21:34.770554    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m" podUID="c1e98fa0-b8f3-4de9-bce9-793214ffd7f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jun 02 16:21:44.770394 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:21:44.770352    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m" podUID="c1e98fa0-b8f3-4de9-bce9-793214ffd7f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jun 02 16:21:54.771279 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:21:54.771228    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m" podUID="c1e98fa0-b8f3-4de9-bce9-793214ffd7f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jun 02 16:21:58.060367 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:21:58.060323    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m" podUID="c1e98fa0-b8f3-4de9-bce9-793214ffd7f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jun 02 16:22:08.065157 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:08.065128    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m"
Jun 02 16:22:16.162757 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:16.162711    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m"]
Jun 02 16:22:16.163147 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:16.163125    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m" podUID="c1e98fa0-b8f3-4de9-bce9-793214ffd7f6" containerName="kserve-container" containerID="cri-o://f76c399772ddd0732713075251fa8f56be6453476c7060523c1f8aca07954965" gracePeriod=30
Jun 02 16:22:16.163233 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:16.163174    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m" podUID="c1e98fa0-b8f3-4de9-bce9-793214ffd7f6" containerName="kube-rbac-proxy" containerID="cri-o://1692dc5d1a7c353f3f4787b4954ee913d42293d1b66fe5b4c658ca0eeed08c96" gracePeriod=30
Jun 02 16:22:16.272624 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:16.272587    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp"]
Jun 02 16:22:16.272965 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:16.272947    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38a3c9b9-db00-46dc-9524-69e0c0b40366" containerName="storage-initializer"
Jun 02 16:22:16.273046 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:16.272968    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="38a3c9b9-db00-46dc-9524-69e0c0b40366" containerName="storage-initializer"
Jun 02 16:22:16.273046 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:16.272983    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38a3c9b9-db00-46dc-9524-69e0c0b40366" containerName="kube-rbac-proxy"
Jun 02 16:22:16.273046 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:16.272990    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="38a3c9b9-db00-46dc-9524-69e0c0b40366" containerName="kube-rbac-proxy"
Jun 02 16:22:16.273046 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:16.273004    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38a3c9b9-db00-46dc-9524-69e0c0b40366" containerName="kserve-container"
Jun 02 16:22:16.273046 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:16.273013    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="38a3c9b9-db00-46dc-9524-69e0c0b40366" containerName="kserve-container"
Jun 02 16:22:16.273290 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:16.273083    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="38a3c9b9-db00-46dc-9524-69e0c0b40366" containerName="kube-rbac-proxy"
Jun 02 16:22:16.273290 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:16.273097    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="38a3c9b9-db00-46dc-9524-69e0c0b40366" containerName="kserve-container"
Jun 02 16:22:16.276756 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:16.276738    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp"
Jun 02 16:22:16.278960 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:16.278937    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Jun 02 16:22:16.279075 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:16.278945    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Jun 02 16:22:16.286180 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:16.286157    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp"]
Jun 02 16:22:16.436505 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:16.436393    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/883a6f75-9aaf-45a1-b230-5a7e3aec70c9-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-67bc544947-ltgcp\" (UID: \"883a6f75-9aaf-45a1-b230-5a7e3aec70c9\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp"
Jun 02 16:22:16.436505 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:16.436471    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54plx\" (UniqueName: \"kubernetes.io/projected/883a6f75-9aaf-45a1-b230-5a7e3aec70c9-kube-api-access-54plx\") pod \"isvc-pmml-runtime-predictor-67bc544947-ltgcp\" (UID: \"883a6f75-9aaf-45a1-b230-5a7e3aec70c9\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp"
Jun 02 16:22:16.436688 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:16.436512    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/883a6f75-9aaf-45a1-b230-5a7e3aec70c9-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-67bc544947-ltgcp\" (UID: \"883a6f75-9aaf-45a1-b230-5a7e3aec70c9\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp"
Jun 02 16:22:16.436688 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:16.436530    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/883a6f75-9aaf-45a1-b230-5a7e3aec70c9-proxy-tls\") pod \"isvc-pmml-runtime-predictor-67bc544947-ltgcp\" (UID: \"883a6f75-9aaf-45a1-b230-5a7e3aec70c9\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp"
Jun 02 16:22:16.536914 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:16.536878    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/883a6f75-9aaf-45a1-b230-5a7e3aec70c9-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-67bc544947-ltgcp\" (UID: \"883a6f75-9aaf-45a1-b230-5a7e3aec70c9\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp"
Jun 02 16:22:16.536914 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:16.536919    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/883a6f75-9aaf-45a1-b230-5a7e3aec70c9-proxy-tls\") pod \"isvc-pmml-runtime-predictor-67bc544947-ltgcp\" (UID: \"883a6f75-9aaf-45a1-b230-5a7e3aec70c9\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp"
Jun 02 16:22:16.537149 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:16.536968    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/883a6f75-9aaf-45a1-b230-5a7e3aec70c9-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-67bc544947-ltgcp\" (UID: \"883a6f75-9aaf-45a1-b230-5a7e3aec70c9\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp"
Jun 02 16:22:16.537149 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:16.537000    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-54plx\" (UniqueName: \"kubernetes.io/projected/883a6f75-9aaf-45a1-b230-5a7e3aec70c9-kube-api-access-54plx\") pod \"isvc-pmml-runtime-predictor-67bc544947-ltgcp\" (UID: \"883a6f75-9aaf-45a1-b230-5a7e3aec70c9\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp"
Jun 02 16:22:16.537364 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:16.537345    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/883a6f75-9aaf-45a1-b230-5a7e3aec70c9-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-67bc544947-ltgcp\" (UID: \"883a6f75-9aaf-45a1-b230-5a7e3aec70c9\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp"
Jun 02 16:22:16.537816 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:16.537787    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/883a6f75-9aaf-45a1-b230-5a7e3aec70c9-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-67bc544947-ltgcp\" (UID: \"883a6f75-9aaf-45a1-b230-5a7e3aec70c9\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp"
Jun 02 16:22:16.539666 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:16.539646    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/883a6f75-9aaf-45a1-b230-5a7e3aec70c9-proxy-tls\") pod \"isvc-pmml-runtime-predictor-67bc544947-ltgcp\" (UID: \"883a6f75-9aaf-45a1-b230-5a7e3aec70c9\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp"
Jun 02 16:22:16.546719 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:16.546697    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54plx\" (UniqueName: \"kubernetes.io/projected/883a6f75-9aaf-45a1-b230-5a7e3aec70c9-kube-api-access-54plx\") pod \"isvc-pmml-runtime-predictor-67bc544947-ltgcp\" (UID: \"883a6f75-9aaf-45a1-b230-5a7e3aec70c9\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp"
Jun 02 16:22:16.587726 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:16.587690    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp"
Jun 02 16:22:16.720013 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:16.719882    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp"]
Jun 02 16:22:16.722675 ip-10-0-129-156 kubenswrapper[2579]: W0602 16:22:16.722639    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod883a6f75_9aaf_45a1_b230_5a7e3aec70c9.slice/crio-8e0598b5dabc6def92d70d9fd04535bc885f1dd9262970a21012aeb3164a4521 WatchSource:0}: Error finding container 8e0598b5dabc6def92d70d9fd04535bc885f1dd9262970a21012aeb3164a4521: Status 404 returned error can't find the container with id 8e0598b5dabc6def92d70d9fd04535bc885f1dd9262970a21012aeb3164a4521
Jun 02 16:22:16.724558 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:16.724538    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 16:22:17.014935 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:17.014892    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp" event={"ID":"883a6f75-9aaf-45a1-b230-5a7e3aec70c9","Type":"ContainerStarted","Data":"ac6afb169407110d99f9277a6d81065d5dd76f2d238d79471f2c1f3f2f937d93"}
Jun 02 16:22:17.014935 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:17.014941    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp" event={"ID":"883a6f75-9aaf-45a1-b230-5a7e3aec70c9","Type":"ContainerStarted","Data":"8e0598b5dabc6def92d70d9fd04535bc885f1dd9262970a21012aeb3164a4521"}
Jun 02 16:22:17.017010 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:17.016983    2579 generic.go:358] "Generic (PLEG): container finished" podID="c1e98fa0-b8f3-4de9-bce9-793214ffd7f6" containerID="1692dc5d1a7c353f3f4787b4954ee913d42293d1b66fe5b4c658ca0eeed08c96" exitCode=2
Jun 02 16:22:17.017136 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:17.017030    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m" event={"ID":"c1e98fa0-b8f3-4de9-bce9-793214ffd7f6","Type":"ContainerDied","Data":"1692dc5d1a7c353f3f4787b4954ee913d42293d1b66fe5b4c658ca0eeed08c96"}
Jun 02 16:22:18.060774 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:18.060729    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m" podUID="c1e98fa0-b8f3-4de9-bce9-793214ffd7f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jun 02 16:22:19.766432 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:19.766314    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m" podUID="c1e98fa0-b8f3-4de9-bce9-793214ffd7f6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.19:8643/healthz\": dial tcp 10.133.0.19:8643: connect: connection refused"
Jun 02 16:22:20.003810 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:20.003783    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m"
Jun 02 16:22:20.028424 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:20.028320    2579 generic.go:358] "Generic (PLEG): container finished" podID="c1e98fa0-b8f3-4de9-bce9-793214ffd7f6" containerID="f76c399772ddd0732713075251fa8f56be6453476c7060523c1f8aca07954965" exitCode=0
Jun 02 16:22:20.028591 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:20.028449    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m" event={"ID":"c1e98fa0-b8f3-4de9-bce9-793214ffd7f6","Type":"ContainerDied","Data":"f76c399772ddd0732713075251fa8f56be6453476c7060523c1f8aca07954965"}
Jun 02 16:22:20.028591 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:20.028483    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m" event={"ID":"c1e98fa0-b8f3-4de9-bce9-793214ffd7f6","Type":"ContainerDied","Data":"04be41392202b10786012c83f866c33738e8308f2744ec2b059d72fd5b97c85d"}
Jun 02 16:22:20.028591 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:20.028498    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m"
Jun 02 16:22:20.028591 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:20.028507    2579 scope.go:117] "RemoveContainer" containerID="1692dc5d1a7c353f3f4787b4954ee913d42293d1b66fe5b4c658ca0eeed08c96"
Jun 02 16:22:20.039197 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:20.039176    2579 scope.go:117] "RemoveContainer" containerID="f76c399772ddd0732713075251fa8f56be6453476c7060523c1f8aca07954965"
Jun 02 16:22:20.046364 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:20.046340    2579 scope.go:117] "RemoveContainer" containerID="decf40518491ce65ec0afd308af316db56a461a11476fe2acc973b4fd62daea2"
Jun 02 16:22:20.053257 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:20.053239    2579 scope.go:117] "RemoveContainer" containerID="1692dc5d1a7c353f3f4787b4954ee913d42293d1b66fe5b4c658ca0eeed08c96"
Jun 02 16:22:20.053618 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:22:20.053599    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1692dc5d1a7c353f3f4787b4954ee913d42293d1b66fe5b4c658ca0eeed08c96\": container with ID starting with 1692dc5d1a7c353f3f4787b4954ee913d42293d1b66fe5b4c658ca0eeed08c96 not found: ID does not exist" containerID="1692dc5d1a7c353f3f4787b4954ee913d42293d1b66fe5b4c658ca0eeed08c96"
Jun 02 16:22:20.053698 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:20.053626    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1692dc5d1a7c353f3f4787b4954ee913d42293d1b66fe5b4c658ca0eeed08c96"} err="failed to get container status \"1692dc5d1a7c353f3f4787b4954ee913d42293d1b66fe5b4c658ca0eeed08c96\": rpc error: code = NotFound desc = could not find container \"1692dc5d1a7c353f3f4787b4954ee913d42293d1b66fe5b4c658ca0eeed08c96\": container with ID starting with 1692dc5d1a7c353f3f4787b4954ee913d42293d1b66fe5b4c658ca0eeed08c96 not found: ID does not exist"
Jun 02 16:22:20.053698 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:20.053644    2579 scope.go:117] "RemoveContainer" containerID="f76c399772ddd0732713075251fa8f56be6453476c7060523c1f8aca07954965"
Jun 02 16:22:20.053889 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:22:20.053869    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f76c399772ddd0732713075251fa8f56be6453476c7060523c1f8aca07954965\": container with ID starting with f76c399772ddd0732713075251fa8f56be6453476c7060523c1f8aca07954965 not found: ID does not exist" containerID="f76c399772ddd0732713075251fa8f56be6453476c7060523c1f8aca07954965"
Jun 02 16:22:20.053930 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:20.053895    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f76c399772ddd0732713075251fa8f56be6453476c7060523c1f8aca07954965"} err="failed to get container status \"f76c399772ddd0732713075251fa8f56be6453476c7060523c1f8aca07954965\": rpc error: code = NotFound desc = could not find container \"f76c399772ddd0732713075251fa8f56be6453476c7060523c1f8aca07954965\": container with ID starting with f76c399772ddd0732713075251fa8f56be6453476c7060523c1f8aca07954965 not found: ID does not exist"
Jun 02 16:22:20.053930 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:20.053912    2579 scope.go:117] "RemoveContainer" containerID="decf40518491ce65ec0afd308af316db56a461a11476fe2acc973b4fd62daea2"
Jun 02 16:22:20.054174 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:22:20.054152    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"decf40518491ce65ec0afd308af316db56a461a11476fe2acc973b4fd62daea2\": container with ID starting with decf40518491ce65ec0afd308af316db56a461a11476fe2acc973b4fd62daea2 not found: ID does not exist" containerID="decf40518491ce65ec0afd308af316db56a461a11476fe2acc973b4fd62daea2"
Jun 02 16:22:20.054222 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:20.054178    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"decf40518491ce65ec0afd308af316db56a461a11476fe2acc973b4fd62daea2"} err="failed to get container status \"decf40518491ce65ec0afd308af316db56a461a11476fe2acc973b4fd62daea2\": rpc error: code = NotFound desc = could not find container \"decf40518491ce65ec0afd308af316db56a461a11476fe2acc973b4fd62daea2\": container with ID starting with decf40518491ce65ec0afd308af316db56a461a11476fe2acc973b4fd62daea2 not found: ID does not exist"
Jun 02 16:22:20.163721 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:20.163682    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1e98fa0-b8f3-4de9-bce9-793214ffd7f6-proxy-tls\") pod \"c1e98fa0-b8f3-4de9-bce9-793214ffd7f6\" (UID: \"c1e98fa0-b8f3-4de9-bce9-793214ffd7f6\") "
Jun 02 16:22:20.163925 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:20.163740    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1e98fa0-b8f3-4de9-bce9-793214ffd7f6-kserve-provision-location\") pod \"c1e98fa0-b8f3-4de9-bce9-793214ffd7f6\" (UID: \"c1e98fa0-b8f3-4de9-bce9-793214ffd7f6\") "
Jun 02 16:22:20.163925 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:20.163765    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hh6k7\" (UniqueName: \"kubernetes.io/projected/c1e98fa0-b8f3-4de9-bce9-793214ffd7f6-kube-api-access-hh6k7\") pod \"c1e98fa0-b8f3-4de9-bce9-793214ffd7f6\" (UID: \"c1e98fa0-b8f3-4de9-bce9-793214ffd7f6\") "
Jun 02 16:22:20.163925 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:20.163872    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1e98fa0-b8f3-4de9-bce9-793214ffd7f6-isvc-pmml-kube-rbac-proxy-sar-config\") pod \"c1e98fa0-b8f3-4de9-bce9-793214ffd7f6\" (UID: \"c1e98fa0-b8f3-4de9-bce9-793214ffd7f6\") "
Jun 02 16:22:20.164142 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:20.164067    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e98fa0-b8f3-4de9-bce9-793214ffd7f6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c1e98fa0-b8f3-4de9-bce9-793214ffd7f6" (UID: "c1e98fa0-b8f3-4de9-bce9-793214ffd7f6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:22:20.164263 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:20.164198    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1e98fa0-b8f3-4de9-bce9-793214ffd7f6-isvc-pmml-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-pmml-kube-rbac-proxy-sar-config") pod "c1e98fa0-b8f3-4de9-bce9-793214ffd7f6" (UID: "c1e98fa0-b8f3-4de9-bce9-793214ffd7f6"). InnerVolumeSpecName "isvc-pmml-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:22:20.165992 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:20.165969    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1e98fa0-b8f3-4de9-bce9-793214ffd7f6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c1e98fa0-b8f3-4de9-bce9-793214ffd7f6" (UID: "c1e98fa0-b8f3-4de9-bce9-793214ffd7f6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:22:20.166050 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:20.166027    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1e98fa0-b8f3-4de9-bce9-793214ffd7f6-kube-api-access-hh6k7" (OuterVolumeSpecName: "kube-api-access-hh6k7") pod "c1e98fa0-b8f3-4de9-bce9-793214ffd7f6" (UID: "c1e98fa0-b8f3-4de9-bce9-793214ffd7f6"). InnerVolumeSpecName "kube-api-access-hh6k7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:22:20.264976 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:20.264933    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1e98fa0-b8f3-4de9-bce9-793214ffd7f6-isvc-pmml-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:22:20.264976 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:20.264969    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1e98fa0-b8f3-4de9-bce9-793214ffd7f6-proxy-tls\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:22:20.265194 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:20.264985    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1e98fa0-b8f3-4de9-bce9-793214ffd7f6-kserve-provision-location\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:22:20.265194 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:20.265001    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh6k7\" (UniqueName: \"kubernetes.io/projected/c1e98fa0-b8f3-4de9-bce9-793214ffd7f6-kube-api-access-hh6k7\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:22:20.356300 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:20.356259    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m"]
Jun 02 16:22:20.360908 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:20.360876    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p578m"]
Jun 02 16:22:21.033148 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:21.033111    2579 generic.go:358] "Generic (PLEG): container finished" podID="883a6f75-9aaf-45a1-b230-5a7e3aec70c9" containerID="ac6afb169407110d99f9277a6d81065d5dd76f2d238d79471f2c1f3f2f937d93" exitCode=0
Jun 02 16:22:21.033599 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:21.033190    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp" event={"ID":"883a6f75-9aaf-45a1-b230-5a7e3aec70c9","Type":"ContainerDied","Data":"ac6afb169407110d99f9277a6d81065d5dd76f2d238d79471f2c1f3f2f937d93"}
Jun 02 16:22:22.038744 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:22.038708    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp" event={"ID":"883a6f75-9aaf-45a1-b230-5a7e3aec70c9","Type":"ContainerStarted","Data":"948de2d57cc61b5ccfce7047a6b87d69df72b9263fff9a8785014472ac5e246f"}
Jun 02 16:22:22.038744 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:22.038748    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp" event={"ID":"883a6f75-9aaf-45a1-b230-5a7e3aec70c9","Type":"ContainerStarted","Data":"f8d61cce540bd06befe653bb93cba522eacd153a5c9e74b238a3009cebf44de7"}
Jun 02 16:22:22.039309 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:22.038957    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp"
Jun 02 16:22:22.061139 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:22.061069    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp" podStartSLOduration=6.061048488 podStartE2EDuration="6.061048488s" podCreationTimestamp="2026-06-02 16:22:16 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:22:22.059010712 +0000 UTC m=+2150.584919141" watchObservedRunningTime="2026-06-02 16:22:22.061048488 +0000 UTC m=+2150.586956991"
Jun 02 16:22:22.064438 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:22.064393    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1e98fa0-b8f3-4de9-bce9-793214ffd7f6" path="/var/lib/kubelet/pods/c1e98fa0-b8f3-4de9-bce9-793214ffd7f6/volumes"
Jun 02 16:22:23.042323 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:23.042293    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp"
Jun 02 16:22:23.043440 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:23.043394    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp" podUID="883a6f75-9aaf-45a1-b230-5a7e3aec70c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jun 02 16:22:24.044767 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:24.044714    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp" podUID="883a6f75-9aaf-45a1-b230-5a7e3aec70c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jun 02 16:22:29.049035 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:29.049005    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp"
Jun 02 16:22:29.049502 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:29.049475    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp" podUID="883a6f75-9aaf-45a1-b230-5a7e3aec70c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jun 02 16:22:39.050152 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:39.050105    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp" podUID="883a6f75-9aaf-45a1-b230-5a7e3aec70c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jun 02 16:22:49.050020 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:49.049974    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp" podUID="883a6f75-9aaf-45a1-b230-5a7e3aec70c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jun 02 16:22:59.050334 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:22:59.050293    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp" podUID="883a6f75-9aaf-45a1-b230-5a7e3aec70c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jun 02 16:23:09.049988 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:23:09.049949    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp" podUID="883a6f75-9aaf-45a1-b230-5a7e3aec70c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jun 02 16:23:19.050185 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:23:19.050097    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp" podUID="883a6f75-9aaf-45a1-b230-5a7e3aec70c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jun 02 16:23:29.050096 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:23:29.050052    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp" podUID="883a6f75-9aaf-45a1-b230-5a7e3aec70c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jun 02 16:23:39.050313 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:23:39.050273    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp" podUID="883a6f75-9aaf-45a1-b230-5a7e3aec70c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jun 02 16:23:49.050177 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:23:49.050138    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp"
Jun 02 16:23:57.347740 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:23:57.347695    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp"]
Jun 02 16:23:57.348244 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:23:57.348119    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp" podUID="883a6f75-9aaf-45a1-b230-5a7e3aec70c9" containerName="kserve-container" containerID="cri-o://f8d61cce540bd06befe653bb93cba522eacd153a5c9e74b238a3009cebf44de7" gracePeriod=30
Jun 02 16:23:57.348244 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:23:57.348132    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp" podUID="883a6f75-9aaf-45a1-b230-5a7e3aec70c9" containerName="kube-rbac-proxy" containerID="cri-o://948de2d57cc61b5ccfce7047a6b87d69df72b9263fff9a8785014472ac5e246f" gracePeriod=30
Jun 02 16:23:57.435643 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:23:57.435611    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch"]
Jun 02 16:23:57.435903 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:23:57.435890    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1e98fa0-b8f3-4de9-bce9-793214ffd7f6" containerName="kube-rbac-proxy"
Jun 02 16:23:57.435947 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:23:57.435905    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1e98fa0-b8f3-4de9-bce9-793214ffd7f6" containerName="kube-rbac-proxy"
Jun 02 16:23:57.435947 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:23:57.435923    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1e98fa0-b8f3-4de9-bce9-793214ffd7f6" containerName="storage-initializer"
Jun 02 16:23:57.435947 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:23:57.435928    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1e98fa0-b8f3-4de9-bce9-793214ffd7f6" containerName="storage-initializer"
Jun 02 16:23:57.435947 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:23:57.435935    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1e98fa0-b8f3-4de9-bce9-793214ffd7f6" containerName="kserve-container"
Jun 02 16:23:57.435947 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:23:57.435940    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1e98fa0-b8f3-4de9-bce9-793214ffd7f6" containerName="kserve-container"
Jun 02 16:23:57.436108 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:23:57.435987    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="c1e98fa0-b8f3-4de9-bce9-793214ffd7f6" containerName="kserve-container"
Jun 02 16:23:57.436108 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:23:57.435996    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="c1e98fa0-b8f3-4de9-bce9-793214ffd7f6" containerName="kube-rbac-proxy"
Jun 02 16:23:57.439008 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:23:57.438987    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch"
Jun 02 16:23:57.441347 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:23:57.441326    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Jun 02 16:23:57.441480 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:23:57.441390    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Jun 02 16:23:57.448049 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:23:57.448025    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch"]
Jun 02 16:23:57.514380 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:23:57.514334    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjgn2\" (UniqueName: \"kubernetes.io/projected/0f51886f-ad3f-46d0-95a1-88cfc5d975cc-kube-api-access-mjgn2\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch\" (UID: \"0f51886f-ad3f-46d0-95a1-88cfc5d975cc\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch"
Jun 02 16:23:57.514575 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:23:57.514396    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f51886f-ad3f-46d0-95a1-88cfc5d975cc-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch\" (UID: \"0f51886f-ad3f-46d0-95a1-88cfc5d975cc\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch"
Jun 02 16:23:57.514575 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:23:57.514461    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f51886f-ad3f-46d0-95a1-88cfc5d975cc-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch\" (UID: \"0f51886f-ad3f-46d0-95a1-88cfc5d975cc\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch"
Jun 02 16:23:57.514575 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:23:57.514479    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f51886f-ad3f-46d0-95a1-88cfc5d975cc-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch\" (UID: \"0f51886f-ad3f-46d0-95a1-88cfc5d975cc\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch"
Jun 02 16:23:57.615702 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:23:57.615613    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mjgn2\" (UniqueName: \"kubernetes.io/projected/0f51886f-ad3f-46d0-95a1-88cfc5d975cc-kube-api-access-mjgn2\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch\" (UID: \"0f51886f-ad3f-46d0-95a1-88cfc5d975cc\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch"
Jun 02 16:23:57.615702 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:23:57.615663    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f51886f-ad3f-46d0-95a1-88cfc5d975cc-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch\" (UID: \"0f51886f-ad3f-46d0-95a1-88cfc5d975cc\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch"
Jun 02 16:23:57.615908 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:23:57.615707    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f51886f-ad3f-46d0-95a1-88cfc5d975cc-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch\" (UID: \"0f51886f-ad3f-46d0-95a1-88cfc5d975cc\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch"
Jun 02 16:23:57.615908 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:23:57.615734    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f51886f-ad3f-46d0-95a1-88cfc5d975cc-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch\" (UID: \"0f51886f-ad3f-46d0-95a1-88cfc5d975cc\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch"
Jun 02 16:23:57.615908 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:23:57.615870    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-serving-cert: secret "isvc-pmml-v2-kserve-predictor-serving-cert" not found
Jun 02 16:23:57.616061 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:23:57.615936    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f51886f-ad3f-46d0-95a1-88cfc5d975cc-proxy-tls podName:0f51886f-ad3f-46d0-95a1-88cfc5d975cc nodeName:}" failed. No retries permitted until 2026-06-02 16:23:58.11591658 +0000 UTC m=+2246.641824989 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0f51886f-ad3f-46d0-95a1-88cfc5d975cc-proxy-tls") pod "isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch" (UID: "0f51886f-ad3f-46d0-95a1-88cfc5d975cc") : secret "isvc-pmml-v2-kserve-predictor-serving-cert" not found
Jun 02 16:23:57.616127 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:23:57.616096    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f51886f-ad3f-46d0-95a1-88cfc5d975cc-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch\" (UID: \"0f51886f-ad3f-46d0-95a1-88cfc5d975cc\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch"
Jun 02 16:23:57.616493 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:23:57.616469    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f51886f-ad3f-46d0-95a1-88cfc5d975cc-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch\" (UID: \"0f51886f-ad3f-46d0-95a1-88cfc5d975cc\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch"
Jun 02 16:23:57.625142 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:23:57.625116    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjgn2\" (UniqueName: \"kubernetes.io/projected/0f51886f-ad3f-46d0-95a1-88cfc5d975cc-kube-api-access-mjgn2\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch\" (UID: \"0f51886f-ad3f-46d0-95a1-88cfc5d975cc\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch"
Jun 02 16:23:58.120723 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:23:58.120685    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f51886f-ad3f-46d0-95a1-88cfc5d975cc-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch\" (UID: \"0f51886f-ad3f-46d0-95a1-88cfc5d975cc\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch"
Jun 02 16:23:58.123369 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:23:58.123344    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f51886f-ad3f-46d0-95a1-88cfc5d975cc-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch\" (UID: \"0f51886f-ad3f-46d0-95a1-88cfc5d975cc\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch"
Jun 02 16:23:58.298145 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:23:58.298105    2579 generic.go:358] "Generic (PLEG): container finished" podID="883a6f75-9aaf-45a1-b230-5a7e3aec70c9" containerID="948de2d57cc61b5ccfce7047a6b87d69df72b9263fff9a8785014472ac5e246f" exitCode=2
Jun 02 16:23:58.298327 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:23:58.298177    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp" event={"ID":"883a6f75-9aaf-45a1-b230-5a7e3aec70c9","Type":"ContainerDied","Data":"948de2d57cc61b5ccfce7047a6b87d69df72b9263fff9a8785014472ac5e246f"}
Jun 02 16:23:58.349916 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:23:58.349860    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch"
Jun 02 16:23:58.478061 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:23:58.478034    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch"]
Jun 02 16:23:58.480703 ip-10-0-129-156 kubenswrapper[2579]: W0602 16:23:58.480675    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f51886f_ad3f_46d0_95a1_88cfc5d975cc.slice/crio-424a0dd81306ace1ddeae2f3f4adbd9570de6c5eca37fc262a06524c549936b9 WatchSource:0}: Error finding container 424a0dd81306ace1ddeae2f3f4adbd9570de6c5eca37fc262a06524c549936b9: Status 404 returned error can't find the container with id 424a0dd81306ace1ddeae2f3f4adbd9570de6c5eca37fc262a06524c549936b9
Jun 02 16:23:59.045435 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:23:59.045366    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp" podUID="883a6f75-9aaf-45a1-b230-5a7e3aec70c9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.20:8643/healthz\": dial tcp 10.133.0.20:8643: connect: connection refused"
Jun 02 16:23:59.049766 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:23:59.049733    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp" podUID="883a6f75-9aaf-45a1-b230-5a7e3aec70c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jun 02 16:23:59.302459 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:23:59.302330    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch" event={"ID":"0f51886f-ad3f-46d0-95a1-88cfc5d975cc","Type":"ContainerStarted","Data":"7d0e8bba4095f52f85fbb872c40cc5b3755d95711d3704aa39eb1dd65c9e6fd2"}
Jun 02 16:23:59.302459 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:23:59.302370    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch" event={"ID":"0f51886f-ad3f-46d0-95a1-88cfc5d975cc","Type":"ContainerStarted","Data":"424a0dd81306ace1ddeae2f3f4adbd9570de6c5eca37fc262a06524c549936b9"}
Jun 02 16:24:01.090174 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:01.090149    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp"
Jun 02 16:24:01.143631 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:01.143549    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/883a6f75-9aaf-45a1-b230-5a7e3aec70c9-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") pod \"883a6f75-9aaf-45a1-b230-5a7e3aec70c9\" (UID: \"883a6f75-9aaf-45a1-b230-5a7e3aec70c9\") "
Jun 02 16:24:01.143631 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:01.143592    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/883a6f75-9aaf-45a1-b230-5a7e3aec70c9-proxy-tls\") pod \"883a6f75-9aaf-45a1-b230-5a7e3aec70c9\" (UID: \"883a6f75-9aaf-45a1-b230-5a7e3aec70c9\") "
Jun 02 16:24:01.143631 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:01.143615    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54plx\" (UniqueName: \"kubernetes.io/projected/883a6f75-9aaf-45a1-b230-5a7e3aec70c9-kube-api-access-54plx\") pod \"883a6f75-9aaf-45a1-b230-5a7e3aec70c9\" (UID: \"883a6f75-9aaf-45a1-b230-5a7e3aec70c9\") "
Jun 02 16:24:01.143877 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:01.143656    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/883a6f75-9aaf-45a1-b230-5a7e3aec70c9-kserve-provision-location\") pod \"883a6f75-9aaf-45a1-b230-5a7e3aec70c9\" (UID: \"883a6f75-9aaf-45a1-b230-5a7e3aec70c9\") "
Jun 02 16:24:01.143973 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:01.143925    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/883a6f75-9aaf-45a1-b230-5a7e3aec70c9-isvc-pmml-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-pmml-runtime-kube-rbac-proxy-sar-config") pod "883a6f75-9aaf-45a1-b230-5a7e3aec70c9" (UID: "883a6f75-9aaf-45a1-b230-5a7e3aec70c9"). InnerVolumeSpecName "isvc-pmml-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:24:01.144091 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:01.144055    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/883a6f75-9aaf-45a1-b230-5a7e3aec70c9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "883a6f75-9aaf-45a1-b230-5a7e3aec70c9" (UID: "883a6f75-9aaf-45a1-b230-5a7e3aec70c9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:24:01.145845 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:01.145825    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/883a6f75-9aaf-45a1-b230-5a7e3aec70c9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "883a6f75-9aaf-45a1-b230-5a7e3aec70c9" (UID: "883a6f75-9aaf-45a1-b230-5a7e3aec70c9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:24:01.145914 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:01.145883    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/883a6f75-9aaf-45a1-b230-5a7e3aec70c9-kube-api-access-54plx" (OuterVolumeSpecName: "kube-api-access-54plx") pod "883a6f75-9aaf-45a1-b230-5a7e3aec70c9" (UID: "883a6f75-9aaf-45a1-b230-5a7e3aec70c9"). InnerVolumeSpecName "kube-api-access-54plx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:24:01.249281 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:01.245138    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/883a6f75-9aaf-45a1-b230-5a7e3aec70c9-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:24:01.249281 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:01.245182    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/883a6f75-9aaf-45a1-b230-5a7e3aec70c9-proxy-tls\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:24:01.249281 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:01.245209    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54plx\" (UniqueName: \"kubernetes.io/projected/883a6f75-9aaf-45a1-b230-5a7e3aec70c9-kube-api-access-54plx\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:24:01.249281 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:01.245271    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/883a6f75-9aaf-45a1-b230-5a7e3aec70c9-kserve-provision-location\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:24:01.310117 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:01.310080    2579 generic.go:358] "Generic (PLEG): container finished" podID="883a6f75-9aaf-45a1-b230-5a7e3aec70c9" containerID="f8d61cce540bd06befe653bb93cba522eacd153a5c9e74b238a3009cebf44de7" exitCode=0
Jun 02 16:24:01.310271 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:01.310116    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp" event={"ID":"883a6f75-9aaf-45a1-b230-5a7e3aec70c9","Type":"ContainerDied","Data":"f8d61cce540bd06befe653bb93cba522eacd153a5c9e74b238a3009cebf44de7"}
Jun 02 16:24:01.310271 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:01.310167    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp" event={"ID":"883a6f75-9aaf-45a1-b230-5a7e3aec70c9","Type":"ContainerDied","Data":"8e0598b5dabc6def92d70d9fd04535bc885f1dd9262970a21012aeb3164a4521"}
Jun 02 16:24:01.310271 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:01.310190    2579 scope.go:117] "RemoveContainer" containerID="948de2d57cc61b5ccfce7047a6b87d69df72b9263fff9a8785014472ac5e246f"
Jun 02 16:24:01.310271 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:01.310216    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp"
Jun 02 16:24:01.318949 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:01.318931    2579 scope.go:117] "RemoveContainer" containerID="f8d61cce540bd06befe653bb93cba522eacd153a5c9e74b238a3009cebf44de7"
Jun 02 16:24:01.326054 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:01.326037    2579 scope.go:117] "RemoveContainer" containerID="ac6afb169407110d99f9277a6d81065d5dd76f2d238d79471f2c1f3f2f937d93"
Jun 02 16:24:01.332504 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:01.332480    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp"]
Jun 02 16:24:01.333630 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:01.333605    2579 scope.go:117] "RemoveContainer" containerID="948de2d57cc61b5ccfce7047a6b87d69df72b9263fff9a8785014472ac5e246f"
Jun 02 16:24:01.333920 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:24:01.333901    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"948de2d57cc61b5ccfce7047a6b87d69df72b9263fff9a8785014472ac5e246f\": container with ID starting with 948de2d57cc61b5ccfce7047a6b87d69df72b9263fff9a8785014472ac5e246f not found: ID does not exist" containerID="948de2d57cc61b5ccfce7047a6b87d69df72b9263fff9a8785014472ac5e246f"
Jun 02 16:24:01.333980 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:01.333929    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"948de2d57cc61b5ccfce7047a6b87d69df72b9263fff9a8785014472ac5e246f"} err="failed to get container status \"948de2d57cc61b5ccfce7047a6b87d69df72b9263fff9a8785014472ac5e246f\": rpc error: code = NotFound desc = could not find container \"948de2d57cc61b5ccfce7047a6b87d69df72b9263fff9a8785014472ac5e246f\": container with ID starting with 948de2d57cc61b5ccfce7047a6b87d69df72b9263fff9a8785014472ac5e246f not found: ID does not exist"
Jun 02 16:24:01.333980 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:01.333949    2579 scope.go:117] "RemoveContainer" containerID="f8d61cce540bd06befe653bb93cba522eacd153a5c9e74b238a3009cebf44de7"
Jun 02 16:24:01.334194 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:24:01.334177    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8d61cce540bd06befe653bb93cba522eacd153a5c9e74b238a3009cebf44de7\": container with ID starting with f8d61cce540bd06befe653bb93cba522eacd153a5c9e74b238a3009cebf44de7 not found: ID does not exist" containerID="f8d61cce540bd06befe653bb93cba522eacd153a5c9e74b238a3009cebf44de7"
Jun 02 16:24:01.334237 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:01.334201    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8d61cce540bd06befe653bb93cba522eacd153a5c9e74b238a3009cebf44de7"} err="failed to get container status \"f8d61cce540bd06befe653bb93cba522eacd153a5c9e74b238a3009cebf44de7\": rpc error: code = NotFound desc = could not find container \"f8d61cce540bd06befe653bb93cba522eacd153a5c9e74b238a3009cebf44de7\": container with ID starting with f8d61cce540bd06befe653bb93cba522eacd153a5c9e74b238a3009cebf44de7 not found: ID does not exist"
Jun 02 16:24:01.334237 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:01.334217    2579 scope.go:117] "RemoveContainer" containerID="ac6afb169407110d99f9277a6d81065d5dd76f2d238d79471f2c1f3f2f937d93"
Jun 02 16:24:01.334478 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:24:01.334458    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac6afb169407110d99f9277a6d81065d5dd76f2d238d79471f2c1f3f2f937d93\": container with ID starting with ac6afb169407110d99f9277a6d81065d5dd76f2d238d79471f2c1f3f2f937d93 not found: ID does not exist" containerID="ac6afb169407110d99f9277a6d81065d5dd76f2d238d79471f2c1f3f2f937d93"
Jun 02 16:24:01.334563 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:01.334483    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac6afb169407110d99f9277a6d81065d5dd76f2d238d79471f2c1f3f2f937d93"} err="failed to get container status \"ac6afb169407110d99f9277a6d81065d5dd76f2d238d79471f2c1f3f2f937d93\": rpc error: code = NotFound desc = could not find container \"ac6afb169407110d99f9277a6d81065d5dd76f2d238d79471f2c1f3f2f937d93\": container with ID starting with ac6afb169407110d99f9277a6d81065d5dd76f2d238d79471f2c1f3f2f937d93 not found: ID does not exist"
Jun 02 16:24:01.336796 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:01.336768    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ltgcp"]
Jun 02 16:24:02.064766 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:02.064735    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="883a6f75-9aaf-45a1-b230-5a7e3aec70c9" path="/var/lib/kubelet/pods/883a6f75-9aaf-45a1-b230-5a7e3aec70c9/volumes"
Jun 02 16:24:02.315687 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:02.315591    2579 generic.go:358] "Generic (PLEG): container finished" podID="0f51886f-ad3f-46d0-95a1-88cfc5d975cc" containerID="7d0e8bba4095f52f85fbb872c40cc5b3755d95711d3704aa39eb1dd65c9e6fd2" exitCode=0
Jun 02 16:24:02.315687 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:02.315660    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch" event={"ID":"0f51886f-ad3f-46d0-95a1-88cfc5d975cc","Type":"ContainerDied","Data":"7d0e8bba4095f52f85fbb872c40cc5b3755d95711d3704aa39eb1dd65c9e6fd2"}
Jun 02 16:24:03.320640 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:03.320605    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch" event={"ID":"0f51886f-ad3f-46d0-95a1-88cfc5d975cc","Type":"ContainerStarted","Data":"b35a048a43a42ed42a14a4c50bb6d5f078513618849f66510f3f9aa6fd0de114"}
Jun 02 16:24:03.321092 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:03.320649    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch" event={"ID":"0f51886f-ad3f-46d0-95a1-88cfc5d975cc","Type":"ContainerStarted","Data":"11176f3daddd91f51fc56c0398bdf750c75e355b3fd329ff1a1ba3e295b60be1"}
Jun 02 16:24:03.321092 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:03.320874    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch"
Jun 02 16:24:03.374902 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:03.374842    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch" podStartSLOduration=6.3748235300000005 podStartE2EDuration="6.37482353s" podCreationTimestamp="2026-06-02 16:23:57 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:24:03.373693027 +0000 UTC m=+2251.899601458" watchObservedRunningTime="2026-06-02 16:24:03.37482353 +0000 UTC m=+2251.900731961"
Jun 02 16:24:04.323441 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:04.323387    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch"
Jun 02 16:24:04.324720 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:04.324693    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch" podUID="0f51886f-ad3f-46d0-95a1-88cfc5d975cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jun 02 16:24:05.326070 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:05.326021    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch" podUID="0f51886f-ad3f-46d0-95a1-88cfc5d975cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jun 02 16:24:10.330171 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:10.330141    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch"
Jun 02 16:24:10.330801 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:10.330777    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch" podUID="0f51886f-ad3f-46d0-95a1-88cfc5d975cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jun 02 16:24:20.330789 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:20.330749    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch" podUID="0f51886f-ad3f-46d0-95a1-88cfc5d975cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jun 02 16:24:30.331469 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:30.331426    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch" podUID="0f51886f-ad3f-46d0-95a1-88cfc5d975cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jun 02 16:24:40.330847 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:40.330799    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch" podUID="0f51886f-ad3f-46d0-95a1-88cfc5d975cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jun 02 16:24:50.331379 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:24:50.331338    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch" podUID="0f51886f-ad3f-46d0-95a1-88cfc5d975cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jun 02 16:25:00.331068 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:25:00.331031    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch" podUID="0f51886f-ad3f-46d0-95a1-88cfc5d975cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jun 02 16:25:10.331556 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:25:10.331516    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch" podUID="0f51886f-ad3f-46d0-95a1-88cfc5d975cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jun 02 16:25:20.331177 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:25:20.331092    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch" podUID="0f51886f-ad3f-46d0-95a1-88cfc5d975cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jun 02 16:25:22.061940 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:25:22.061899    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch" podUID="0f51886f-ad3f-46d0-95a1-88cfc5d975cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jun 02 16:25:32.063138 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:25:32.063107    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch"
Jun 02 16:25:38.641389 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:25:38.641351    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch"]
Jun 02 16:25:38.641809 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:25:38.641676    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch" podUID="0f51886f-ad3f-46d0-95a1-88cfc5d975cc" containerName="kserve-container" containerID="cri-o://11176f3daddd91f51fc56c0398bdf750c75e355b3fd329ff1a1ba3e295b60be1" gracePeriod=30
Jun 02 16:25:38.641809 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:25:38.641735    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch" podUID="0f51886f-ad3f-46d0-95a1-88cfc5d975cc" containerName="kube-rbac-proxy" containerID="cri-o://b35a048a43a42ed42a14a4c50bb6d5f078513618849f66510f3f9aa6fd0de114" gracePeriod=30
Jun 02 16:25:39.584439 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:25:39.584382    2579 generic.go:358] "Generic (PLEG): container finished" podID="0f51886f-ad3f-46d0-95a1-88cfc5d975cc" containerID="b35a048a43a42ed42a14a4c50bb6d5f078513618849f66510f3f9aa6fd0de114" exitCode=2
Jun 02 16:25:39.584624 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:25:39.584464    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch" event={"ID":"0f51886f-ad3f-46d0-95a1-88cfc5d975cc","Type":"ContainerDied","Data":"b35a048a43a42ed42a14a4c50bb6d5f078513618849f66510f3f9aa6fd0de114"}
Jun 02 16:25:40.326218 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:25:40.326172    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch" podUID="0f51886f-ad3f-46d0-95a1-88cfc5d975cc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
Jun 02 16:25:42.062063 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:25:42.062018    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch" podUID="0f51886f-ad3f-46d0-95a1-88cfc5d975cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jun 02 16:25:42.594354 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:25:42.594321    2579 generic.go:358] "Generic (PLEG): container finished" podID="0f51886f-ad3f-46d0-95a1-88cfc5d975cc" containerID="11176f3daddd91f51fc56c0398bdf750c75e355b3fd329ff1a1ba3e295b60be1" exitCode=0
Jun 02 16:25:42.594531 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:25:42.594389    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch" event={"ID":"0f51886f-ad3f-46d0-95a1-88cfc5d975cc","Type":"ContainerDied","Data":"11176f3daddd91f51fc56c0398bdf750c75e355b3fd329ff1a1ba3e295b60be1"}
Jun 02 16:25:42.683788 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:25:42.683767    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch"
Jun 02 16:25:42.862641 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:25:42.862539    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjgn2\" (UniqueName: \"kubernetes.io/projected/0f51886f-ad3f-46d0-95a1-88cfc5d975cc-kube-api-access-mjgn2\") pod \"0f51886f-ad3f-46d0-95a1-88cfc5d975cc\" (UID: \"0f51886f-ad3f-46d0-95a1-88cfc5d975cc\") "
Jun 02 16:25:42.862641 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:25:42.862589    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f51886f-ad3f-46d0-95a1-88cfc5d975cc-proxy-tls\") pod \"0f51886f-ad3f-46d0-95a1-88cfc5d975cc\" (UID: \"0f51886f-ad3f-46d0-95a1-88cfc5d975cc\") "
Jun 02 16:25:42.862641 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:25:42.862628    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f51886f-ad3f-46d0-95a1-88cfc5d975cc-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") pod \"0f51886f-ad3f-46d0-95a1-88cfc5d975cc\" (UID: \"0f51886f-ad3f-46d0-95a1-88cfc5d975cc\") "
Jun 02 16:25:42.862933 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:25:42.862669    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f51886f-ad3f-46d0-95a1-88cfc5d975cc-kserve-provision-location\") pod \"0f51886f-ad3f-46d0-95a1-88cfc5d975cc\" (UID: \"0f51886f-ad3f-46d0-95a1-88cfc5d975cc\") "
Jun 02 16:25:42.863018 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:25:42.862990    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f51886f-ad3f-46d0-95a1-88cfc5d975cc-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config") pod "0f51886f-ad3f-46d0-95a1-88cfc5d975cc" (UID: "0f51886f-ad3f-46d0-95a1-88cfc5d975cc"). InnerVolumeSpecName "isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:25:42.863072 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:25:42.863028    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f51886f-ad3f-46d0-95a1-88cfc5d975cc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0f51886f-ad3f-46d0-95a1-88cfc5d975cc" (UID: "0f51886f-ad3f-46d0-95a1-88cfc5d975cc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:25:42.864875 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:25:42.864844    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f51886f-ad3f-46d0-95a1-88cfc5d975cc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0f51886f-ad3f-46d0-95a1-88cfc5d975cc" (UID: "0f51886f-ad3f-46d0-95a1-88cfc5d975cc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:25:42.864875 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:25:42.864856    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f51886f-ad3f-46d0-95a1-88cfc5d975cc-kube-api-access-mjgn2" (OuterVolumeSpecName: "kube-api-access-mjgn2") pod "0f51886f-ad3f-46d0-95a1-88cfc5d975cc" (UID: "0f51886f-ad3f-46d0-95a1-88cfc5d975cc"). InnerVolumeSpecName "kube-api-access-mjgn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:25:42.964202 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:25:42.964154    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f51886f-ad3f-46d0-95a1-88cfc5d975cc-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:25:42.964202 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:25:42.964206    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f51886f-ad3f-46d0-95a1-88cfc5d975cc-kserve-provision-location\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:25:42.964484 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:25:42.964232    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjgn2\" (UniqueName: \"kubernetes.io/projected/0f51886f-ad3f-46d0-95a1-88cfc5d975cc-kube-api-access-mjgn2\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:25:42.964484 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:25:42.964247    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f51886f-ad3f-46d0-95a1-88cfc5d975cc-proxy-tls\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:25:43.598477 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:25:43.598439    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch" event={"ID":"0f51886f-ad3f-46d0-95a1-88cfc5d975cc","Type":"ContainerDied","Data":"424a0dd81306ace1ddeae2f3f4adbd9570de6c5eca37fc262a06524c549936b9"}
Jun 02 16:25:43.598953 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:25:43.598493    2579 scope.go:117] "RemoveContainer" containerID="b35a048a43a42ed42a14a4c50bb6d5f078513618849f66510f3f9aa6fd0de114"
Jun 02 16:25:43.598953 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:25:43.598519    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch"
Jun 02 16:25:43.606958 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:25:43.606939    2579 scope.go:117] "RemoveContainer" containerID="11176f3daddd91f51fc56c0398bdf750c75e355b3fd329ff1a1ba3e295b60be1"
Jun 02 16:25:43.614581 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:25:43.614563    2579 scope.go:117] "RemoveContainer" containerID="7d0e8bba4095f52f85fbb872c40cc5b3755d95711d3704aa39eb1dd65c9e6fd2"
Jun 02 16:25:43.619592 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:25:43.619562    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch"]
Jun 02 16:25:43.623865 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:25:43.623835    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-fzpch"]
Jun 02 16:25:44.063959 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:25:44.063927    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f51886f-ad3f-46d0-95a1-88cfc5d975cc" path="/var/lib/kubelet/pods/0f51886f-ad3f-46d0-95a1-88cfc5d975cc/volumes"
Jun 02 16:26:32.060622 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:26:32.060584    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lqx9k_6127e33c-aa77-437e-b443-e956037afedb/ovn-acl-logging/0.log"
Jun 02 16:26:32.062976 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:26:32.062948    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lqx9k_6127e33c-aa77-437e-b443-e956037afedb/ovn-acl-logging/0.log"
Jun 02 16:27:20.188121 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:20.188084    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4"]
Jun 02 16:27:20.188621 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:20.188332    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f51886f-ad3f-46d0-95a1-88cfc5d975cc" containerName="kserve-container"
Jun 02 16:27:20.188621 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:20.188343    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f51886f-ad3f-46d0-95a1-88cfc5d975cc" containerName="kserve-container"
Jun 02 16:27:20.188621 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:20.188352    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="883a6f75-9aaf-45a1-b230-5a7e3aec70c9" containerName="kube-rbac-proxy"
Jun 02 16:27:20.188621 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:20.188357    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="883a6f75-9aaf-45a1-b230-5a7e3aec70c9" containerName="kube-rbac-proxy"
Jun 02 16:27:20.188621 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:20.188363    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f51886f-ad3f-46d0-95a1-88cfc5d975cc" containerName="kube-rbac-proxy"
Jun 02 16:27:20.188621 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:20.188369    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f51886f-ad3f-46d0-95a1-88cfc5d975cc" containerName="kube-rbac-proxy"
Jun 02 16:27:20.188621 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:20.188384    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="883a6f75-9aaf-45a1-b230-5a7e3aec70c9" containerName="kserve-container"
Jun 02 16:27:20.188621 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:20.188391    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="883a6f75-9aaf-45a1-b230-5a7e3aec70c9" containerName="kserve-container"
Jun 02 16:27:20.188621 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:20.188420    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f51886f-ad3f-46d0-95a1-88cfc5d975cc" containerName="storage-initializer"
Jun 02 16:27:20.188621 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:20.188426    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f51886f-ad3f-46d0-95a1-88cfc5d975cc" containerName="storage-initializer"
Jun 02 16:27:20.188621 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:20.188434    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="883a6f75-9aaf-45a1-b230-5a7e3aec70c9" containerName="storage-initializer"
Jun 02 16:27:20.188621 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:20.188439    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="883a6f75-9aaf-45a1-b230-5a7e3aec70c9" containerName="storage-initializer"
Jun 02 16:27:20.188621 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:20.188477    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="0f51886f-ad3f-46d0-95a1-88cfc5d975cc" containerName="kserve-container"
Jun 02 16:27:20.188621 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:20.188484    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="0f51886f-ad3f-46d0-95a1-88cfc5d975cc" containerName="kube-rbac-proxy"
Jun 02 16:27:20.188621 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:20.188491    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="883a6f75-9aaf-45a1-b230-5a7e3aec70c9" containerName="kube-rbac-proxy"
Jun 02 16:27:20.188621 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:20.188497    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="883a6f75-9aaf-45a1-b230-5a7e3aec70c9" containerName="kserve-container"
Jun 02 16:27:20.191412 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:20.191386    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4"
Jun 02 16:27:20.193798 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:20.193767    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Jun 02 16:27:20.193952 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:20.193881    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jun 02 16:27:20.193952 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:20.193877    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Jun 02 16:27:20.194628 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:20.194604    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ztrcr\""
Jun 02 16:27:20.194812 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:20.194685    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jun 02 16:27:20.200840 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:20.200813    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4"]
Jun 02 16:27:20.286219 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:20.286185    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4daf93d8-b90b-47db-b838-59fc667bee46-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4\" (UID: \"4daf93d8-b90b-47db-b838-59fc667bee46\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4"
Jun 02 16:27:20.286375 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:20.286289    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjvnz\" (UniqueName: \"kubernetes.io/projected/4daf93d8-b90b-47db-b838-59fc667bee46-kube-api-access-sjvnz\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4\" (UID: \"4daf93d8-b90b-47db-b838-59fc667bee46\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4"
Jun 02 16:27:20.286375 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:20.286324    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4daf93d8-b90b-47db-b838-59fc667bee46-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4\" (UID: \"4daf93d8-b90b-47db-b838-59fc667bee46\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4"
Jun 02 16:27:20.286375 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:20.286349    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4daf93d8-b90b-47db-b838-59fc667bee46-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4\" (UID: \"4daf93d8-b90b-47db-b838-59fc667bee46\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4"
Jun 02 16:27:20.387114 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:20.387077    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sjvnz\" (UniqueName: \"kubernetes.io/projected/4daf93d8-b90b-47db-b838-59fc667bee46-kube-api-access-sjvnz\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4\" (UID: \"4daf93d8-b90b-47db-b838-59fc667bee46\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4"
Jun 02 16:27:20.387294 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:20.387137    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4daf93d8-b90b-47db-b838-59fc667bee46-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4\" (UID: \"4daf93d8-b90b-47db-b838-59fc667bee46\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4"
Jun 02 16:27:20.387294 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:20.387170    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4daf93d8-b90b-47db-b838-59fc667bee46-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4\" (UID: \"4daf93d8-b90b-47db-b838-59fc667bee46\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4"
Jun 02 16:27:20.387294 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:20.387214    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4daf93d8-b90b-47db-b838-59fc667bee46-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4\" (UID: \"4daf93d8-b90b-47db-b838-59fc667bee46\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4"
Jun 02 16:27:20.387453 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:27:20.387338    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-serving-cert: secret "isvc-predictive-sklearn-predictor-serving-cert" not found
Jun 02 16:27:20.387453 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:27:20.387449    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4daf93d8-b90b-47db-b838-59fc667bee46-proxy-tls podName:4daf93d8-b90b-47db-b838-59fc667bee46 nodeName:}" failed. No retries permitted until 2026-06-02 16:27:20.887430693 +0000 UTC m=+2449.413339105 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4daf93d8-b90b-47db-b838-59fc667bee46-proxy-tls") pod "isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4" (UID: "4daf93d8-b90b-47db-b838-59fc667bee46") : secret "isvc-predictive-sklearn-predictor-serving-cert" not found
Jun 02 16:27:20.387676 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:20.387654    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4daf93d8-b90b-47db-b838-59fc667bee46-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4\" (UID: \"4daf93d8-b90b-47db-b838-59fc667bee46\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4"
Jun 02 16:27:20.387928 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:20.387912    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4daf93d8-b90b-47db-b838-59fc667bee46-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4\" (UID: \"4daf93d8-b90b-47db-b838-59fc667bee46\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4"
Jun 02 16:27:20.399799 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:20.399764    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjvnz\" (UniqueName: \"kubernetes.io/projected/4daf93d8-b90b-47db-b838-59fc667bee46-kube-api-access-sjvnz\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4\" (UID: \"4daf93d8-b90b-47db-b838-59fc667bee46\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4"
Jun 02 16:27:20.890909 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:20.890859    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4daf93d8-b90b-47db-b838-59fc667bee46-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4\" (UID: \"4daf93d8-b90b-47db-b838-59fc667bee46\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4"
Jun 02 16:27:20.893471 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:20.893443    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4daf93d8-b90b-47db-b838-59fc667bee46-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4\" (UID: \"4daf93d8-b90b-47db-b838-59fc667bee46\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4"
Jun 02 16:27:21.102644 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:21.102602    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4"
Jun 02 16:27:21.233804 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:21.233776    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4"]
Jun 02 16:27:21.236161 ip-10-0-129-156 kubenswrapper[2579]: W0602 16:27:21.236132    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4daf93d8_b90b_47db_b838_59fc667bee46.slice/crio-0dc57fae0f552f50e886f7b68939472b6ae31d6ff25ea94294df287e89f769c5 WatchSource:0}: Error finding container 0dc57fae0f552f50e886f7b68939472b6ae31d6ff25ea94294df287e89f769c5: Status 404 returned error can't find the container with id 0dc57fae0f552f50e886f7b68939472b6ae31d6ff25ea94294df287e89f769c5
Jun 02 16:27:21.238029 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:21.238004    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 16:27:21.869701 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:21.869664    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4" event={"ID":"4daf93d8-b90b-47db-b838-59fc667bee46","Type":"ContainerStarted","Data":"aca9ed6f05094a5528db68af3f718f70e05420280730090567bd38a19b17a0b2"}
Jun 02 16:27:21.869701 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:21.869703    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4" event={"ID":"4daf93d8-b90b-47db-b838-59fc667bee46","Type":"ContainerStarted","Data":"0dc57fae0f552f50e886f7b68939472b6ae31d6ff25ea94294df287e89f769c5"}
Jun 02 16:27:24.878726 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:24.878688    2579 generic.go:358] "Generic (PLEG): container finished" podID="4daf93d8-b90b-47db-b838-59fc667bee46" containerID="aca9ed6f05094a5528db68af3f718f70e05420280730090567bd38a19b17a0b2" exitCode=0
Jun 02 16:27:24.879198 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:24.878740    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4" event={"ID":"4daf93d8-b90b-47db-b838-59fc667bee46","Type":"ContainerDied","Data":"aca9ed6f05094a5528db68af3f718f70e05420280730090567bd38a19b17a0b2"}
Jun 02 16:27:46.948475 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:46.948437    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4" event={"ID":"4daf93d8-b90b-47db-b838-59fc667bee46","Type":"ContainerStarted","Data":"3cc0c842ce28a2ae2eb26fcd3f16e6b001ceacbf4e1776ae41239e4c2ec33b5b"}
Jun 02 16:27:46.948475 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:46.948483    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4" event={"ID":"4daf93d8-b90b-47db-b838-59fc667bee46","Type":"ContainerStarted","Data":"e36ed7648d3bf9119c331a83c9643c55be14dbf9c0ed4666e5f4b7046dfabe7e"}
Jun 02 16:27:46.949020 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:46.948733    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4"
Jun 02 16:27:46.967762 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:46.967684    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4" podStartSLOduration=5.41822859 podStartE2EDuration="26.967666876s" podCreationTimestamp="2026-06-02 16:27:20 +0000 UTC" firstStartedPulling="2026-06-02 16:27:24.880033816 +0000 UTC m=+2453.405942225" lastFinishedPulling="2026-06-02 16:27:46.429472102 +0000 UTC m=+2474.955380511" observedRunningTime="2026-06-02 16:27:46.965881311 +0000 UTC m=+2475.491789743" watchObservedRunningTime="2026-06-02 16:27:46.967666876 +0000 UTC m=+2475.493575308"
Jun 02 16:27:47.951013 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:47.950981    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4"
Jun 02 16:27:47.952355 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:47.952327    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4" podUID="4daf93d8-b90b-47db-b838-59fc667bee46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jun 02 16:27:48.953927 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:48.953886    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4" podUID="4daf93d8-b90b-47db-b838-59fc667bee46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jun 02 16:27:53.958698 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:53.958667    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4"
Jun 02 16:27:53.959205 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:27:53.959179    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4" podUID="4daf93d8-b90b-47db-b838-59fc667bee46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jun 02 16:28:03.959707 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:28:03.959663    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4" podUID="4daf93d8-b90b-47db-b838-59fc667bee46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jun 02 16:28:13.959290 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:28:13.959248    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4" podUID="4daf93d8-b90b-47db-b838-59fc667bee46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jun 02 16:28:23.959921 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:28:23.959837    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4" podUID="4daf93d8-b90b-47db-b838-59fc667bee46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jun 02 16:28:33.959497 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:28:33.959452    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4" podUID="4daf93d8-b90b-47db-b838-59fc667bee46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jun 02 16:28:43.959787 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:28:43.959740    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4" podUID="4daf93d8-b90b-47db-b838-59fc667bee46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jun 02 16:28:53.959505 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:28:53.959455    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4" podUID="4daf93d8-b90b-47db-b838-59fc667bee46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jun 02 16:29:03.959583 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:03.959547    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4"
Jun 02 16:29:10.304092 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:10.304057    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4"]
Jun 02 16:29:10.304688 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:10.304532    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4" podUID="4daf93d8-b90b-47db-b838-59fc667bee46" containerName="kserve-container" containerID="cri-o://e36ed7648d3bf9119c331a83c9643c55be14dbf9c0ed4666e5f4b7046dfabe7e" gracePeriod=30
Jun 02 16:29:10.304688 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:10.304595    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4" podUID="4daf93d8-b90b-47db-b838-59fc667bee46" containerName="kube-rbac-proxy" containerID="cri-o://3cc0c842ce28a2ae2eb26fcd3f16e6b001ceacbf4e1776ae41239e4c2ec33b5b" gracePeriod=30
Jun 02 16:29:10.491758 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:10.491719    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2"]
Jun 02 16:29:10.494992 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:10.494971    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2"
Jun 02 16:29:10.497149 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:10.497127    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Jun 02 16:29:10.497266 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:10.497183    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Jun 02 16:29:10.504295 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:10.504257    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2"]
Jun 02 16:29:10.564833 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:10.564740    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02a0823c-e86c-48ff-927b-6c46a1837930-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2\" (UID: \"02a0823c-e86c-48ff-927b-6c46a1837930\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2"
Jun 02 16:29:10.564833 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:10.564787    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02a0823c-e86c-48ff-927b-6c46a1837930-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2\" (UID: \"02a0823c-e86c-48ff-927b-6c46a1837930\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2"
Jun 02 16:29:10.565039 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:10.564841    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wvrn\" (UniqueName: \"kubernetes.io/projected/02a0823c-e86c-48ff-927b-6c46a1837930-kube-api-access-7wvrn\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2\" (UID: \"02a0823c-e86c-48ff-927b-6c46a1837930\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2"
Jun 02 16:29:10.565039 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:10.564905    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02a0823c-e86c-48ff-927b-6c46a1837930-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2\" (UID: \"02a0823c-e86c-48ff-927b-6c46a1837930\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2"
Jun 02 16:29:10.666227 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:10.666186    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7wvrn\" (UniqueName: \"kubernetes.io/projected/02a0823c-e86c-48ff-927b-6c46a1837930-kube-api-access-7wvrn\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2\" (UID: \"02a0823c-e86c-48ff-927b-6c46a1837930\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2"
Jun 02 16:29:10.666227 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:10.666227    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02a0823c-e86c-48ff-927b-6c46a1837930-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2\" (UID: \"02a0823c-e86c-48ff-927b-6c46a1837930\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2"
Jun 02 16:29:10.666523 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:10.666280    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02a0823c-e86c-48ff-927b-6c46a1837930-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2\" (UID: \"02a0823c-e86c-48ff-927b-6c46a1837930\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2"
Jun 02 16:29:10.666523 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:10.666306    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02a0823c-e86c-48ff-927b-6c46a1837930-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2\" (UID: \"02a0823c-e86c-48ff-927b-6c46a1837930\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2"
Jun 02 16:29:10.666523 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:29:10.666435    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-serving-cert: secret "isvc-predictive-xgboost-predictor-serving-cert" not found
Jun 02 16:29:10.666523 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:29:10.666513    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02a0823c-e86c-48ff-927b-6c46a1837930-proxy-tls podName:02a0823c-e86c-48ff-927b-6c46a1837930 nodeName:}" failed. No retries permitted until 2026-06-02 16:29:11.166495688 +0000 UTC m=+2559.692404097 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/02a0823c-e86c-48ff-927b-6c46a1837930-proxy-tls") pod "isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2" (UID: "02a0823c-e86c-48ff-927b-6c46a1837930") : secret "isvc-predictive-xgboost-predictor-serving-cert" not found
Jun 02 16:29:10.666754 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:10.666735    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02a0823c-e86c-48ff-927b-6c46a1837930-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2\" (UID: \"02a0823c-e86c-48ff-927b-6c46a1837930\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2"
Jun 02 16:29:10.666989 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:10.666971    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02a0823c-e86c-48ff-927b-6c46a1837930-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2\" (UID: \"02a0823c-e86c-48ff-927b-6c46a1837930\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2"
Jun 02 16:29:10.674480 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:10.674450    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wvrn\" (UniqueName: \"kubernetes.io/projected/02a0823c-e86c-48ff-927b-6c46a1837930-kube-api-access-7wvrn\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2\" (UID: \"02a0823c-e86c-48ff-927b-6c46a1837930\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2"
Jun 02 16:29:11.170825 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:11.170773    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02a0823c-e86c-48ff-927b-6c46a1837930-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2\" (UID: \"02a0823c-e86c-48ff-927b-6c46a1837930\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2"
Jun 02 16:29:11.173568 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:11.173536    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02a0823c-e86c-48ff-927b-6c46a1837930-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2\" (UID: \"02a0823c-e86c-48ff-927b-6c46a1837930\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2"
Jun 02 16:29:11.179966 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:11.179936    2579 generic.go:358] "Generic (PLEG): container finished" podID="4daf93d8-b90b-47db-b838-59fc667bee46" containerID="3cc0c842ce28a2ae2eb26fcd3f16e6b001ceacbf4e1776ae41239e4c2ec33b5b" exitCode=2
Jun 02 16:29:11.180079 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:11.180010    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4" event={"ID":"4daf93d8-b90b-47db-b838-59fc667bee46","Type":"ContainerDied","Data":"3cc0c842ce28a2ae2eb26fcd3f16e6b001ceacbf4e1776ae41239e4c2ec33b5b"}
Jun 02 16:29:11.405647 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:11.405605    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2"
Jun 02 16:29:11.535152 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:11.535124    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2"]
Jun 02 16:29:11.537162 ip-10-0-129-156 kubenswrapper[2579]: W0602 16:29:11.537135    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02a0823c_e86c_48ff_927b_6c46a1837930.slice/crio-afc15790c7e8a8930e7f63d05db49659e5159c9a30065e798f9d71b78806015a WatchSource:0}: Error finding container afc15790c7e8a8930e7f63d05db49659e5159c9a30065e798f9d71b78806015a: Status 404 returned error can't find the container with id afc15790c7e8a8930e7f63d05db49659e5159c9a30065e798f9d71b78806015a
Jun 02 16:29:12.183713 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:12.183679    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2" event={"ID":"02a0823c-e86c-48ff-927b-6c46a1837930","Type":"ContainerStarted","Data":"cc9f63c172855993a3f3d812c41f347b9838e24f572460f3659679d2dfc2e8f9"}
Jun 02 16:29:12.183713 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:12.183717    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2" event={"ID":"02a0823c-e86c-48ff-927b-6c46a1837930","Type":"ContainerStarted","Data":"afc15790c7e8a8930e7f63d05db49659e5159c9a30065e798f9d71b78806015a"}
Jun 02 16:29:13.954161 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:13.954116    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4" podUID="4daf93d8-b90b-47db-b838-59fc667bee46" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Jun 02 16:29:13.959538 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:13.959513    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4" podUID="4daf93d8-b90b-47db-b838-59fc667bee46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jun 02 16:29:15.649882 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:15.649857    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4"
Jun 02 16:29:15.811397 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:15.811308    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4daf93d8-b90b-47db-b838-59fc667bee46-kserve-provision-location\") pod \"4daf93d8-b90b-47db-b838-59fc667bee46\" (UID: \"4daf93d8-b90b-47db-b838-59fc667bee46\") "
Jun 02 16:29:15.811397 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:15.811380    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4daf93d8-b90b-47db-b838-59fc667bee46-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") pod \"4daf93d8-b90b-47db-b838-59fc667bee46\" (UID: \"4daf93d8-b90b-47db-b838-59fc667bee46\") "
Jun 02 16:29:15.811671 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:15.811455    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjvnz\" (UniqueName: \"kubernetes.io/projected/4daf93d8-b90b-47db-b838-59fc667bee46-kube-api-access-sjvnz\") pod \"4daf93d8-b90b-47db-b838-59fc667bee46\" (UID: \"4daf93d8-b90b-47db-b838-59fc667bee46\") "
Jun 02 16:29:15.811671 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:15.811506    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4daf93d8-b90b-47db-b838-59fc667bee46-proxy-tls\") pod \"4daf93d8-b90b-47db-b838-59fc667bee46\" (UID: \"4daf93d8-b90b-47db-b838-59fc667bee46\") "
Jun 02 16:29:15.811776 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:15.811718    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4daf93d8-b90b-47db-b838-59fc667bee46-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4daf93d8-b90b-47db-b838-59fc667bee46" (UID: "4daf93d8-b90b-47db-b838-59fc667bee46"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:29:15.811832 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:15.811769    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4daf93d8-b90b-47db-b838-59fc667bee46-isvc-predictive-sklearn-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-sklearn-kube-rbac-proxy-sar-config") pod "4daf93d8-b90b-47db-b838-59fc667bee46" (UID: "4daf93d8-b90b-47db-b838-59fc667bee46"). InnerVolumeSpecName "isvc-predictive-sklearn-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:29:15.813674 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:15.813650    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4daf93d8-b90b-47db-b838-59fc667bee46-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4daf93d8-b90b-47db-b838-59fc667bee46" (UID: "4daf93d8-b90b-47db-b838-59fc667bee46"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:29:15.813759 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:15.813711    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4daf93d8-b90b-47db-b838-59fc667bee46-kube-api-access-sjvnz" (OuterVolumeSpecName: "kube-api-access-sjvnz") pod "4daf93d8-b90b-47db-b838-59fc667bee46" (UID: "4daf93d8-b90b-47db-b838-59fc667bee46"). InnerVolumeSpecName "kube-api-access-sjvnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:29:15.912599 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:15.912548    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4daf93d8-b90b-47db-b838-59fc667bee46-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:29:15.912599 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:15.912594    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjvnz\" (UniqueName: \"kubernetes.io/projected/4daf93d8-b90b-47db-b838-59fc667bee46-kube-api-access-sjvnz\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:29:15.912599 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:15.912604    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4daf93d8-b90b-47db-b838-59fc667bee46-proxy-tls\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:29:15.912599 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:15.912615    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4daf93d8-b90b-47db-b838-59fc667bee46-kserve-provision-location\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:29:16.195002 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:16.194966    2579 generic.go:358] "Generic (PLEG): container finished" podID="02a0823c-e86c-48ff-927b-6c46a1837930" containerID="cc9f63c172855993a3f3d812c41f347b9838e24f572460f3659679d2dfc2e8f9" exitCode=0
Jun 02 16:29:16.195199 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:16.195048    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2" event={"ID":"02a0823c-e86c-48ff-927b-6c46a1837930","Type":"ContainerDied","Data":"cc9f63c172855993a3f3d812c41f347b9838e24f572460f3659679d2dfc2e8f9"}
Jun 02 16:29:16.196792 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:16.196764    2579 generic.go:358] "Generic (PLEG): container finished" podID="4daf93d8-b90b-47db-b838-59fc667bee46" containerID="e36ed7648d3bf9119c331a83c9643c55be14dbf9c0ed4666e5f4b7046dfabe7e" exitCode=0
Jun 02 16:29:16.196968 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:16.196798    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4" event={"ID":"4daf93d8-b90b-47db-b838-59fc667bee46","Type":"ContainerDied","Data":"e36ed7648d3bf9119c331a83c9643c55be14dbf9c0ed4666e5f4b7046dfabe7e"}
Jun 02 16:29:16.196968 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:16.196838    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4" event={"ID":"4daf93d8-b90b-47db-b838-59fc667bee46","Type":"ContainerDied","Data":"0dc57fae0f552f50e886f7b68939472b6ae31d6ff25ea94294df287e89f769c5"}
Jun 02 16:29:16.196968 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:16.196846    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4"
Jun 02 16:29:16.196968 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:16.196860    2579 scope.go:117] "RemoveContainer" containerID="3cc0c842ce28a2ae2eb26fcd3f16e6b001ceacbf4e1776ae41239e4c2ec33b5b"
Jun 02 16:29:16.204930 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:16.204911    2579 scope.go:117] "RemoveContainer" containerID="e36ed7648d3bf9119c331a83c9643c55be14dbf9c0ed4666e5f4b7046dfabe7e"
Jun 02 16:29:16.213654 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:16.213632    2579 scope.go:117] "RemoveContainer" containerID="aca9ed6f05094a5528db68af3f718f70e05420280730090567bd38a19b17a0b2"
Jun 02 16:29:16.222444 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:16.222427    2579 scope.go:117] "RemoveContainer" containerID="3cc0c842ce28a2ae2eb26fcd3f16e6b001ceacbf4e1776ae41239e4c2ec33b5b"
Jun 02 16:29:16.222718 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:29:16.222696    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cc0c842ce28a2ae2eb26fcd3f16e6b001ceacbf4e1776ae41239e4c2ec33b5b\": container with ID starting with 3cc0c842ce28a2ae2eb26fcd3f16e6b001ceacbf4e1776ae41239e4c2ec33b5b not found: ID does not exist" containerID="3cc0c842ce28a2ae2eb26fcd3f16e6b001ceacbf4e1776ae41239e4c2ec33b5b"
Jun 02 16:29:16.222765 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:16.222728    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cc0c842ce28a2ae2eb26fcd3f16e6b001ceacbf4e1776ae41239e4c2ec33b5b"} err="failed to get container status \"3cc0c842ce28a2ae2eb26fcd3f16e6b001ceacbf4e1776ae41239e4c2ec33b5b\": rpc error: code = NotFound desc = could not find container \"3cc0c842ce28a2ae2eb26fcd3f16e6b001ceacbf4e1776ae41239e4c2ec33b5b\": container with ID starting with 3cc0c842ce28a2ae2eb26fcd3f16e6b001ceacbf4e1776ae41239e4c2ec33b5b not found: ID does not exist"
Jun 02 16:29:16.222765 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:16.222745    2579 scope.go:117] "RemoveContainer" containerID="e36ed7648d3bf9119c331a83c9643c55be14dbf9c0ed4666e5f4b7046dfabe7e"
Jun 02 16:29:16.223004 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:29:16.222984    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e36ed7648d3bf9119c331a83c9643c55be14dbf9c0ed4666e5f4b7046dfabe7e\": container with ID starting with e36ed7648d3bf9119c331a83c9643c55be14dbf9c0ed4666e5f4b7046dfabe7e not found: ID does not exist" containerID="e36ed7648d3bf9119c331a83c9643c55be14dbf9c0ed4666e5f4b7046dfabe7e"
Jun 02 16:29:16.223044 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:16.223012    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e36ed7648d3bf9119c331a83c9643c55be14dbf9c0ed4666e5f4b7046dfabe7e"} err="failed to get container status \"e36ed7648d3bf9119c331a83c9643c55be14dbf9c0ed4666e5f4b7046dfabe7e\": rpc error: code = NotFound desc = could not find container \"e36ed7648d3bf9119c331a83c9643c55be14dbf9c0ed4666e5f4b7046dfabe7e\": container with ID starting with e36ed7648d3bf9119c331a83c9643c55be14dbf9c0ed4666e5f4b7046dfabe7e not found: ID does not exist"
Jun 02 16:29:16.223044 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:16.223030    2579 scope.go:117] "RemoveContainer" containerID="aca9ed6f05094a5528db68af3f718f70e05420280730090567bd38a19b17a0b2"
Jun 02 16:29:16.223277 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:29:16.223254    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aca9ed6f05094a5528db68af3f718f70e05420280730090567bd38a19b17a0b2\": container with ID starting with aca9ed6f05094a5528db68af3f718f70e05420280730090567bd38a19b17a0b2 not found: ID does not exist" containerID="aca9ed6f05094a5528db68af3f718f70e05420280730090567bd38a19b17a0b2"
Jun 02 16:29:16.223312 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:16.223282    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aca9ed6f05094a5528db68af3f718f70e05420280730090567bd38a19b17a0b2"} err="failed to get container status \"aca9ed6f05094a5528db68af3f718f70e05420280730090567bd38a19b17a0b2\": rpc error: code = NotFound desc = could not find container \"aca9ed6f05094a5528db68af3f718f70e05420280730090567bd38a19b17a0b2\": container with ID starting with aca9ed6f05094a5528db68af3f718f70e05420280730090567bd38a19b17a0b2 not found: ID does not exist"
Jun 02 16:29:16.229846 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:16.229825    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4"]
Jun 02 16:29:16.234438 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:16.234377    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mvcm4"]
Jun 02 16:29:17.201647 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:17.201607    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2" event={"ID":"02a0823c-e86c-48ff-927b-6c46a1837930","Type":"ContainerStarted","Data":"66ccc4f2185f172bd5d9e45d15b7cb61f270dafea19a527b6990ccc034f82a1a"}
Jun 02 16:29:17.201647 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:17.201653    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2" event={"ID":"02a0823c-e86c-48ff-927b-6c46a1837930","Type":"ContainerStarted","Data":"026de9bb2098313bc88264a6baf3e0592bcca34fc2c8d5d348d8ce6d9bf5637c"}
Jun 02 16:29:17.202194 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:17.201931    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2"
Jun 02 16:29:17.222215 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:17.222161    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2" podStartSLOduration=7.222147288 podStartE2EDuration="7.222147288s" podCreationTimestamp="2026-06-02 16:29:10 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:29:17.220903737 +0000 UTC m=+2565.746812168" watchObservedRunningTime="2026-06-02 16:29:17.222147288 +0000 UTC m=+2565.748055718"
Jun 02 16:29:18.064454 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:18.064381    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4daf93d8-b90b-47db-b838-59fc667bee46" path="/var/lib/kubelet/pods/4daf93d8-b90b-47db-b838-59fc667bee46/volumes"
Jun 02 16:29:18.205239 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:18.205211    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2"
Jun 02 16:29:18.206457 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:18.206429    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2" podUID="02a0823c-e86c-48ff-927b-6c46a1837930" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jun 02 16:29:19.208419 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:19.208361    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2" podUID="02a0823c-e86c-48ff-927b-6c46a1837930" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jun 02 16:29:24.212088 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:24.212059    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2"
Jun 02 16:29:24.212552 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:24.212526    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2" podUID="02a0823c-e86c-48ff-927b-6c46a1837930" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jun 02 16:29:34.212601 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:34.212559    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2" podUID="02a0823c-e86c-48ff-927b-6c46a1837930" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jun 02 16:29:44.212684 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:44.212636    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2" podUID="02a0823c-e86c-48ff-927b-6c46a1837930" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jun 02 16:29:54.212956 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:29:54.212871    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2" podUID="02a0823c-e86c-48ff-927b-6c46a1837930" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jun 02 16:30:04.213154 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:04.213109    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2" podUID="02a0823c-e86c-48ff-927b-6c46a1837930" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jun 02 16:30:14.213248 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:14.213206    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2" podUID="02a0823c-e86c-48ff-927b-6c46a1837930" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jun 02 16:30:24.212535 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:24.212494    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2" podUID="02a0823c-e86c-48ff-927b-6c46a1837930" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jun 02 16:30:34.213120 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:34.213088    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2"
Jun 02 16:30:40.534290 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:40.534255    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2"]
Jun 02 16:30:40.534816 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:40.534592    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2" podUID="02a0823c-e86c-48ff-927b-6c46a1837930" containerName="kserve-container" containerID="cri-o://026de9bb2098313bc88264a6baf3e0592bcca34fc2c8d5d348d8ce6d9bf5637c" gracePeriod=30
Jun 02 16:30:40.534816 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:40.534643    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2" podUID="02a0823c-e86c-48ff-927b-6c46a1837930" containerName="kube-rbac-proxy" containerID="cri-o://66ccc4f2185f172bd5d9e45d15b7cb61f270dafea19a527b6990ccc034f82a1a" gracePeriod=30
Jun 02 16:30:40.650212 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:40.650180    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn"]
Jun 02 16:30:40.650518 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:40.650504    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4daf93d8-b90b-47db-b838-59fc667bee46" containerName="storage-initializer"
Jun 02 16:30:40.650577 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:40.650520    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4daf93d8-b90b-47db-b838-59fc667bee46" containerName="storage-initializer"
Jun 02 16:30:40.650577 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:40.650540    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4daf93d8-b90b-47db-b838-59fc667bee46" containerName="kserve-container"
Jun 02 16:30:40.650577 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:40.650545    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4daf93d8-b90b-47db-b838-59fc667bee46" containerName="kserve-container"
Jun 02 16:30:40.650577 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:40.650550    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4daf93d8-b90b-47db-b838-59fc667bee46" containerName="kube-rbac-proxy"
Jun 02 16:30:40.650577 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:40.650556    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4daf93d8-b90b-47db-b838-59fc667bee46" containerName="kube-rbac-proxy"
Jun 02 16:30:40.650729 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:40.650597    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="4daf93d8-b90b-47db-b838-59fc667bee46" containerName="kserve-container"
Jun 02 16:30:40.650729 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:40.650606    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="4daf93d8-b90b-47db-b838-59fc667bee46" containerName="kube-rbac-proxy"
Jun 02 16:30:40.653430 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:40.653413    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn"
Jun 02 16:30:40.655524 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:40.655493    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Jun 02 16:30:40.655644 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:40.655545    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Jun 02 16:30:40.663483 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:40.663457    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn"]
Jun 02 16:30:40.778104 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:40.778067    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsdvr\" (UniqueName: \"kubernetes.io/projected/a361a146-f052-4eae-b388-921853b68269-kube-api-access-rsdvr\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn\" (UID: \"a361a146-f052-4eae-b388-921853b68269\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn"
Jun 02 16:30:40.778291 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:40.778121    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a361a146-f052-4eae-b388-921853b68269-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn\" (UID: \"a361a146-f052-4eae-b388-921853b68269\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn"
Jun 02 16:30:40.778291 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:40.778216    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a361a146-f052-4eae-b388-921853b68269-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn\" (UID: \"a361a146-f052-4eae-b388-921853b68269\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn"
Jun 02 16:30:40.778291 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:40.778247    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a361a146-f052-4eae-b388-921853b68269-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn\" (UID: \"a361a146-f052-4eae-b388-921853b68269\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn"
Jun 02 16:30:40.879047 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:40.878939    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a361a146-f052-4eae-b388-921853b68269-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn\" (UID: \"a361a146-f052-4eae-b388-921853b68269\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn"
Jun 02 16:30:40.879047 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:40.878995    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a361a146-f052-4eae-b388-921853b68269-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn\" (UID: \"a361a146-f052-4eae-b388-921853b68269\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn"
Jun 02 16:30:40.879047 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:40.879045    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rsdvr\" (UniqueName: \"kubernetes.io/projected/a361a146-f052-4eae-b388-921853b68269-kube-api-access-rsdvr\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn\" (UID: \"a361a146-f052-4eae-b388-921853b68269\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn"
Jun 02 16:30:40.879329 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:40.879090    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a361a146-f052-4eae-b388-921853b68269-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn\" (UID: \"a361a146-f052-4eae-b388-921853b68269\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn"
Jun 02 16:30:40.879500 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:40.879472    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a361a146-f052-4eae-b388-921853b68269-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn\" (UID: \"a361a146-f052-4eae-b388-921853b68269\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn"
Jun 02 16:30:40.879807 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:40.879743    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a361a146-f052-4eae-b388-921853b68269-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn\" (UID: \"a361a146-f052-4eae-b388-921853b68269\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn"
Jun 02 16:30:40.881779 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:40.881754    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a361a146-f052-4eae-b388-921853b68269-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn\" (UID: \"a361a146-f052-4eae-b388-921853b68269\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn"
Jun 02 16:30:40.889309 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:40.889287    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsdvr\" (UniqueName: \"kubernetes.io/projected/a361a146-f052-4eae-b388-921853b68269-kube-api-access-rsdvr\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn\" (UID: \"a361a146-f052-4eae-b388-921853b68269\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn"
Jun 02 16:30:40.963831 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:40.963795    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn"
Jun 02 16:30:41.096234 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:41.096200    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn"]
Jun 02 16:30:41.100349 ip-10-0-129-156 kubenswrapper[2579]: W0602 16:30:41.100320    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda361a146_f052_4eae_b388_921853b68269.slice/crio-1b8a4424dac33bd516f60aee37769aaa5269ec561586fc6624d0e811234dcdf8 WatchSource:0}: Error finding container 1b8a4424dac33bd516f60aee37769aaa5269ec561586fc6624d0e811234dcdf8: Status 404 returned error can't find the container with id 1b8a4424dac33bd516f60aee37769aaa5269ec561586fc6624d0e811234dcdf8
Jun 02 16:30:41.436737 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:41.436659    2579 generic.go:358] "Generic (PLEG): container finished" podID="02a0823c-e86c-48ff-927b-6c46a1837930" containerID="66ccc4f2185f172bd5d9e45d15b7cb61f270dafea19a527b6990ccc034f82a1a" exitCode=2
Jun 02 16:30:41.436893 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:41.436733    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2" event={"ID":"02a0823c-e86c-48ff-927b-6c46a1837930","Type":"ContainerDied","Data":"66ccc4f2185f172bd5d9e45d15b7cb61f270dafea19a527b6990ccc034f82a1a"}
Jun 02 16:30:41.438076 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:41.438040    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn" event={"ID":"a361a146-f052-4eae-b388-921853b68269","Type":"ContainerStarted","Data":"8e4d0e77f85d38c3e2c5a578699a237a555c5b7d47441a7cef07d7d630751d3b"}
Jun 02 16:30:41.438076 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:41.438066    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn" event={"ID":"a361a146-f052-4eae-b388-921853b68269","Type":"ContainerStarted","Data":"1b8a4424dac33bd516f60aee37769aaa5269ec561586fc6624d0e811234dcdf8"}
Jun 02 16:30:44.209545 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:44.209491    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2" podUID="02a0823c-e86c-48ff-927b-6c46a1837930" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Jun 02 16:30:44.212823 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:44.212791    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2" podUID="02a0823c-e86c-48ff-927b-6c46a1837930" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jun 02 16:30:45.449555 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:45.449520    2579 generic.go:358] "Generic (PLEG): container finished" podID="a361a146-f052-4eae-b388-921853b68269" containerID="8e4d0e77f85d38c3e2c5a578699a237a555c5b7d47441a7cef07d7d630751d3b" exitCode=0
Jun 02 16:30:45.449993 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:45.449591    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn" event={"ID":"a361a146-f052-4eae-b388-921853b68269","Type":"ContainerDied","Data":"8e4d0e77f85d38c3e2c5a578699a237a555c5b7d47441a7cef07d7d630751d3b"}
Jun 02 16:30:45.884102 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:45.884075    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2"
Jun 02 16:30:46.020333 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:46.020238    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02a0823c-e86c-48ff-927b-6c46a1837930-kserve-provision-location\") pod \"02a0823c-e86c-48ff-927b-6c46a1837930\" (UID: \"02a0823c-e86c-48ff-927b-6c46a1837930\") "
Jun 02 16:30:46.020333 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:46.020311    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02a0823c-e86c-48ff-927b-6c46a1837930-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") pod \"02a0823c-e86c-48ff-927b-6c46a1837930\" (UID: \"02a0823c-e86c-48ff-927b-6c46a1837930\") "
Jun 02 16:30:46.020592 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:46.020372    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wvrn\" (UniqueName: \"kubernetes.io/projected/02a0823c-e86c-48ff-927b-6c46a1837930-kube-api-access-7wvrn\") pod \"02a0823c-e86c-48ff-927b-6c46a1837930\" (UID: \"02a0823c-e86c-48ff-927b-6c46a1837930\") "
Jun 02 16:30:46.020592 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:46.020394    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02a0823c-e86c-48ff-927b-6c46a1837930-proxy-tls\") pod \"02a0823c-e86c-48ff-927b-6c46a1837930\" (UID: \"02a0823c-e86c-48ff-927b-6c46a1837930\") "
Jun 02 16:30:46.020738 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:46.020704    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02a0823c-e86c-48ff-927b-6c46a1837930-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "02a0823c-e86c-48ff-927b-6c46a1837930" (UID: "02a0823c-e86c-48ff-927b-6c46a1837930"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:30:46.020738 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:46.020723    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02a0823c-e86c-48ff-927b-6c46a1837930-isvc-predictive-xgboost-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-xgboost-kube-rbac-proxy-sar-config") pod "02a0823c-e86c-48ff-927b-6c46a1837930" (UID: "02a0823c-e86c-48ff-927b-6c46a1837930"). InnerVolumeSpecName "isvc-predictive-xgboost-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:30:46.022679 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:46.022654    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02a0823c-e86c-48ff-927b-6c46a1837930-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "02a0823c-e86c-48ff-927b-6c46a1837930" (UID: "02a0823c-e86c-48ff-927b-6c46a1837930"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:30:46.022818 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:46.022801    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02a0823c-e86c-48ff-927b-6c46a1837930-kube-api-access-7wvrn" (OuterVolumeSpecName: "kube-api-access-7wvrn") pod "02a0823c-e86c-48ff-927b-6c46a1837930" (UID: "02a0823c-e86c-48ff-927b-6c46a1837930"). InnerVolumeSpecName "kube-api-access-7wvrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:30:46.121476 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:46.121437    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02a0823c-e86c-48ff-927b-6c46a1837930-proxy-tls\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:30:46.121476 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:46.121469    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02a0823c-e86c-48ff-927b-6c46a1837930-kserve-provision-location\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:30:46.121476 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:46.121482    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02a0823c-e86c-48ff-927b-6c46a1837930-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:30:46.121714 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:46.121492    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wvrn\" (UniqueName: \"kubernetes.io/projected/02a0823c-e86c-48ff-927b-6c46a1837930-kube-api-access-7wvrn\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:30:46.453757 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:46.453655    2579 generic.go:358] "Generic (PLEG): container finished" podID="02a0823c-e86c-48ff-927b-6c46a1837930" containerID="026de9bb2098313bc88264a6baf3e0592bcca34fc2c8d5d348d8ce6d9bf5637c" exitCode=0
Jun 02 16:30:46.453757 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:46.453720    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2" event={"ID":"02a0823c-e86c-48ff-927b-6c46a1837930","Type":"ContainerDied","Data":"026de9bb2098313bc88264a6baf3e0592bcca34fc2c8d5d348d8ce6d9bf5637c"}
Jun 02 16:30:46.454253 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:46.453774    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2" event={"ID":"02a0823c-e86c-48ff-927b-6c46a1837930","Type":"ContainerDied","Data":"afc15790c7e8a8930e7f63d05db49659e5159c9a30065e798f9d71b78806015a"}
Jun 02 16:30:46.454253 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:46.453797    2579 scope.go:117] "RemoveContainer" containerID="66ccc4f2185f172bd5d9e45d15b7cb61f270dafea19a527b6990ccc034f82a1a"
Jun 02 16:30:46.454253 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:46.453811    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2"
Jun 02 16:30:46.455727 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:46.455703    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn" event={"ID":"a361a146-f052-4eae-b388-921853b68269","Type":"ContainerStarted","Data":"1e0e659e0c5691fac8598302eeb79ce212b2f23978840b8847c3e401c5872bcc"}
Jun 02 16:30:46.455841 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:46.455749    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn" event={"ID":"a361a146-f052-4eae-b388-921853b68269","Type":"ContainerStarted","Data":"59de977e565e504b2786914bc915a004efd9c27084a1af0b1184d45813cc31fe"}
Jun 02 16:30:46.456232 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:46.456213    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn"
Jun 02 16:30:46.456329 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:46.456240    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn"
Jun 02 16:30:46.457647 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:46.457621    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn" podUID="a361a146-f052-4eae-b388-921853b68269" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jun 02 16:30:46.461949 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:46.461932    2579 scope.go:117] "RemoveContainer" containerID="026de9bb2098313bc88264a6baf3e0592bcca34fc2c8d5d348d8ce6d9bf5637c"
Jun 02 16:30:46.469086 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:46.469057    2579 scope.go:117] "RemoveContainer" containerID="cc9f63c172855993a3f3d812c41f347b9838e24f572460f3659679d2dfc2e8f9"
Jun 02 16:30:46.476156 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:46.476139    2579 scope.go:117] "RemoveContainer" containerID="66ccc4f2185f172bd5d9e45d15b7cb61f270dafea19a527b6990ccc034f82a1a"
Jun 02 16:30:46.476313 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:46.476291    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2"]
Jun 02 16:30:46.476488 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:30:46.476466    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66ccc4f2185f172bd5d9e45d15b7cb61f270dafea19a527b6990ccc034f82a1a\": container with ID starting with 66ccc4f2185f172bd5d9e45d15b7cb61f270dafea19a527b6990ccc034f82a1a not found: ID does not exist" containerID="66ccc4f2185f172bd5d9e45d15b7cb61f270dafea19a527b6990ccc034f82a1a"
Jun 02 16:30:46.476561 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:46.476513    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66ccc4f2185f172bd5d9e45d15b7cb61f270dafea19a527b6990ccc034f82a1a"} err="failed to get container status \"66ccc4f2185f172bd5d9e45d15b7cb61f270dafea19a527b6990ccc034f82a1a\": rpc error: code = NotFound desc = could not find container \"66ccc4f2185f172bd5d9e45d15b7cb61f270dafea19a527b6990ccc034f82a1a\": container with ID starting with 66ccc4f2185f172bd5d9e45d15b7cb61f270dafea19a527b6990ccc034f82a1a not found: ID does not exist"
Jun 02 16:30:46.476561 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:46.476536    2579 scope.go:117] "RemoveContainer" containerID="026de9bb2098313bc88264a6baf3e0592bcca34fc2c8d5d348d8ce6d9bf5637c"
Jun 02 16:30:46.476817 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:30:46.476801    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"026de9bb2098313bc88264a6baf3e0592bcca34fc2c8d5d348d8ce6d9bf5637c\": container with ID starting with 026de9bb2098313bc88264a6baf3e0592bcca34fc2c8d5d348d8ce6d9bf5637c not found: ID does not exist" containerID="026de9bb2098313bc88264a6baf3e0592bcca34fc2c8d5d348d8ce6d9bf5637c"
Jun 02 16:30:46.476862 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:46.476822    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"026de9bb2098313bc88264a6baf3e0592bcca34fc2c8d5d348d8ce6d9bf5637c"} err="failed to get container status \"026de9bb2098313bc88264a6baf3e0592bcca34fc2c8d5d348d8ce6d9bf5637c\": rpc error: code = NotFound desc = could not find container \"026de9bb2098313bc88264a6baf3e0592bcca34fc2c8d5d348d8ce6d9bf5637c\": container with ID starting with 026de9bb2098313bc88264a6baf3e0592bcca34fc2c8d5d348d8ce6d9bf5637c not found: ID does not exist"
Jun 02 16:30:46.476862 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:46.476837    2579 scope.go:117] "RemoveContainer" containerID="cc9f63c172855993a3f3d812c41f347b9838e24f572460f3659679d2dfc2e8f9"
Jun 02 16:30:46.477057 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:30:46.477043    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc9f63c172855993a3f3d812c41f347b9838e24f572460f3659679d2dfc2e8f9\": container with ID starting with cc9f63c172855993a3f3d812c41f347b9838e24f572460f3659679d2dfc2e8f9 not found: ID does not exist" containerID="cc9f63c172855993a3f3d812c41f347b9838e24f572460f3659679d2dfc2e8f9"
Jun 02 16:30:46.477103 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:46.477059    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc9f63c172855993a3f3d812c41f347b9838e24f572460f3659679d2dfc2e8f9"} err="failed to get container status \"cc9f63c172855993a3f3d812c41f347b9838e24f572460f3659679d2dfc2e8f9\": rpc error: code = NotFound desc = could not find container \"cc9f63c172855993a3f3d812c41f347b9838e24f572460f3659679d2dfc2e8f9\": container with ID starting with cc9f63c172855993a3f3d812c41f347b9838e24f572460f3659679d2dfc2e8f9 not found: ID does not exist"
Jun 02 16:30:46.481548 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:46.481528    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-bdld2"]
Jun 02 16:30:46.501238 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:46.501195    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn" podStartSLOduration=6.501181652 podStartE2EDuration="6.501181652s" podCreationTimestamp="2026-06-02 16:30:40 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:30:46.499699854 +0000 UTC m=+2655.025608298" watchObservedRunningTime="2026-06-02 16:30:46.501181652 +0000 UTC m=+2655.027090082"
Jun 02 16:30:47.460154 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:47.460118    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn" podUID="a361a146-f052-4eae-b388-921853b68269" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jun 02 16:30:48.063364 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:48.063330    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02a0823c-e86c-48ff-927b-6c46a1837930" path="/var/lib/kubelet/pods/02a0823c-e86c-48ff-927b-6c46a1837930/volumes"
Jun 02 16:30:52.465625 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:52.465596    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn"
Jun 02 16:30:52.466150 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:30:52.466125    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn" podUID="a361a146-f052-4eae-b388-921853b68269" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jun 02 16:31:02.466476 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:31:02.466425    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn" podUID="a361a146-f052-4eae-b388-921853b68269" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jun 02 16:31:12.466784 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:31:12.466738    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn" podUID="a361a146-f052-4eae-b388-921853b68269" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jun 02 16:31:22.466610 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:31:22.466526    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn" podUID="a361a146-f052-4eae-b388-921853b68269" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jun 02 16:31:32.078981 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:31:32.078951    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lqx9k_6127e33c-aa77-437e-b443-e956037afedb/ovn-acl-logging/0.log"
Jun 02 16:31:32.082867 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:31:32.082845    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lqx9k_6127e33c-aa77-437e-b443-e956037afedb/ovn-acl-logging/0.log"
Jun 02 16:31:32.466800 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:31:32.466712    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn" podUID="a361a146-f052-4eae-b388-921853b68269" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jun 02 16:31:42.466665 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:31:42.466624    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn" podUID="a361a146-f052-4eae-b388-921853b68269" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jun 02 16:31:52.466794 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:31:52.466746    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn" podUID="a361a146-f052-4eae-b388-921853b68269" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jun 02 16:32:02.467232 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:02.467202    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn"
Jun 02 16:32:11.026689 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:11.026640    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn"]
Jun 02 16:32:11.027157 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:11.027089    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn" podUID="a361a146-f052-4eae-b388-921853b68269" containerName="kserve-container" containerID="cri-o://59de977e565e504b2786914bc915a004efd9c27084a1af0b1184d45813cc31fe" gracePeriod=30
Jun 02 16:32:11.027229 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:11.027136    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn" podUID="a361a146-f052-4eae-b388-921853b68269" containerName="kube-rbac-proxy" containerID="cri-o://1e0e659e0c5691fac8598302eeb79ce212b2f23978840b8847c3e401c5872bcc" gracePeriod=30
Jun 02 16:32:11.196186 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:11.196150    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd"]
Jun 02 16:32:11.196464 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:11.196451    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02a0823c-e86c-48ff-927b-6c46a1837930" containerName="storage-initializer"
Jun 02 16:32:11.196516 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:11.196466    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="02a0823c-e86c-48ff-927b-6c46a1837930" containerName="storage-initializer"
Jun 02 16:32:11.196516 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:11.196474    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02a0823c-e86c-48ff-927b-6c46a1837930" containerName="kserve-container"
Jun 02 16:32:11.196516 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:11.196480    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="02a0823c-e86c-48ff-927b-6c46a1837930" containerName="kserve-container"
Jun 02 16:32:11.196516 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:11.196489    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02a0823c-e86c-48ff-927b-6c46a1837930" containerName="kube-rbac-proxy"
Jun 02 16:32:11.196516 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:11.196494    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="02a0823c-e86c-48ff-927b-6c46a1837930" containerName="kube-rbac-proxy"
Jun 02 16:32:11.196663 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:11.196547    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="02a0823c-e86c-48ff-927b-6c46a1837930" containerName="kserve-container"
Jun 02 16:32:11.196663 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:11.196558    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="02a0823c-e86c-48ff-927b-6c46a1837930" containerName="kube-rbac-proxy"
Jun 02 16:32:11.199529 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:11.199506    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd"
Jun 02 16:32:11.202831 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:11.202813    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Jun 02 16:32:11.203106 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:11.203090    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Jun 02 16:32:11.216537 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:11.216500    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd"]
Jun 02 16:32:11.286378 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:11.286280    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd\" (UID: \"67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd"
Jun 02 16:32:11.286378 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:11.286322    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd\" (UID: \"67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd"
Jun 02 16:32:11.286378 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:11.286367    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqf77\" (UniqueName: \"kubernetes.io/projected/67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8-kube-api-access-sqf77\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd\" (UID: \"67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd"
Jun 02 16:32:11.286673 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:11.286432    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd\" (UID: \"67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd"
Jun 02 16:32:11.387099 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:11.387053    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd\" (UID: \"67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd"
Jun 02 16:32:11.387099 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:11.387099    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd\" (UID: \"67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd"
Jun 02 16:32:11.387366 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:11.387142    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sqf77\" (UniqueName: \"kubernetes.io/projected/67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8-kube-api-access-sqf77\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd\" (UID: \"67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd"
Jun 02 16:32:11.387366 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:11.387159    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd\" (UID: \"67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd"
Jun 02 16:32:11.387366 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:32:11.387296    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-serving-cert: secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Jun 02 16:32:11.387366 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:32:11.387362    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8-proxy-tls podName:67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8 nodeName:}" failed. No retries permitted until 2026-06-02 16:32:11.887342967 +0000 UTC m=+2740.413251392 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8-proxy-tls") pod "isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd" (UID: "67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8") : secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Jun 02 16:32:11.387615 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:11.387596    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd\" (UID: \"67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd"
Jun 02 16:32:11.387822 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:11.387803    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd\" (UID: \"67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd"
Jun 02 16:32:11.398091 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:11.398069    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqf77\" (UniqueName: \"kubernetes.io/projected/67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8-kube-api-access-sqf77\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd\" (UID: \"67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd"
Jun 02 16:32:11.690778 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:11.690691    2579 generic.go:358] "Generic (PLEG): container finished" podID="a361a146-f052-4eae-b388-921853b68269" containerID="1e0e659e0c5691fac8598302eeb79ce212b2f23978840b8847c3e401c5872bcc" exitCode=2
Jun 02 16:32:11.690778 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:11.690748    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn" event={"ID":"a361a146-f052-4eae-b388-921853b68269","Type":"ContainerDied","Data":"1e0e659e0c5691fac8598302eeb79ce212b2f23978840b8847c3e401c5872bcc"}
Jun 02 16:32:11.890997 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:11.890961    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd\" (UID: \"67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd"
Jun 02 16:32:11.893544 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:11.893524    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd\" (UID: \"67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd"
Jun 02 16:32:12.109488 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:12.109440    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd"
Jun 02 16:32:12.244564 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:12.244537    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd"]
Jun 02 16:32:12.247025 ip-10-0-129-156 kubenswrapper[2579]: W0602 16:32:12.246999    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b2fbd9_03f8_4b5e_b7db_e673e5bc90e8.slice/crio-fe585dbcf170b8e3192ae69a288e6a3977fa16615a21002a9aff12dd64df4492 WatchSource:0}: Error finding container fe585dbcf170b8e3192ae69a288e6a3977fa16615a21002a9aff12dd64df4492: Status 404 returned error can't find the container with id fe585dbcf170b8e3192ae69a288e6a3977fa16615a21002a9aff12dd64df4492
Jun 02 16:32:12.460628 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:12.460530    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn" podUID="a361a146-f052-4eae-b388-921853b68269" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Jun 02 16:32:12.466082 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:12.466052    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn" podUID="a361a146-f052-4eae-b388-921853b68269" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jun 02 16:32:12.695937 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:12.695898    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd" event={"ID":"67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8","Type":"ContainerStarted","Data":"9682ed22cd78583d58c1a154054ab64a4161402f573e1402caf917fbb1427fbd"}
Jun 02 16:32:12.695937 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:12.695942    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd" event={"ID":"67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8","Type":"ContainerStarted","Data":"fe585dbcf170b8e3192ae69a288e6a3977fa16615a21002a9aff12dd64df4492"}
Jun 02 16:32:16.707510 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:16.707477    2579 generic.go:358] "Generic (PLEG): container finished" podID="67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8" containerID="9682ed22cd78583d58c1a154054ab64a4161402f573e1402caf917fbb1427fbd" exitCode=0
Jun 02 16:32:16.707940 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:16.707555    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd" event={"ID":"67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8","Type":"ContainerDied","Data":"9682ed22cd78583d58c1a154054ab64a4161402f573e1402caf917fbb1427fbd"}
Jun 02 16:32:16.709573 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:16.709551    2579 generic.go:358] "Generic (PLEG): container finished" podID="a361a146-f052-4eae-b388-921853b68269" containerID="59de977e565e504b2786914bc915a004efd9c27084a1af0b1184d45813cc31fe" exitCode=0
Jun 02 16:32:16.709670 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:16.709607    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn" event={"ID":"a361a146-f052-4eae-b388-921853b68269","Type":"ContainerDied","Data":"59de977e565e504b2786914bc915a004efd9c27084a1af0b1184d45813cc31fe"}
Jun 02 16:32:17.183416 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:17.183382    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn"
Jun 02 16:32:17.233396 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:17.233358    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rsdvr\" (UniqueName: \"kubernetes.io/projected/a361a146-f052-4eae-b388-921853b68269-kube-api-access-rsdvr\") pod \"a361a146-f052-4eae-b388-921853b68269\" (UID: \"a361a146-f052-4eae-b388-921853b68269\") "
Jun 02 16:32:17.233601 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:17.233440    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a361a146-f052-4eae-b388-921853b68269-proxy-tls\") pod \"a361a146-f052-4eae-b388-921853b68269\" (UID: \"a361a146-f052-4eae-b388-921853b68269\") "
Jun 02 16:32:17.233601 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:17.233490    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a361a146-f052-4eae-b388-921853b68269-kserve-provision-location\") pod \"a361a146-f052-4eae-b388-921853b68269\" (UID: \"a361a146-f052-4eae-b388-921853b68269\") "
Jun 02 16:32:17.233601 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:17.233512    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a361a146-f052-4eae-b388-921853b68269-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") pod \"a361a146-f052-4eae-b388-921853b68269\" (UID: \"a361a146-f052-4eae-b388-921853b68269\") "
Jun 02 16:32:17.233836 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:17.233805    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a361a146-f052-4eae-b388-921853b68269-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a361a146-f052-4eae-b388-921853b68269" (UID: "a361a146-f052-4eae-b388-921853b68269"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:32:17.233915 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:17.233857    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a361a146-f052-4eae-b388-921853b68269-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-lightgbm-kube-rbac-proxy-sar-config") pod "a361a146-f052-4eae-b388-921853b68269" (UID: "a361a146-f052-4eae-b388-921853b68269"). InnerVolumeSpecName "isvc-predictive-lightgbm-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:32:17.235698 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:17.235676    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a361a146-f052-4eae-b388-921853b68269-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a361a146-f052-4eae-b388-921853b68269" (UID: "a361a146-f052-4eae-b388-921853b68269"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:32:17.235767 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:17.235743    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a361a146-f052-4eae-b388-921853b68269-kube-api-access-rsdvr" (OuterVolumeSpecName: "kube-api-access-rsdvr") pod "a361a146-f052-4eae-b388-921853b68269" (UID: "a361a146-f052-4eae-b388-921853b68269"). InnerVolumeSpecName "kube-api-access-rsdvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:32:17.334638 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:17.334521    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsdvr\" (UniqueName: \"kubernetes.io/projected/a361a146-f052-4eae-b388-921853b68269-kube-api-access-rsdvr\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:32:17.334638 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:17.334562    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a361a146-f052-4eae-b388-921853b68269-proxy-tls\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:32:17.334638 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:17.334578    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a361a146-f052-4eae-b388-921853b68269-kserve-provision-location\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:32:17.334638 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:17.334591    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a361a146-f052-4eae-b388-921853b68269-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:32:17.715519 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:17.715427    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd" event={"ID":"67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8","Type":"ContainerStarted","Data":"18ae9d4ade447647c9c886abfeb2101d9799259bc7dfa1dc959a7e9abe29ad6e"}
Jun 02 16:32:17.715519 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:17.715473    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd" event={"ID":"67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8","Type":"ContainerStarted","Data":"aad84373c21d5814002ef6ae00222da4465875f7c0540650e46a1c4adac0ec49"}
Jun 02 16:32:17.716017 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:17.715730    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd"
Jun 02 16:32:17.717049 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:17.717021    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn" event={"ID":"a361a146-f052-4eae-b388-921853b68269","Type":"ContainerDied","Data":"1b8a4424dac33bd516f60aee37769aaa5269ec561586fc6624d0e811234dcdf8"}
Jun 02 16:32:17.717155 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:17.717061    2579 scope.go:117] "RemoveContainer" containerID="1e0e659e0c5691fac8598302eeb79ce212b2f23978840b8847c3e401c5872bcc"
Jun 02 16:32:17.717155 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:17.717074    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn"
Jun 02 16:32:17.724862 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:17.724845    2579 scope.go:117] "RemoveContainer" containerID="59de977e565e504b2786914bc915a004efd9c27084a1af0b1184d45813cc31fe"
Jun 02 16:32:17.731916 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:17.731897    2579 scope.go:117] "RemoveContainer" containerID="8e4d0e77f85d38c3e2c5a578699a237a555c5b7d47441a7cef07d7d630751d3b"
Jun 02 16:32:17.734926 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:17.734876    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd" podStartSLOduration=6.734861956 podStartE2EDuration="6.734861956s" podCreationTimestamp="2026-06-02 16:32:11 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:32:17.733727429 +0000 UTC m=+2746.259635862" watchObservedRunningTime="2026-06-02 16:32:17.734861956 +0000 UTC m=+2746.260770388"
Jun 02 16:32:17.746420 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:17.746380    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn"]
Jun 02 16:32:17.750584 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:17.750554    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rxcnn"]
Jun 02 16:32:18.063175 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:18.063143    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a361a146-f052-4eae-b388-921853b68269" path="/var/lib/kubelet/pods/a361a146-f052-4eae-b388-921853b68269/volumes"
Jun 02 16:32:18.720439 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:18.720390    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd"
Jun 02 16:32:24.729001 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:24.728970    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd"
Jun 02 16:32:54.730041 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:32:54.729953    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd" podUID="67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.25:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.25:8080: connect: connection refused"
Jun 02 16:33:04.729993 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:04.729950    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd" podUID="67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.25:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.25:8080: connect: connection refused"
Jun 02 16:33:14.730515 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:14.730469    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd" podUID="67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.25:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.25:8080: connect: connection refused"
Jun 02 16:33:24.729997 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:24.729956    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd" podUID="67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.25:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.25:8080: connect: connection refused"
Jun 02 16:33:30.060571 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:30.060527    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd" podUID="67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.25:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.25:8080: connect: connection refused"
Jun 02 16:33:40.064356 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:40.064325    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd"
Jun 02 16:33:41.258471 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:41.258431    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd"]
Jun 02 16:33:41.258883 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:41.258742    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd" podUID="67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8" containerName="kserve-container" containerID="cri-o://aad84373c21d5814002ef6ae00222da4465875f7c0540650e46a1c4adac0ec49" gracePeriod=30
Jun 02 16:33:41.258883 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:41.258799    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd" podUID="67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8" containerName="kube-rbac-proxy" containerID="cri-o://18ae9d4ade447647c9c886abfeb2101d9799259bc7dfa1dc959a7e9abe29ad6e" gracePeriod=30
Jun 02 16:33:41.381628 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:41.381597    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf"]
Jun 02 16:33:41.381904 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:41.381889    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a361a146-f052-4eae-b388-921853b68269" containerName="kserve-container"
Jun 02 16:33:41.381947 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:41.381908    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a361a146-f052-4eae-b388-921853b68269" containerName="kserve-container"
Jun 02 16:33:41.381947 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:41.381918    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a361a146-f052-4eae-b388-921853b68269" containerName="kube-rbac-proxy"
Jun 02 16:33:41.381947 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:41.381924    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a361a146-f052-4eae-b388-921853b68269" containerName="kube-rbac-proxy"
Jun 02 16:33:41.381947 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:41.381934    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a361a146-f052-4eae-b388-921853b68269" containerName="storage-initializer"
Jun 02 16:33:41.381947 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:41.381940    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a361a146-f052-4eae-b388-921853b68269" containerName="storage-initializer"
Jun 02 16:33:41.382091 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:41.381994    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="a361a146-f052-4eae-b388-921853b68269" containerName="kserve-container"
Jun 02 16:33:41.382091 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:41.382004    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="a361a146-f052-4eae-b388-921853b68269" containerName="kube-rbac-proxy"
Jun 02 16:33:41.384963 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:41.384947    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf"
Jun 02 16:33:41.387318 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:41.387295    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Jun 02 16:33:41.387449 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:41.387306    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Jun 02 16:33:41.394117 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:41.394092    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf"]
Jun 02 16:33:41.488187 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:41.488146    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edb22358-f20f-4ec0-96ca-4bd54f6d4090-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf\" (UID: \"edb22358-f20f-4ec0-96ca-4bd54f6d4090\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf"
Jun 02 16:33:41.488187 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:41.488189    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/edb22358-f20f-4ec0-96ca-4bd54f6d4090-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf\" (UID: \"edb22358-f20f-4ec0-96ca-4bd54f6d4090\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf"
Jun 02 16:33:41.488493 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:41.488218    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/edb22358-f20f-4ec0-96ca-4bd54f6d4090-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf\" (UID: \"edb22358-f20f-4ec0-96ca-4bd54f6d4090\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf"
Jun 02 16:33:41.488493 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:41.488300    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxmzl\" (UniqueName: \"kubernetes.io/projected/edb22358-f20f-4ec0-96ca-4bd54f6d4090-kube-api-access-mxmzl\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf\" (UID: \"edb22358-f20f-4ec0-96ca-4bd54f6d4090\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf"
Jun 02 16:33:41.589351 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:41.589220    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mxmzl\" (UniqueName: \"kubernetes.io/projected/edb22358-f20f-4ec0-96ca-4bd54f6d4090-kube-api-access-mxmzl\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf\" (UID: \"edb22358-f20f-4ec0-96ca-4bd54f6d4090\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf"
Jun 02 16:33:41.589351 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:41.589292    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edb22358-f20f-4ec0-96ca-4bd54f6d4090-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf\" (UID: \"edb22358-f20f-4ec0-96ca-4bd54f6d4090\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf"
Jun 02 16:33:41.589351 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:41.589323    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/edb22358-f20f-4ec0-96ca-4bd54f6d4090-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf\" (UID: \"edb22358-f20f-4ec0-96ca-4bd54f6d4090\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf"
Jun 02 16:33:41.589351 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:41.589349    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/edb22358-f20f-4ec0-96ca-4bd54f6d4090-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf\" (UID: \"edb22358-f20f-4ec0-96ca-4bd54f6d4090\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf"
Jun 02 16:33:41.589811 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:41.589785    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edb22358-f20f-4ec0-96ca-4bd54f6d4090-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf\" (UID: \"edb22358-f20f-4ec0-96ca-4bd54f6d4090\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf"
Jun 02 16:33:41.590048 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:41.590030    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/edb22358-f20f-4ec0-96ca-4bd54f6d4090-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf\" (UID: \"edb22358-f20f-4ec0-96ca-4bd54f6d4090\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf"
Jun 02 16:33:41.592033 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:41.592014    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/edb22358-f20f-4ec0-96ca-4bd54f6d4090-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf\" (UID: \"edb22358-f20f-4ec0-96ca-4bd54f6d4090\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf"
Jun 02 16:33:41.598752 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:41.598724    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxmzl\" (UniqueName: \"kubernetes.io/projected/edb22358-f20f-4ec0-96ca-4bd54f6d4090-kube-api-access-mxmzl\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf\" (UID: \"edb22358-f20f-4ec0-96ca-4bd54f6d4090\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf"
Jun 02 16:33:41.695262 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:41.695223    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf"
Jun 02 16:33:41.826541 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:41.826517    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf"]
Jun 02 16:33:41.829139 ip-10-0-129-156 kubenswrapper[2579]: W0602 16:33:41.829108    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb22358_f20f_4ec0_96ca_4bd54f6d4090.slice/crio-7f12142ec5b846be854b77d0a1adf8d56850090b92c667cc358cd050335cffef WatchSource:0}: Error finding container 7f12142ec5b846be854b77d0a1adf8d56850090b92c667cc358cd050335cffef: Status 404 returned error can't find the container with id 7f12142ec5b846be854b77d0a1adf8d56850090b92c667cc358cd050335cffef
Jun 02 16:33:41.830974 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:41.830960    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 16:33:41.957920 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:41.957876    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf" event={"ID":"edb22358-f20f-4ec0-96ca-4bd54f6d4090","Type":"ContainerStarted","Data":"e392b7b8dcc3fba4b643819961e901f73d2eb251257e11e7a26d166d9689443a"}
Jun 02 16:33:41.958125 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:41.957928    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf" event={"ID":"edb22358-f20f-4ec0-96ca-4bd54f6d4090","Type":"ContainerStarted","Data":"7f12142ec5b846be854b77d0a1adf8d56850090b92c667cc358cd050335cffef"}
Jun 02 16:33:41.960057 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:41.960031    2579 generic.go:358] "Generic (PLEG): container finished" podID="67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8" containerID="18ae9d4ade447647c9c886abfeb2101d9799259bc7dfa1dc959a7e9abe29ad6e" exitCode=2
Jun 02 16:33:41.960168 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:41.960093    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd" event={"ID":"67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8","Type":"ContainerDied","Data":"18ae9d4ade447647c9c886abfeb2101d9799259bc7dfa1dc959a7e9abe29ad6e"}
Jun 02 16:33:44.724422 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:44.724352    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd" podUID="67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Jun 02 16:33:45.973043 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:45.972939    2579 generic.go:358] "Generic (PLEG): container finished" podID="edb22358-f20f-4ec0-96ca-4bd54f6d4090" containerID="e392b7b8dcc3fba4b643819961e901f73d2eb251257e11e7a26d166d9689443a" exitCode=0
Jun 02 16:33:45.973043 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:45.972983    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf" event={"ID":"edb22358-f20f-4ec0-96ca-4bd54f6d4090","Type":"ContainerDied","Data":"e392b7b8dcc3fba4b643819961e901f73d2eb251257e11e7a26d166d9689443a"}
Jun 02 16:33:46.697162 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:46.697138    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd"
Jun 02 16:33:46.735570 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:46.735540    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8\" (UID: \"67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8\") "
Jun 02 16:33:46.735752 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:46.735582    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqf77\" (UniqueName: \"kubernetes.io/projected/67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8-kube-api-access-sqf77\") pod \"67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8\" (UID: \"67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8\") "
Jun 02 16:33:46.735752 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:46.735612    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8-kserve-provision-location\") pod \"67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8\" (UID: \"67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8\") "
Jun 02 16:33:46.735752 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:46.735634    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8-proxy-tls\") pod \"67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8\" (UID: \"67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8\") "
Jun 02 16:33:46.735966 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:46.735942    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8" (UID: "67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:33:46.736047 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:46.735971    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config") pod "67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8" (UID: "67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8"). InnerVolumeSpecName "isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:33:46.737840 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:46.737818    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8-kube-api-access-sqf77" (OuterVolumeSpecName: "kube-api-access-sqf77") pod "67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8" (UID: "67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8"). InnerVolumeSpecName "kube-api-access-sqf77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:33:46.737938 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:46.737847    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8" (UID: "67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:33:46.836700 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:46.836662    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8-kserve-provision-location\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:33:46.836700 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:46.836692    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8-proxy-tls\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:33:46.836700 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:46.836702    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:33:46.836938 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:46.836714    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqf77\" (UniqueName: \"kubernetes.io/projected/67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8-kube-api-access-sqf77\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:33:46.977907 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:46.977868    2579 generic.go:358] "Generic (PLEG): container finished" podID="67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8" containerID="aad84373c21d5814002ef6ae00222da4465875f7c0540650e46a1c4adac0ec49" exitCode=0
Jun 02 16:33:46.978372 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:46.977934    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd" event={"ID":"67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8","Type":"ContainerDied","Data":"aad84373c21d5814002ef6ae00222da4465875f7c0540650e46a1c4adac0ec49"}
Jun 02 16:33:46.978372 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:46.977947    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd"
Jun 02 16:33:46.978372 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:46.977969    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd" event={"ID":"67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8","Type":"ContainerDied","Data":"fe585dbcf170b8e3192ae69a288e6a3977fa16615a21002a9aff12dd64df4492"}
Jun 02 16:33:46.978372 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:46.977985    2579 scope.go:117] "RemoveContainer" containerID="18ae9d4ade447647c9c886abfeb2101d9799259bc7dfa1dc959a7e9abe29ad6e"
Jun 02 16:33:46.980114 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:46.980095    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf" event={"ID":"edb22358-f20f-4ec0-96ca-4bd54f6d4090","Type":"ContainerStarted","Data":"f71924036df30ffe950657bf0d6270ba22989e04ec5b67dde10a943d76965c8f"}
Jun 02 16:33:46.980221 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:46.980120    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf" event={"ID":"edb22358-f20f-4ec0-96ca-4bd54f6d4090","Type":"ContainerStarted","Data":"ec39ce23fdd54b535be4b5c29bac18074093bf5da38397185642eae6d0fcd824"}
Jun 02 16:33:46.980347 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:46.980333    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf"
Jun 02 16:33:46.980463 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:46.980360    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf"
Jun 02 16:33:46.987130 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:46.987108    2579 scope.go:117] "RemoveContainer" containerID="aad84373c21d5814002ef6ae00222da4465875f7c0540650e46a1c4adac0ec49"
Jun 02 16:33:46.995062 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:46.995038    2579 scope.go:117] "RemoveContainer" containerID="9682ed22cd78583d58c1a154054ab64a4161402f573e1402caf917fbb1427fbd"
Jun 02 16:33:47.002177 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:47.002133    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf" podStartSLOduration=6.002119057 podStartE2EDuration="6.002119057s" podCreationTimestamp="2026-06-02 16:33:41 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:33:47.000347344 +0000 UTC m=+2835.526255810" watchObservedRunningTime="2026-06-02 16:33:47.002119057 +0000 UTC m=+2835.528027487"
Jun 02 16:33:47.002370 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:47.002324    2579 scope.go:117] "RemoveContainer" containerID="18ae9d4ade447647c9c886abfeb2101d9799259bc7dfa1dc959a7e9abe29ad6e"
Jun 02 16:33:47.002633 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:33:47.002611    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18ae9d4ade447647c9c886abfeb2101d9799259bc7dfa1dc959a7e9abe29ad6e\": container with ID starting with 18ae9d4ade447647c9c886abfeb2101d9799259bc7dfa1dc959a7e9abe29ad6e not found: ID does not exist" containerID="18ae9d4ade447647c9c886abfeb2101d9799259bc7dfa1dc959a7e9abe29ad6e"
Jun 02 16:33:47.002673 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:47.002644    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18ae9d4ade447647c9c886abfeb2101d9799259bc7dfa1dc959a7e9abe29ad6e"} err="failed to get container status \"18ae9d4ade447647c9c886abfeb2101d9799259bc7dfa1dc959a7e9abe29ad6e\": rpc error: code = NotFound desc = could not find container \"18ae9d4ade447647c9c886abfeb2101d9799259bc7dfa1dc959a7e9abe29ad6e\": container with ID starting with 18ae9d4ade447647c9c886abfeb2101d9799259bc7dfa1dc959a7e9abe29ad6e not found: ID does not exist"
Jun 02 16:33:47.002673 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:47.002663    2579 scope.go:117] "RemoveContainer" containerID="aad84373c21d5814002ef6ae00222da4465875f7c0540650e46a1c4adac0ec49"
Jun 02 16:33:47.002912 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:33:47.002893    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aad84373c21d5814002ef6ae00222da4465875f7c0540650e46a1c4adac0ec49\": container with ID starting with aad84373c21d5814002ef6ae00222da4465875f7c0540650e46a1c4adac0ec49 not found: ID does not exist" containerID="aad84373c21d5814002ef6ae00222da4465875f7c0540650e46a1c4adac0ec49"
Jun 02 16:33:47.002979 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:47.002920    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aad84373c21d5814002ef6ae00222da4465875f7c0540650e46a1c4adac0ec49"} err="failed to get container status \"aad84373c21d5814002ef6ae00222da4465875f7c0540650e46a1c4adac0ec49\": rpc error: code = NotFound desc = could not find container \"aad84373c21d5814002ef6ae00222da4465875f7c0540650e46a1c4adac0ec49\": container with ID starting with aad84373c21d5814002ef6ae00222da4465875f7c0540650e46a1c4adac0ec49 not found: ID does not exist"
Jun 02 16:33:47.002979 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:47.002936    2579 scope.go:117] "RemoveContainer" containerID="9682ed22cd78583d58c1a154054ab64a4161402f573e1402caf917fbb1427fbd"
Jun 02 16:33:47.003177 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:33:47.003160    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9682ed22cd78583d58c1a154054ab64a4161402f573e1402caf917fbb1427fbd\": container with ID starting with 9682ed22cd78583d58c1a154054ab64a4161402f573e1402caf917fbb1427fbd not found: ID does not exist" containerID="9682ed22cd78583d58c1a154054ab64a4161402f573e1402caf917fbb1427fbd"
Jun 02 16:33:47.003224 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:47.003180    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9682ed22cd78583d58c1a154054ab64a4161402f573e1402caf917fbb1427fbd"} err="failed to get container status \"9682ed22cd78583d58c1a154054ab64a4161402f573e1402caf917fbb1427fbd\": rpc error: code = NotFound desc = could not find container \"9682ed22cd78583d58c1a154054ab64a4161402f573e1402caf917fbb1427fbd\": container with ID starting with 9682ed22cd78583d58c1a154054ab64a4161402f573e1402caf917fbb1427fbd not found: ID does not exist"
Jun 02 16:33:47.014073 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:47.014047    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd"]
Jun 02 16:33:47.018205 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:47.018179    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-blvmd"]
Jun 02 16:33:48.063218 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:48.063179    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8" path="/var/lib/kubelet/pods/67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8/volumes"
Jun 02 16:33:52.989283 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:33:52.989253    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf"
Jun 02 16:34:22.990733 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:34:22.990648    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf" podUID="edb22358-f20f-4ec0-96ca-4bd54f6d4090" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.26:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.26:8080: connect: connection refused"
Jun 02 16:34:32.990182 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:34:32.990142    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf" podUID="edb22358-f20f-4ec0-96ca-4bd54f6d4090" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.26:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.26:8080: connect: connection refused"
Jun 02 16:34:42.990236 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:34:42.990200    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf" podUID="edb22358-f20f-4ec0-96ca-4bd54f6d4090" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.26:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.26:8080: connect: connection refused"
Jun 02 16:34:52.990649 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:34:52.990602    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf" podUID="edb22358-f20f-4ec0-96ca-4bd54f6d4090" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.26:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.26:8080: connect: connection refused"
Jun 02 16:35:02.993930 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:02.993897    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf"
Jun 02 16:35:11.433163 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:11.433116    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf"]
Jun 02 16:35:11.433675 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:11.433575    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf" podUID="edb22358-f20f-4ec0-96ca-4bd54f6d4090" containerName="kserve-container" containerID="cri-o://ec39ce23fdd54b535be4b5c29bac18074093bf5da38397185642eae6d0fcd824" gracePeriod=30
Jun 02 16:35:11.433774 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:11.433731    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf" podUID="edb22358-f20f-4ec0-96ca-4bd54f6d4090" containerName="kube-rbac-proxy" containerID="cri-o://f71924036df30ffe950657bf0d6270ba22989e04ec5b67dde10a943d76965c8f" gracePeriod=30
Jun 02 16:35:11.554159 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:11.554126    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8"]
Jun 02 16:35:11.554484 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:11.554469    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8" containerName="kserve-container"
Jun 02 16:35:11.554484 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:11.554486    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8" containerName="kserve-container"
Jun 02 16:35:11.554588 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:11.554494    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8" containerName="kube-rbac-proxy"
Jun 02 16:35:11.554588 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:11.554500    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8" containerName="kube-rbac-proxy"
Jun 02 16:35:11.554588 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:11.554518    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8" containerName="storage-initializer"
Jun 02 16:35:11.554588 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:11.554523    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8" containerName="storage-initializer"
Jun 02 16:35:11.554588 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:11.554563    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8" containerName="kube-rbac-proxy"
Jun 02 16:35:11.554588 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:11.554571    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="67b2fbd9-03f8-4b5e-b7db-e673e5bc90e8" containerName="kserve-container"
Jun 02 16:35:11.556319 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:11.556303    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8"
Jun 02 16:35:11.558758 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:11.558738    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Jun 02 16:35:11.558864 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:11.558774    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Jun 02 16:35:11.568114 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:11.568089    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8"]
Jun 02 16:35:11.669051 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:11.669017    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c209106-cb79-4124-8808-78b1acb753c7-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8\" (UID: \"5c209106-cb79-4124-8808-78b1acb753c7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8"
Jun 02 16:35:11.669222 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:11.669062    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsdxb\" (UniqueName: \"kubernetes.io/projected/5c209106-cb79-4124-8808-78b1acb753c7-kube-api-access-gsdxb\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8\" (UID: \"5c209106-cb79-4124-8808-78b1acb753c7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8"
Jun 02 16:35:11.669222 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:11.669136    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c209106-cb79-4124-8808-78b1acb753c7-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8\" (UID: \"5c209106-cb79-4124-8808-78b1acb753c7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8"
Jun 02 16:35:11.669222 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:11.669194    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c209106-cb79-4124-8808-78b1acb753c7-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8\" (UID: \"5c209106-cb79-4124-8808-78b1acb753c7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8"
Jun 02 16:35:11.769637 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:11.769602    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c209106-cb79-4124-8808-78b1acb753c7-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8\" (UID: \"5c209106-cb79-4124-8808-78b1acb753c7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8"
Jun 02 16:35:11.769834 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:11.769661    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c209106-cb79-4124-8808-78b1acb753c7-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8\" (UID: \"5c209106-cb79-4124-8808-78b1acb753c7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8"
Jun 02 16:35:11.769834 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:11.769699    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c209106-cb79-4124-8808-78b1acb753c7-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8\" (UID: \"5c209106-cb79-4124-8808-78b1acb753c7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8"
Jun 02 16:35:11.769834 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:11.769726    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gsdxb\" (UniqueName: \"kubernetes.io/projected/5c209106-cb79-4124-8808-78b1acb753c7-kube-api-access-gsdxb\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8\" (UID: \"5c209106-cb79-4124-8808-78b1acb753c7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8"
Jun 02 16:35:11.770118 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:11.770099    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c209106-cb79-4124-8808-78b1acb753c7-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8\" (UID: \"5c209106-cb79-4124-8808-78b1acb753c7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8"
Jun 02 16:35:11.770451 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:11.770423    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c209106-cb79-4124-8808-78b1acb753c7-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8\" (UID: \"5c209106-cb79-4124-8808-78b1acb753c7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8"
Jun 02 16:35:11.772396 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:11.772372    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c209106-cb79-4124-8808-78b1acb753c7-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8\" (UID: \"5c209106-cb79-4124-8808-78b1acb753c7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8"
Jun 02 16:35:11.778619 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:11.778593    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsdxb\" (UniqueName: \"kubernetes.io/projected/5c209106-cb79-4124-8808-78b1acb753c7-kube-api-access-gsdxb\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8\" (UID: \"5c209106-cb79-4124-8808-78b1acb753c7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8"
Jun 02 16:35:11.866504 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:11.866453    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8"
Jun 02 16:35:12.044693 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:12.044664    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8"]
Jun 02 16:35:12.047063 ip-10-0-129-156 kubenswrapper[2579]: W0602 16:35:12.047033    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c209106_cb79_4124_8808_78b1acb753c7.slice/crio-d8c8d2bb93f520a1d2aec1d0845d5017eba6a118f65ece70a465afbce8ab5ca1 WatchSource:0}: Error finding container d8c8d2bb93f520a1d2aec1d0845d5017eba6a118f65ece70a465afbce8ab5ca1: Status 404 returned error can't find the container with id d8c8d2bb93f520a1d2aec1d0845d5017eba6a118f65ece70a465afbce8ab5ca1
Jun 02 16:35:12.217812 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:12.217772    2579 generic.go:358] "Generic (PLEG): container finished" podID="edb22358-f20f-4ec0-96ca-4bd54f6d4090" containerID="f71924036df30ffe950657bf0d6270ba22989e04ec5b67dde10a943d76965c8f" exitCode=2
Jun 02 16:35:12.217994 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:12.217829    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf" event={"ID":"edb22358-f20f-4ec0-96ca-4bd54f6d4090","Type":"ContainerDied","Data":"f71924036df30ffe950657bf0d6270ba22989e04ec5b67dde10a943d76965c8f"}
Jun 02 16:35:12.221827 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:12.221794    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8" event={"ID":"5c209106-cb79-4124-8808-78b1acb753c7","Type":"ContainerStarted","Data":"3517a0450f10fa8ac107fcc27d32bc175e209b1573715d8baa4e9c6601ca2979"}
Jun 02 16:35:12.221988 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:12.221831    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8" event={"ID":"5c209106-cb79-4124-8808-78b1acb753c7","Type":"ContainerStarted","Data":"d8c8d2bb93f520a1d2aec1d0845d5017eba6a118f65ece70a465afbce8ab5ca1"}
Jun 02 16:35:12.985287 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:12.985245    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf" podUID="edb22358-f20f-4ec0-96ca-4bd54f6d4090" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Jun 02 16:35:12.990043 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:12.990016    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf" podUID="edb22358-f20f-4ec0-96ca-4bd54f6d4090" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.26:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.26:8080: connect: connection refused"
Jun 02 16:35:16.234136 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:16.234098    2579 generic.go:358] "Generic (PLEG): container finished" podID="5c209106-cb79-4124-8808-78b1acb753c7" containerID="3517a0450f10fa8ac107fcc27d32bc175e209b1573715d8baa4e9c6601ca2979" exitCode=0
Jun 02 16:35:16.234585 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:16.234172    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8" event={"ID":"5c209106-cb79-4124-8808-78b1acb753c7","Type":"ContainerDied","Data":"3517a0450f10fa8ac107fcc27d32bc175e209b1573715d8baa4e9c6601ca2979"}
Jun 02 16:35:16.779860 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:16.779833    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf"
Jun 02 16:35:16.914311 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:16.914220    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edb22358-f20f-4ec0-96ca-4bd54f6d4090-kserve-provision-location\") pod \"edb22358-f20f-4ec0-96ca-4bd54f6d4090\" (UID: \"edb22358-f20f-4ec0-96ca-4bd54f6d4090\") "
Jun 02 16:35:16.914311 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:16.914267    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxmzl\" (UniqueName: \"kubernetes.io/projected/edb22358-f20f-4ec0-96ca-4bd54f6d4090-kube-api-access-mxmzl\") pod \"edb22358-f20f-4ec0-96ca-4bd54f6d4090\" (UID: \"edb22358-f20f-4ec0-96ca-4bd54f6d4090\") "
Jun 02 16:35:16.914550 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:16.914329    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/edb22358-f20f-4ec0-96ca-4bd54f6d4090-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"edb22358-f20f-4ec0-96ca-4bd54f6d4090\" (UID: \"edb22358-f20f-4ec0-96ca-4bd54f6d4090\") "
Jun 02 16:35:16.914550 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:16.914376    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/edb22358-f20f-4ec0-96ca-4bd54f6d4090-proxy-tls\") pod \"edb22358-f20f-4ec0-96ca-4bd54f6d4090\" (UID: \"edb22358-f20f-4ec0-96ca-4bd54f6d4090\") "
Jun 02 16:35:16.914659 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:16.914543    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edb22358-f20f-4ec0-96ca-4bd54f6d4090-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "edb22358-f20f-4ec0-96ca-4bd54f6d4090" (UID: "edb22358-f20f-4ec0-96ca-4bd54f6d4090"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:35:16.914714 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:16.914672    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edb22358-f20f-4ec0-96ca-4bd54f6d4090-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config") pod "edb22358-f20f-4ec0-96ca-4bd54f6d4090" (UID: "edb22358-f20f-4ec0-96ca-4bd54f6d4090"). InnerVolumeSpecName "isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:35:16.916591 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:16.916562    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edb22358-f20f-4ec0-96ca-4bd54f6d4090-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "edb22358-f20f-4ec0-96ca-4bd54f6d4090" (UID: "edb22358-f20f-4ec0-96ca-4bd54f6d4090"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:35:16.916709 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:16.916566    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edb22358-f20f-4ec0-96ca-4bd54f6d4090-kube-api-access-mxmzl" (OuterVolumeSpecName: "kube-api-access-mxmzl") pod "edb22358-f20f-4ec0-96ca-4bd54f6d4090" (UID: "edb22358-f20f-4ec0-96ca-4bd54f6d4090"). InnerVolumeSpecName "kube-api-access-mxmzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:35:17.015565 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:17.015522    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edb22358-f20f-4ec0-96ca-4bd54f6d4090-kserve-provision-location\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:35:17.015565 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:17.015557    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxmzl\" (UniqueName: \"kubernetes.io/projected/edb22358-f20f-4ec0-96ca-4bd54f6d4090-kube-api-access-mxmzl\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:35:17.015565 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:17.015568    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/edb22358-f20f-4ec0-96ca-4bd54f6d4090-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:35:17.015804 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:17.015578    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/edb22358-f20f-4ec0-96ca-4bd54f6d4090-proxy-tls\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:35:17.238743 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:17.238702    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8" event={"ID":"5c209106-cb79-4124-8808-78b1acb753c7","Type":"ContainerStarted","Data":"ce9b2d9a361dfd89f9627a075079a4907a613fd32d80498998fbb3b5689106cd"}
Jun 02 16:35:17.239176 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:17.238754    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8" event={"ID":"5c209106-cb79-4124-8808-78b1acb753c7","Type":"ContainerStarted","Data":"2d0242c2aeda403452317847c5a2de2865af5f6b96c7ecf289e7dbb76b1d6761"}
Jun 02 16:35:17.239176 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:17.239008    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8"
Jun 02 16:35:17.239176 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:17.239071    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8"
Jun 02 16:35:17.240488 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:17.240463    2579 generic.go:358] "Generic (PLEG): container finished" podID="edb22358-f20f-4ec0-96ca-4bd54f6d4090" containerID="ec39ce23fdd54b535be4b5c29bac18074093bf5da38397185642eae6d0fcd824" exitCode=0
Jun 02 16:35:17.240598 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:17.240527    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf" event={"ID":"edb22358-f20f-4ec0-96ca-4bd54f6d4090","Type":"ContainerDied","Data":"ec39ce23fdd54b535be4b5c29bac18074093bf5da38397185642eae6d0fcd824"}
Jun 02 16:35:17.240598 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:17.240534    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf"
Jun 02 16:35:17.240598 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:17.240551    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf" event={"ID":"edb22358-f20f-4ec0-96ca-4bd54f6d4090","Type":"ContainerDied","Data":"7f12142ec5b846be854b77d0a1adf8d56850090b92c667cc358cd050335cffef"}
Jun 02 16:35:17.240598 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:17.240566    2579 scope.go:117] "RemoveContainer" containerID="f71924036df30ffe950657bf0d6270ba22989e04ec5b67dde10a943d76965c8f"
Jun 02 16:35:17.248913 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:17.248894    2579 scope.go:117] "RemoveContainer" containerID="ec39ce23fdd54b535be4b5c29bac18074093bf5da38397185642eae6d0fcd824"
Jun 02 16:35:17.256266 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:17.256246    2579 scope.go:117] "RemoveContainer" containerID="e392b7b8dcc3fba4b643819961e901f73d2eb251257e11e7a26d166d9689443a"
Jun 02 16:35:17.259671 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:17.259623    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8" podStartSLOduration=6.259606719 podStartE2EDuration="6.259606719s" podCreationTimestamp="2026-06-02 16:35:11 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:35:17.258148838 +0000 UTC m=+2925.784057270" watchObservedRunningTime="2026-06-02 16:35:17.259606719 +0000 UTC m=+2925.785515151"
Jun 02 16:35:17.263427 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:17.263392    2579 scope.go:117] "RemoveContainer" containerID="f71924036df30ffe950657bf0d6270ba22989e04ec5b67dde10a943d76965c8f"
Jun 02 16:35:17.263685 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:35:17.263666    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f71924036df30ffe950657bf0d6270ba22989e04ec5b67dde10a943d76965c8f\": container with ID starting with f71924036df30ffe950657bf0d6270ba22989e04ec5b67dde10a943d76965c8f not found: ID does not exist" containerID="f71924036df30ffe950657bf0d6270ba22989e04ec5b67dde10a943d76965c8f"
Jun 02 16:35:17.263734 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:17.263695    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f71924036df30ffe950657bf0d6270ba22989e04ec5b67dde10a943d76965c8f"} err="failed to get container status \"f71924036df30ffe950657bf0d6270ba22989e04ec5b67dde10a943d76965c8f\": rpc error: code = NotFound desc = could not find container \"f71924036df30ffe950657bf0d6270ba22989e04ec5b67dde10a943d76965c8f\": container with ID starting with f71924036df30ffe950657bf0d6270ba22989e04ec5b67dde10a943d76965c8f not found: ID does not exist"
Jun 02 16:35:17.263734 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:17.263714    2579 scope.go:117] "RemoveContainer" containerID="ec39ce23fdd54b535be4b5c29bac18074093bf5da38397185642eae6d0fcd824"
Jun 02 16:35:17.263952 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:35:17.263935    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec39ce23fdd54b535be4b5c29bac18074093bf5da38397185642eae6d0fcd824\": container with ID starting with ec39ce23fdd54b535be4b5c29bac18074093bf5da38397185642eae6d0fcd824 not found: ID does not exist" containerID="ec39ce23fdd54b535be4b5c29bac18074093bf5da38397185642eae6d0fcd824"
Jun 02 16:35:17.264017 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:17.263959    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec39ce23fdd54b535be4b5c29bac18074093bf5da38397185642eae6d0fcd824"} err="failed to get container status \"ec39ce23fdd54b535be4b5c29bac18074093bf5da38397185642eae6d0fcd824\": rpc error: code = NotFound desc = could not find container \"ec39ce23fdd54b535be4b5c29bac18074093bf5da38397185642eae6d0fcd824\": container with ID starting with ec39ce23fdd54b535be4b5c29bac18074093bf5da38397185642eae6d0fcd824 not found: ID does not exist"
Jun 02 16:35:17.264017 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:17.263974    2579 scope.go:117] "RemoveContainer" containerID="e392b7b8dcc3fba4b643819961e901f73d2eb251257e11e7a26d166d9689443a"
Jun 02 16:35:17.264186 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:35:17.264169    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e392b7b8dcc3fba4b643819961e901f73d2eb251257e11e7a26d166d9689443a\": container with ID starting with e392b7b8dcc3fba4b643819961e901f73d2eb251257e11e7a26d166d9689443a not found: ID does not exist" containerID="e392b7b8dcc3fba4b643819961e901f73d2eb251257e11e7a26d166d9689443a"
Jun 02 16:35:17.264232 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:17.264191    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e392b7b8dcc3fba4b643819961e901f73d2eb251257e11e7a26d166d9689443a"} err="failed to get container status \"e392b7b8dcc3fba4b643819961e901f73d2eb251257e11e7a26d166d9689443a\": rpc error: code = NotFound desc = could not find container \"e392b7b8dcc3fba4b643819961e901f73d2eb251257e11e7a26d166d9689443a\": container with ID starting with e392b7b8dcc3fba4b643819961e901f73d2eb251257e11e7a26d166d9689443a not found: ID does not exist"
Jun 02 16:35:17.277433 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:17.274297    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf"]
Jun 02 16:35:17.280965 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:17.280932    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-z9nsf"]
Jun 02 16:35:18.062905 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:18.062871    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edb22358-f20f-4ec0-96ca-4bd54f6d4090" path="/var/lib/kubelet/pods/edb22358-f20f-4ec0-96ca-4bd54f6d4090/volumes"
Jun 02 16:35:23.250585 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:23.250551    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8"
Jun 02 16:35:53.251788 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:35:53.251702    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8" podUID="5c209106-cb79-4124-8808-78b1acb753c7" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.27:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.27:8080: connect: connection refused"
Jun 02 16:36:03.251604 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:36:03.251562    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8" podUID="5c209106-cb79-4124-8808-78b1acb753c7" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.27:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.27:8080: connect: connection refused"
Jun 02 16:36:13.251175 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:36:13.251139    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8" podUID="5c209106-cb79-4124-8808-78b1acb753c7" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.27:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.27:8080: connect: connection refused"
Jun 02 16:36:23.251798 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:36:23.251750    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8" podUID="5c209106-cb79-4124-8808-78b1acb753c7" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.27:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.27:8080: connect: connection refused"
Jun 02 16:36:32.097629 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:36:32.097599    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lqx9k_6127e33c-aa77-437e-b443-e956037afedb/ovn-acl-logging/0.log"
Jun 02 16:36:32.101927 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:36:32.101902    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lqx9k_6127e33c-aa77-437e-b443-e956037afedb/ovn-acl-logging/0.log"
Jun 02 16:36:33.255007 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:36:33.254972    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8"
Jun 02 16:36:41.662655 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:36:41.662624    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8"]
Jun 02 16:36:41.663171 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:36:41.662969    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8" podUID="5c209106-cb79-4124-8808-78b1acb753c7" containerName="kserve-container" containerID="cri-o://2d0242c2aeda403452317847c5a2de2865af5f6b96c7ecf289e7dbb76b1d6761" gracePeriod=30
Jun 02 16:36:41.663171 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:36:41.663005    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8" podUID="5c209106-cb79-4124-8808-78b1acb753c7" containerName="kube-rbac-proxy" containerID="cri-o://ce9b2d9a361dfd89f9627a075079a4907a613fd32d80498998fbb3b5689106cd" gracePeriod=30
Jun 02 16:36:42.480615 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:36:42.480576    2579 generic.go:358] "Generic (PLEG): container finished" podID="5c209106-cb79-4124-8808-78b1acb753c7" containerID="ce9b2d9a361dfd89f9627a075079a4907a613fd32d80498998fbb3b5689106cd" exitCode=2
Jun 02 16:36:42.480800 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:36:42.480652    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8" event={"ID":"5c209106-cb79-4124-8808-78b1acb753c7","Type":"ContainerDied","Data":"ce9b2d9a361dfd89f9627a075079a4907a613fd32d80498998fbb3b5689106cd"}
Jun 02 16:36:43.245375 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:36:43.245333    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8" podUID="5c209106-cb79-4124-8808-78b1acb753c7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Jun 02 16:36:43.251934 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:36:43.251905    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8" podUID="5c209106-cb79-4124-8808-78b1acb753c7" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.27:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.27:8080: connect: connection refused"
Jun 02 16:36:48.004574 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:36:48.004549    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8"
Jun 02 16:36:48.109084 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:36:48.109043    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c209106-cb79-4124-8808-78b1acb753c7-proxy-tls\") pod \"5c209106-cb79-4124-8808-78b1acb753c7\" (UID: \"5c209106-cb79-4124-8808-78b1acb753c7\") "
Jun 02 16:36:48.109289 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:36:48.109125    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsdxb\" (UniqueName: \"kubernetes.io/projected/5c209106-cb79-4124-8808-78b1acb753c7-kube-api-access-gsdxb\") pod \"5c209106-cb79-4124-8808-78b1acb753c7\" (UID: \"5c209106-cb79-4124-8808-78b1acb753c7\") "
Jun 02 16:36:48.109289 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:36:48.109152    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c209106-cb79-4124-8808-78b1acb753c7-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") pod \"5c209106-cb79-4124-8808-78b1acb753c7\" (UID: \"5c209106-cb79-4124-8808-78b1acb753c7\") "
Jun 02 16:36:48.109289 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:36:48.109169    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c209106-cb79-4124-8808-78b1acb753c7-kserve-provision-location\") pod \"5c209106-cb79-4124-8808-78b1acb753c7\" (UID: \"5c209106-cb79-4124-8808-78b1acb753c7\") "
Jun 02 16:36:48.109602 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:36:48.109568    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c209106-cb79-4124-8808-78b1acb753c7-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config") pod "5c209106-cb79-4124-8808-78b1acb753c7" (UID: "5c209106-cb79-4124-8808-78b1acb753c7"). InnerVolumeSpecName "isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:36:48.109746 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:36:48.109619    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c209106-cb79-4124-8808-78b1acb753c7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5c209106-cb79-4124-8808-78b1acb753c7" (UID: "5c209106-cb79-4124-8808-78b1acb753c7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:36:48.111571 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:36:48.111549    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c209106-cb79-4124-8808-78b1acb753c7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5c209106-cb79-4124-8808-78b1acb753c7" (UID: "5c209106-cb79-4124-8808-78b1acb753c7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:36:48.111703 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:36:48.111687    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c209106-cb79-4124-8808-78b1acb753c7-kube-api-access-gsdxb" (OuterVolumeSpecName: "kube-api-access-gsdxb") pod "5c209106-cb79-4124-8808-78b1acb753c7" (UID: "5c209106-cb79-4124-8808-78b1acb753c7"). InnerVolumeSpecName "kube-api-access-gsdxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:36:48.210630 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:36:48.210535    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c209106-cb79-4124-8808-78b1acb753c7-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:36:48.210630 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:36:48.210568    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c209106-cb79-4124-8808-78b1acb753c7-kserve-provision-location\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:36:48.210630 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:36:48.210580    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c209106-cb79-4124-8808-78b1acb753c7-proxy-tls\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:36:48.210630 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:36:48.210590    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsdxb\" (UniqueName: \"kubernetes.io/projected/5c209106-cb79-4124-8808-78b1acb753c7-kube-api-access-gsdxb\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:36:48.497657 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:36:48.497618    2579 generic.go:358] "Generic (PLEG): container finished" podID="5c209106-cb79-4124-8808-78b1acb753c7" containerID="2d0242c2aeda403452317847c5a2de2865af5f6b96c7ecf289e7dbb76b1d6761" exitCode=0
Jun 02 16:36:48.497866 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:36:48.497665    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8" event={"ID":"5c209106-cb79-4124-8808-78b1acb753c7","Type":"ContainerDied","Data":"2d0242c2aeda403452317847c5a2de2865af5f6b96c7ecf289e7dbb76b1d6761"}
Jun 02 16:36:48.497866 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:36:48.497698    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8" event={"ID":"5c209106-cb79-4124-8808-78b1acb753c7","Type":"ContainerDied","Data":"d8c8d2bb93f520a1d2aec1d0845d5017eba6a118f65ece70a465afbce8ab5ca1"}
Jun 02 16:36:48.497866 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:36:48.497711    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8"
Jun 02 16:36:48.497866 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:36:48.497719    2579 scope.go:117] "RemoveContainer" containerID="ce9b2d9a361dfd89f9627a075079a4907a613fd32d80498998fbb3b5689106cd"
Jun 02 16:36:48.506173 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:36:48.506153    2579 scope.go:117] "RemoveContainer" containerID="2d0242c2aeda403452317847c5a2de2865af5f6b96c7ecf289e7dbb76b1d6761"
Jun 02 16:36:48.513194 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:36:48.513174    2579 scope.go:117] "RemoveContainer" containerID="3517a0450f10fa8ac107fcc27d32bc175e209b1573715d8baa4e9c6601ca2979"
Jun 02 16:36:48.520194 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:36:48.520169    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8"]
Jun 02 16:36:48.520555 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:36:48.520529    2579 scope.go:117] "RemoveContainer" containerID="ce9b2d9a361dfd89f9627a075079a4907a613fd32d80498998fbb3b5689106cd"
Jun 02 16:36:48.520945 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:36:48.520926    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce9b2d9a361dfd89f9627a075079a4907a613fd32d80498998fbb3b5689106cd\": container with ID starting with ce9b2d9a361dfd89f9627a075079a4907a613fd32d80498998fbb3b5689106cd not found: ID does not exist" containerID="ce9b2d9a361dfd89f9627a075079a4907a613fd32d80498998fbb3b5689106cd"
Jun 02 16:36:48.521022 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:36:48.520958    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce9b2d9a361dfd89f9627a075079a4907a613fd32d80498998fbb3b5689106cd"} err="failed to get container status \"ce9b2d9a361dfd89f9627a075079a4907a613fd32d80498998fbb3b5689106cd\": rpc error: code = NotFound desc = could not find container \"ce9b2d9a361dfd89f9627a075079a4907a613fd32d80498998fbb3b5689106cd\": container with ID starting with ce9b2d9a361dfd89f9627a075079a4907a613fd32d80498998fbb3b5689106cd not found: ID does not exist"
Jun 02 16:36:48.521022 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:36:48.520980    2579 scope.go:117] "RemoveContainer" containerID="2d0242c2aeda403452317847c5a2de2865af5f6b96c7ecf289e7dbb76b1d6761"
Jun 02 16:36:48.521253 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:36:48.521234    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d0242c2aeda403452317847c5a2de2865af5f6b96c7ecf289e7dbb76b1d6761\": container with ID starting with 2d0242c2aeda403452317847c5a2de2865af5f6b96c7ecf289e7dbb76b1d6761 not found: ID does not exist" containerID="2d0242c2aeda403452317847c5a2de2865af5f6b96c7ecf289e7dbb76b1d6761"
Jun 02 16:36:48.521294 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:36:48.521259    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d0242c2aeda403452317847c5a2de2865af5f6b96c7ecf289e7dbb76b1d6761"} err="failed to get container status \"2d0242c2aeda403452317847c5a2de2865af5f6b96c7ecf289e7dbb76b1d6761\": rpc error: code = NotFound desc = could not find container \"2d0242c2aeda403452317847c5a2de2865af5f6b96c7ecf289e7dbb76b1d6761\": container with ID starting with 2d0242c2aeda403452317847c5a2de2865af5f6b96c7ecf289e7dbb76b1d6761 not found: ID does not exist"
Jun 02 16:36:48.521294 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:36:48.521276    2579 scope.go:117] "RemoveContainer" containerID="3517a0450f10fa8ac107fcc27d32bc175e209b1573715d8baa4e9c6601ca2979"
Jun 02 16:36:48.521566 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:36:48.521547    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3517a0450f10fa8ac107fcc27d32bc175e209b1573715d8baa4e9c6601ca2979\": container with ID starting with 3517a0450f10fa8ac107fcc27d32bc175e209b1573715d8baa4e9c6601ca2979 not found: ID does not exist" containerID="3517a0450f10fa8ac107fcc27d32bc175e209b1573715d8baa4e9c6601ca2979"
Jun 02 16:36:48.521642 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:36:48.521578    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3517a0450f10fa8ac107fcc27d32bc175e209b1573715d8baa4e9c6601ca2979"} err="failed to get container status \"3517a0450f10fa8ac107fcc27d32bc175e209b1573715d8baa4e9c6601ca2979\": rpc error: code = NotFound desc = could not find container \"3517a0450f10fa8ac107fcc27d32bc175e209b1573715d8baa4e9c6601ca2979\": container with ID starting with 3517a0450f10fa8ac107fcc27d32bc175e209b1573715d8baa4e9c6601ca2979 not found: ID does not exist"
Jun 02 16:36:48.523842 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:36:48.523821    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hwvn8"]
Jun 02 16:36:50.067013 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:36:50.063937    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c209106-cb79-4124-8808-78b1acb753c7" path="/var/lib/kubelet/pods/5c209106-cb79-4124-8808-78b1acb753c7/volumes"
Jun 02 16:41:32.115772 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:41:32.115744    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lqx9k_6127e33c-aa77-437e-b443-e956037afedb/ovn-acl-logging/0.log"
Jun 02 16:41:32.119742 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:41:32.119721    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lqx9k_6127e33c-aa77-437e-b443-e956037afedb/ovn-acl-logging/0.log"
Jun 02 16:43:22.127169 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:22.127072    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n"]
Jun 02 16:43:22.127652 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:22.127345    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edb22358-f20f-4ec0-96ca-4bd54f6d4090" containerName="storage-initializer"
Jun 02 16:43:22.127652 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:22.127357    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="edb22358-f20f-4ec0-96ca-4bd54f6d4090" containerName="storage-initializer"
Jun 02 16:43:22.127652 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:22.127365    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edb22358-f20f-4ec0-96ca-4bd54f6d4090" containerName="kube-rbac-proxy"
Jun 02 16:43:22.127652 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:22.127371    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="edb22358-f20f-4ec0-96ca-4bd54f6d4090" containerName="kube-rbac-proxy"
Jun 02 16:43:22.127652 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:22.127377    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c209106-cb79-4124-8808-78b1acb753c7" containerName="storage-initializer"
Jun 02 16:43:22.127652 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:22.127383    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c209106-cb79-4124-8808-78b1acb753c7" containerName="storage-initializer"
Jun 02 16:43:22.127652 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:22.127396    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edb22358-f20f-4ec0-96ca-4bd54f6d4090" containerName="kserve-container"
Jun 02 16:43:22.127652 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:22.127416    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="edb22358-f20f-4ec0-96ca-4bd54f6d4090" containerName="kserve-container"
Jun 02 16:43:22.127652 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:22.127427    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c209106-cb79-4124-8808-78b1acb753c7" containerName="kserve-container"
Jun 02 16:43:22.127652 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:22.127432    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c209106-cb79-4124-8808-78b1acb753c7" containerName="kserve-container"
Jun 02 16:43:22.127652 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:22.127438    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c209106-cb79-4124-8808-78b1acb753c7" containerName="kube-rbac-proxy"
Jun 02 16:43:22.127652 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:22.127443    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c209106-cb79-4124-8808-78b1acb753c7" containerName="kube-rbac-proxy"
Jun 02 16:43:22.127652 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:22.127485    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="edb22358-f20f-4ec0-96ca-4bd54f6d4090" containerName="kube-rbac-proxy"
Jun 02 16:43:22.127652 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:22.127491    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="5c209106-cb79-4124-8808-78b1acb753c7" containerName="kube-rbac-proxy"
Jun 02 16:43:22.127652 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:22.127497    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="5c209106-cb79-4124-8808-78b1acb753c7" containerName="kserve-container"
Jun 02 16:43:22.127652 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:22.127503    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="edb22358-f20f-4ec0-96ca-4bd54f6d4090" containerName="kserve-container"
Jun 02 16:43:22.130472 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:22.130451    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n"
Jun 02 16:43:22.133254 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:22.133229    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jun 02 16:43:22.133362 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:22.133271    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Jun 02 16:43:22.133362 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:22.133278    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ztrcr\""
Jun 02 16:43:22.133362 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:22.133287    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jun 02 16:43:22.133362 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:22.133242    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Jun 02 16:43:22.141374 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:22.141351    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n"]
Jun 02 16:43:22.214333 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:22.214292    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d22107e-40c7-421d-b9d5-fa40ff1657dd-kserve-provision-location\") pod \"isvc-tensorflow-predictor-6756f669d7-b9j7n\" (UID: \"9d22107e-40c7-421d-b9d5-fa40ff1657dd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n"
Jun 02 16:43:22.214333 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:22.214333    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d6rf\" (UniqueName: \"kubernetes.io/projected/9d22107e-40c7-421d-b9d5-fa40ff1657dd-kube-api-access-7d6rf\") pod \"isvc-tensorflow-predictor-6756f669d7-b9j7n\" (UID: \"9d22107e-40c7-421d-b9d5-fa40ff1657dd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n"
Jun 02 16:43:22.214602 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:22.214355    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d22107e-40c7-421d-b9d5-fa40ff1657dd-isvc-tensorflow-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-6756f669d7-b9j7n\" (UID: \"9d22107e-40c7-421d-b9d5-fa40ff1657dd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n"
Jun 02 16:43:22.214602 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:22.214455    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d22107e-40c7-421d-b9d5-fa40ff1657dd-proxy-tls\") pod \"isvc-tensorflow-predictor-6756f669d7-b9j7n\" (UID: \"9d22107e-40c7-421d-b9d5-fa40ff1657dd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n"
Jun 02 16:43:22.315641 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:22.315607    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d22107e-40c7-421d-b9d5-fa40ff1657dd-kserve-provision-location\") pod \"isvc-tensorflow-predictor-6756f669d7-b9j7n\" (UID: \"9d22107e-40c7-421d-b9d5-fa40ff1657dd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n"
Jun 02 16:43:22.315641 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:22.315644    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7d6rf\" (UniqueName: \"kubernetes.io/projected/9d22107e-40c7-421d-b9d5-fa40ff1657dd-kube-api-access-7d6rf\") pod \"isvc-tensorflow-predictor-6756f669d7-b9j7n\" (UID: \"9d22107e-40c7-421d-b9d5-fa40ff1657dd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n"
Jun 02 16:43:22.315819 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:22.315666    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d22107e-40c7-421d-b9d5-fa40ff1657dd-isvc-tensorflow-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-6756f669d7-b9j7n\" (UID: \"9d22107e-40c7-421d-b9d5-fa40ff1657dd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n"
Jun 02 16:43:22.315819 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:22.315689    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d22107e-40c7-421d-b9d5-fa40ff1657dd-proxy-tls\") pod \"isvc-tensorflow-predictor-6756f669d7-b9j7n\" (UID: \"9d22107e-40c7-421d-b9d5-fa40ff1657dd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n"
Jun 02 16:43:22.315819 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:43:22.315784    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-predictor-serving-cert: secret "isvc-tensorflow-predictor-serving-cert" not found
Jun 02 16:43:22.315938 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:43:22.315871    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d22107e-40c7-421d-b9d5-fa40ff1657dd-proxy-tls podName:9d22107e-40c7-421d-b9d5-fa40ff1657dd nodeName:}" failed. No retries permitted until 2026-06-02 16:43:22.815835534 +0000 UTC m=+3411.341743959 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9d22107e-40c7-421d-b9d5-fa40ff1657dd-proxy-tls") pod "isvc-tensorflow-predictor-6756f669d7-b9j7n" (UID: "9d22107e-40c7-421d-b9d5-fa40ff1657dd") : secret "isvc-tensorflow-predictor-serving-cert" not found
Jun 02 16:43:22.316075 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:22.316057    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d22107e-40c7-421d-b9d5-fa40ff1657dd-kserve-provision-location\") pod \"isvc-tensorflow-predictor-6756f669d7-b9j7n\" (UID: \"9d22107e-40c7-421d-b9d5-fa40ff1657dd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n"
Jun 02 16:43:22.316299 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:22.316282    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d22107e-40c7-421d-b9d5-fa40ff1657dd-isvc-tensorflow-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-6756f669d7-b9j7n\" (UID: \"9d22107e-40c7-421d-b9d5-fa40ff1657dd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n"
Jun 02 16:43:22.324156 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:22.324133    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d6rf\" (UniqueName: \"kubernetes.io/projected/9d22107e-40c7-421d-b9d5-fa40ff1657dd-kube-api-access-7d6rf\") pod \"isvc-tensorflow-predictor-6756f669d7-b9j7n\" (UID: \"9d22107e-40c7-421d-b9d5-fa40ff1657dd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n"
Jun 02 16:43:22.820290 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:22.820243    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d22107e-40c7-421d-b9d5-fa40ff1657dd-proxy-tls\") pod \"isvc-tensorflow-predictor-6756f669d7-b9j7n\" (UID: \"9d22107e-40c7-421d-b9d5-fa40ff1657dd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n"
Jun 02 16:43:22.822882 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:22.822855    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d22107e-40c7-421d-b9d5-fa40ff1657dd-proxy-tls\") pod \"isvc-tensorflow-predictor-6756f669d7-b9j7n\" (UID: \"9d22107e-40c7-421d-b9d5-fa40ff1657dd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n"
Jun 02 16:43:23.041718 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:23.041671    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n"
Jun 02 16:43:23.175069 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:23.175027    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n"]
Jun 02 16:43:23.178126 ip-10-0-129-156 kubenswrapper[2579]: W0602 16:43:23.178087    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d22107e_40c7_421d_b9d5_fa40ff1657dd.slice/crio-cd2f8436dab3492ecf7e3fef45e3e90985b0457ee0db02e00732414f8278af9d WatchSource:0}: Error finding container cd2f8436dab3492ecf7e3fef45e3e90985b0457ee0db02e00732414f8278af9d: Status 404 returned error can't find the container with id cd2f8436dab3492ecf7e3fef45e3e90985b0457ee0db02e00732414f8278af9d
Jun 02 16:43:23.180030 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:23.180012    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 16:43:23.562706 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:23.562666    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n" event={"ID":"9d22107e-40c7-421d-b9d5-fa40ff1657dd","Type":"ContainerStarted","Data":"602a96d4678f418965cda8c4552bab2146426e1dfa58a2806c6cbc7775aaac7b"}
Jun 02 16:43:23.562875 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:23.562712    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n" event={"ID":"9d22107e-40c7-421d-b9d5-fa40ff1657dd","Type":"ContainerStarted","Data":"cd2f8436dab3492ecf7e3fef45e3e90985b0457ee0db02e00732414f8278af9d"}
Jun 02 16:43:28.576156 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:28.576123    2579 generic.go:358] "Generic (PLEG): container finished" podID="9d22107e-40c7-421d-b9d5-fa40ff1657dd" containerID="602a96d4678f418965cda8c4552bab2146426e1dfa58a2806c6cbc7775aaac7b" exitCode=0
Jun 02 16:43:28.576568 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:28.576161    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n" event={"ID":"9d22107e-40c7-421d-b9d5-fa40ff1657dd","Type":"ContainerDied","Data":"602a96d4678f418965cda8c4552bab2146426e1dfa58a2806c6cbc7775aaac7b"}
Jun 02 16:43:33.593003 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:33.592965    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n" event={"ID":"9d22107e-40c7-421d-b9d5-fa40ff1657dd","Type":"ContainerStarted","Data":"145af398b173adccf8dc1ca500711cf78f24d385d132234820228a22c9dee908"}
Jun 02 16:43:33.593496 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:33.593012    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n" event={"ID":"9d22107e-40c7-421d-b9d5-fa40ff1657dd","Type":"ContainerStarted","Data":"0a1c845b581a9a90229cdc0ec9ab14f8fd6335d0829efa318e4e0f0976765536"}
Jun 02 16:43:33.593496 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:33.593245    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n"
Jun 02 16:43:33.611722 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:33.611672    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n" podStartSLOduration=7.391895845 podStartE2EDuration="11.611657688s" podCreationTimestamp="2026-06-02 16:43:22 +0000 UTC" firstStartedPulling="2026-06-02 16:43:28.577359625 +0000 UTC m=+3417.103268035" lastFinishedPulling="2026-06-02 16:43:32.79712147 +0000 UTC m=+3421.323029878" observedRunningTime="2026-06-02 16:43:33.610455831 +0000 UTC m=+3422.136364253" watchObservedRunningTime="2026-06-02 16:43:33.611657688 +0000 UTC m=+3422.137566117"
Jun 02 16:43:34.595975 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:34.595942    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n"
Jun 02 16:43:34.597045 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:34.597024    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n" podUID="9d22107e-40c7-421d-b9d5-fa40ff1657dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jun 02 16:43:35.599083 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:35.599036    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n" podUID="9d22107e-40c7-421d-b9d5-fa40ff1657dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jun 02 16:43:40.604170 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:40.604139    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n"
Jun 02 16:43:40.604672 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:40.604644    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n" podUID="9d22107e-40c7-421d-b9d5-fa40ff1657dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jun 02 16:43:50.604811 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:43:50.604784    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n"
Jun 02 16:44:03.817070 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:03.817026    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n"]
Jun 02 16:44:03.817603 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:03.817477    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n" podUID="9d22107e-40c7-421d-b9d5-fa40ff1657dd" containerName="kserve-container" containerID="cri-o://0a1c845b581a9a90229cdc0ec9ab14f8fd6335d0829efa318e4e0f0976765536" gracePeriod=30
Jun 02 16:44:03.817603 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:03.817551    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n" podUID="9d22107e-40c7-421d-b9d5-fa40ff1657dd" containerName="kube-rbac-proxy" containerID="cri-o://145af398b173adccf8dc1ca500711cf78f24d385d132234820228a22c9dee908" gracePeriod=30
Jun 02 16:44:03.896216 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:03.896176    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j"]
Jun 02 16:44:03.899813 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:03.899790    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j"
Jun 02 16:44:03.901908 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:03.901879    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Jun 02 16:44:03.902035 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:03.901975    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Jun 02 16:44:03.908436 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:03.908393    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j"]
Jun 02 16:44:03.950895 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:03.950863    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6bbb1e76-f916-4edb-9f9f-8a3dc4075950-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j\" (UID: \"6bbb1e76-f916-4edb-9f9f-8a3dc4075950\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j"
Jun 02 16:44:03.951031 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:03.950902    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6bbb1e76-f916-4edb-9f9f-8a3dc4075950-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j\" (UID: \"6bbb1e76-f916-4edb-9f9f-8a3dc4075950\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j"
Jun 02 16:44:03.951031 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:03.950964    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phcvv\" (UniqueName: \"kubernetes.io/projected/6bbb1e76-f916-4edb-9f9f-8a3dc4075950-kube-api-access-phcvv\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j\" (UID: \"6bbb1e76-f916-4edb-9f9f-8a3dc4075950\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j"
Jun 02 16:44:03.951031 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:03.951005    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6bbb1e76-f916-4edb-9f9f-8a3dc4075950-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j\" (UID: \"6bbb1e76-f916-4edb-9f9f-8a3dc4075950\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j"
Jun 02 16:44:04.051665 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:04.051632    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6bbb1e76-f916-4edb-9f9f-8a3dc4075950-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j\" (UID: \"6bbb1e76-f916-4edb-9f9f-8a3dc4075950\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j"
Jun 02 16:44:04.051836 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:04.051685    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6bbb1e76-f916-4edb-9f9f-8a3dc4075950-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j\" (UID: \"6bbb1e76-f916-4edb-9f9f-8a3dc4075950\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j"
Jun 02 16:44:04.051836 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:04.051731    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-phcvv\" (UniqueName: \"kubernetes.io/projected/6bbb1e76-f916-4edb-9f9f-8a3dc4075950-kube-api-access-phcvv\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j\" (UID: \"6bbb1e76-f916-4edb-9f9f-8a3dc4075950\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j"
Jun 02 16:44:04.051836 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:04.051767    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6bbb1e76-f916-4edb-9f9f-8a3dc4075950-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j\" (UID: \"6bbb1e76-f916-4edb-9f9f-8a3dc4075950\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j"
Jun 02 16:44:04.052222 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:04.052195    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6bbb1e76-f916-4edb-9f9f-8a3dc4075950-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j\" (UID: \"6bbb1e76-f916-4edb-9f9f-8a3dc4075950\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j"
Jun 02 16:44:04.052414 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:04.052380    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6bbb1e76-f916-4edb-9f9f-8a3dc4075950-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j\" (UID: \"6bbb1e76-f916-4edb-9f9f-8a3dc4075950\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j"
Jun 02 16:44:04.054356 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:04.054338    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6bbb1e76-f916-4edb-9f9f-8a3dc4075950-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j\" (UID: \"6bbb1e76-f916-4edb-9f9f-8a3dc4075950\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j"
Jun 02 16:44:04.059704 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:04.059676    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phcvv\" (UniqueName: \"kubernetes.io/projected/6bbb1e76-f916-4edb-9f9f-8a3dc4075950-kube-api-access-phcvv\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j\" (UID: \"6bbb1e76-f916-4edb-9f9f-8a3dc4075950\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j"
Jun 02 16:44:04.210864 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:04.210777    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j"
Jun 02 16:44:04.336259 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:04.336235    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j"]
Jun 02 16:44:04.338876 ip-10-0-129-156 kubenswrapper[2579]: W0602 16:44:04.338842    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bbb1e76_f916_4edb_9f9f_8a3dc4075950.slice/crio-3a0b70daa032eda7c244cd02c4cec14a107634a35177a6387512f7dfd7b3c9f0 WatchSource:0}: Error finding container 3a0b70daa032eda7c244cd02c4cec14a107634a35177a6387512f7dfd7b3c9f0: Status 404 returned error can't find the container with id 3a0b70daa032eda7c244cd02c4cec14a107634a35177a6387512f7dfd7b3c9f0
Jun 02 16:44:04.682208 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:04.682172    2579 generic.go:358] "Generic (PLEG): container finished" podID="9d22107e-40c7-421d-b9d5-fa40ff1657dd" containerID="145af398b173adccf8dc1ca500711cf78f24d385d132234820228a22c9dee908" exitCode=2
Jun 02 16:44:04.682431 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:04.682251    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n" event={"ID":"9d22107e-40c7-421d-b9d5-fa40ff1657dd","Type":"ContainerDied","Data":"145af398b173adccf8dc1ca500711cf78f24d385d132234820228a22c9dee908"}
Jun 02 16:44:04.683571 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:04.683538    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j" event={"ID":"6bbb1e76-f916-4edb-9f9f-8a3dc4075950","Type":"ContainerStarted","Data":"ae3e170003e61cfc1597ab0621613ad5670ec32ff1fde40cadb845e4975f06a0"}
Jun 02 16:44:04.683689 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:04.683577    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j" event={"ID":"6bbb1e76-f916-4edb-9f9f-8a3dc4075950","Type":"ContainerStarted","Data":"3a0b70daa032eda7c244cd02c4cec14a107634a35177a6387512f7dfd7b3c9f0"}
Jun 02 16:44:05.599694 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:05.599651    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n" podUID="9d22107e-40c7-421d-b9d5-fa40ff1657dd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jun 02 16:44:08.695972 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:08.695933    2579 generic.go:358] "Generic (PLEG): container finished" podID="6bbb1e76-f916-4edb-9f9f-8a3dc4075950" containerID="ae3e170003e61cfc1597ab0621613ad5670ec32ff1fde40cadb845e4975f06a0" exitCode=0
Jun 02 16:44:08.696337 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:08.696009    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j" event={"ID":"6bbb1e76-f916-4edb-9f9f-8a3dc4075950","Type":"ContainerDied","Data":"ae3e170003e61cfc1597ab0621613ad5670ec32ff1fde40cadb845e4975f06a0"}
Jun 02 16:44:09.701217 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:09.701184    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j" event={"ID":"6bbb1e76-f916-4edb-9f9f-8a3dc4075950","Type":"ContainerStarted","Data":"0bc70654b45d64c460f889af801cc89832d5d374d0f01eaa27186a160334b532"}
Jun 02 16:44:09.701217 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:09.701222    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j" event={"ID":"6bbb1e76-f916-4edb-9f9f-8a3dc4075950","Type":"ContainerStarted","Data":"5c5758da356a2b745f52df88ef218892b684066e4ed4a14948d9028f8a96dc75"}
Jun 02 16:44:09.701651 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:09.701494    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j"
Jun 02 16:44:09.719534 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:09.719487    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j" podStartSLOduration=6.719473685 podStartE2EDuration="6.719473685s" podCreationTimestamp="2026-06-02 16:44:03 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:44:09.717396028 +0000 UTC m=+3458.243304457" watchObservedRunningTime="2026-06-02 16:44:09.719473685 +0000 UTC m=+3458.245382114"
Jun 02 16:44:10.599858 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:10.599807    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n" podUID="9d22107e-40c7-421d-b9d5-fa40ff1657dd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jun 02 16:44:10.703973 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:10.703933    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j"
Jun 02 16:44:10.705145 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:10.705120    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j" podUID="6bbb1e76-f916-4edb-9f9f-8a3dc4075950" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jun 02 16:44:11.711622 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:11.711581    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j" podUID="6bbb1e76-f916-4edb-9f9f-8a3dc4075950" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jun 02 16:44:15.599515 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:15.599471    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n" podUID="9d22107e-40c7-421d-b9d5-fa40ff1657dd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jun 02 16:44:15.599984 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:15.599603    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n"
Jun 02 16:44:16.713179 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:16.713146    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j"
Jun 02 16:44:16.713802 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:16.713772    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j" podUID="6bbb1e76-f916-4edb-9f9f-8a3dc4075950" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jun 02 16:44:20.600347 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:20.600302    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n" podUID="9d22107e-40c7-421d-b9d5-fa40ff1657dd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jun 02 16:44:25.599519 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:25.599479    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n" podUID="9d22107e-40c7-421d-b9d5-fa40ff1657dd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jun 02 16:44:26.714713 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:26.714682    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j"
Jun 02 16:44:30.599560 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:30.599498    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n" podUID="9d22107e-40c7-421d-b9d5-fa40ff1657dd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jun 02 16:44:34.460016 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:34.459990    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n"
Jun 02 16:44:34.578870 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:34.578811    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d22107e-40c7-421d-b9d5-fa40ff1657dd-kserve-provision-location\") pod \"9d22107e-40c7-421d-b9d5-fa40ff1657dd\" (UID: \"9d22107e-40c7-421d-b9d5-fa40ff1657dd\") "
Jun 02 16:44:34.578870 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:34.578872    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7d6rf\" (UniqueName: \"kubernetes.io/projected/9d22107e-40c7-421d-b9d5-fa40ff1657dd-kube-api-access-7d6rf\") pod \"9d22107e-40c7-421d-b9d5-fa40ff1657dd\" (UID: \"9d22107e-40c7-421d-b9d5-fa40ff1657dd\") "
Jun 02 16:44:34.579110 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:34.578928    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d22107e-40c7-421d-b9d5-fa40ff1657dd-proxy-tls\") pod \"9d22107e-40c7-421d-b9d5-fa40ff1657dd\" (UID: \"9d22107e-40c7-421d-b9d5-fa40ff1657dd\") "
Jun 02 16:44:34.579110 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:34.578969    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d22107e-40c7-421d-b9d5-fa40ff1657dd-isvc-tensorflow-kube-rbac-proxy-sar-config\") pod \"9d22107e-40c7-421d-b9d5-fa40ff1657dd\" (UID: \"9d22107e-40c7-421d-b9d5-fa40ff1657dd\") "
Jun 02 16:44:34.579444 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:34.579384    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d22107e-40c7-421d-b9d5-fa40ff1657dd-isvc-tensorflow-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-tensorflow-kube-rbac-proxy-sar-config") pod "9d22107e-40c7-421d-b9d5-fa40ff1657dd" (UID: "9d22107e-40c7-421d-b9d5-fa40ff1657dd"). InnerVolumeSpecName "isvc-tensorflow-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:44:34.581276 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:34.581252    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d22107e-40c7-421d-b9d5-fa40ff1657dd-kube-api-access-7d6rf" (OuterVolumeSpecName: "kube-api-access-7d6rf") pod "9d22107e-40c7-421d-b9d5-fa40ff1657dd" (UID: "9d22107e-40c7-421d-b9d5-fa40ff1657dd"). InnerVolumeSpecName "kube-api-access-7d6rf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:44:34.581481 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:34.581456    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d22107e-40c7-421d-b9d5-fa40ff1657dd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9d22107e-40c7-421d-b9d5-fa40ff1657dd" (UID: "9d22107e-40c7-421d-b9d5-fa40ff1657dd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:44:34.590353 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:34.590328    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d22107e-40c7-421d-b9d5-fa40ff1657dd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9d22107e-40c7-421d-b9d5-fa40ff1657dd" (UID: "9d22107e-40c7-421d-b9d5-fa40ff1657dd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:44:34.679960 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:34.679916    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d22107e-40c7-421d-b9d5-fa40ff1657dd-proxy-tls\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:44:34.679960 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:34.679951    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d22107e-40c7-421d-b9d5-fa40ff1657dd-isvc-tensorflow-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:44:34.679960 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:34.679962    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d22107e-40c7-421d-b9d5-fa40ff1657dd-kserve-provision-location\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:44:34.680169 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:34.679974    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d6rf\" (UniqueName: \"kubernetes.io/projected/9d22107e-40c7-421d-b9d5-fa40ff1657dd-kube-api-access-7d6rf\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:44:34.776076 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:34.776041    2579 generic.go:358] "Generic (PLEG): container finished" podID="9d22107e-40c7-421d-b9d5-fa40ff1657dd" containerID="0a1c845b581a9a90229cdc0ec9ab14f8fd6335d0829efa318e4e0f0976765536" exitCode=137
Jun 02 16:44:34.776273 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:34.776111    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n" event={"ID":"9d22107e-40c7-421d-b9d5-fa40ff1657dd","Type":"ContainerDied","Data":"0a1c845b581a9a90229cdc0ec9ab14f8fd6335d0829efa318e4e0f0976765536"}
Jun 02 16:44:34.776273 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:34.776129    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n"
Jun 02 16:44:34.776273 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:34.776147    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n" event={"ID":"9d22107e-40c7-421d-b9d5-fa40ff1657dd","Type":"ContainerDied","Data":"cd2f8436dab3492ecf7e3fef45e3e90985b0457ee0db02e00732414f8278af9d"}
Jun 02 16:44:34.776273 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:34.776163    2579 scope.go:117] "RemoveContainer" containerID="145af398b173adccf8dc1ca500711cf78f24d385d132234820228a22c9dee908"
Jun 02 16:44:34.785010 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:34.784994    2579 scope.go:117] "RemoveContainer" containerID="0a1c845b581a9a90229cdc0ec9ab14f8fd6335d0829efa318e4e0f0976765536"
Jun 02 16:44:34.791867 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:34.791845    2579 scope.go:117] "RemoveContainer" containerID="602a96d4678f418965cda8c4552bab2146426e1dfa58a2806c6cbc7775aaac7b"
Jun 02 16:44:34.797294 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:34.797271    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n"]
Jun 02 16:44:34.799465 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:34.799442    2579 scope.go:117] "RemoveContainer" containerID="145af398b173adccf8dc1ca500711cf78f24d385d132234820228a22c9dee908"
Jun 02 16:44:34.799857 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:44:34.799824    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"145af398b173adccf8dc1ca500711cf78f24d385d132234820228a22c9dee908\": container with ID starting with 145af398b173adccf8dc1ca500711cf78f24d385d132234820228a22c9dee908 not found: ID does not exist" containerID="145af398b173adccf8dc1ca500711cf78f24d385d132234820228a22c9dee908"
Jun 02 16:44:34.799949 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:34.799868    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"145af398b173adccf8dc1ca500711cf78f24d385d132234820228a22c9dee908"} err="failed to get container status \"145af398b173adccf8dc1ca500711cf78f24d385d132234820228a22c9dee908\": rpc error: code = NotFound desc = could not find container \"145af398b173adccf8dc1ca500711cf78f24d385d132234820228a22c9dee908\": container with ID starting with 145af398b173adccf8dc1ca500711cf78f24d385d132234820228a22c9dee908 not found: ID does not exist"
Jun 02 16:44:34.799949 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:34.799893    2579 scope.go:117] "RemoveContainer" containerID="0a1c845b581a9a90229cdc0ec9ab14f8fd6335d0829efa318e4e0f0976765536"
Jun 02 16:44:34.800174 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:44:34.800155    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a1c845b581a9a90229cdc0ec9ab14f8fd6335d0829efa318e4e0f0976765536\": container with ID starting with 0a1c845b581a9a90229cdc0ec9ab14f8fd6335d0829efa318e4e0f0976765536 not found: ID does not exist" containerID="0a1c845b581a9a90229cdc0ec9ab14f8fd6335d0829efa318e4e0f0976765536"
Jun 02 16:44:34.800230 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:34.800181    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a1c845b581a9a90229cdc0ec9ab14f8fd6335d0829efa318e4e0f0976765536"} err="failed to get container status \"0a1c845b581a9a90229cdc0ec9ab14f8fd6335d0829efa318e4e0f0976765536\": rpc error: code = NotFound desc = could not find container \"0a1c845b581a9a90229cdc0ec9ab14f8fd6335d0829efa318e4e0f0976765536\": container with ID starting with 0a1c845b581a9a90229cdc0ec9ab14f8fd6335d0829efa318e4e0f0976765536 not found: ID does not exist"
Jun 02 16:44:34.800230 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:34.800200    2579 scope.go:117] "RemoveContainer" containerID="602a96d4678f418965cda8c4552bab2146426e1dfa58a2806c6cbc7775aaac7b"
Jun 02 16:44:34.800547 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:44:34.800519    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"602a96d4678f418965cda8c4552bab2146426e1dfa58a2806c6cbc7775aaac7b\": container with ID starting with 602a96d4678f418965cda8c4552bab2146426e1dfa58a2806c6cbc7775aaac7b not found: ID does not exist" containerID="602a96d4678f418965cda8c4552bab2146426e1dfa58a2806c6cbc7775aaac7b"
Jun 02 16:44:34.800609 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:34.800555    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"602a96d4678f418965cda8c4552bab2146426e1dfa58a2806c6cbc7775aaac7b"} err="failed to get container status \"602a96d4678f418965cda8c4552bab2146426e1dfa58a2806c6cbc7775aaac7b\": rpc error: code = NotFound desc = could not find container \"602a96d4678f418965cda8c4552bab2146426e1dfa58a2806c6cbc7775aaac7b\": container with ID starting with 602a96d4678f418965cda8c4552bab2146426e1dfa58a2806c6cbc7775aaac7b not found: ID does not exist"
Jun 02 16:44:34.801021 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:34.801003    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-b9j7n"]
Jun 02 16:44:36.063939 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:36.063905    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d22107e-40c7-421d-b9d5-fa40ff1657dd" path="/var/lib/kubelet/pods/9d22107e-40c7-421d-b9d5-fa40ff1657dd/volumes"
Jun 02 16:44:45.713703 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:45.713651    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j"]
Jun 02 16:44:45.714256 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:45.714205    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j" podUID="6bbb1e76-f916-4edb-9f9f-8a3dc4075950" containerName="kserve-container" containerID="cri-o://5c5758da356a2b745f52df88ef218892b684066e4ed4a14948d9028f8a96dc75" gracePeriod=30
Jun 02 16:44:45.714443 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:45.714256    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j" podUID="6bbb1e76-f916-4edb-9f9f-8a3dc4075950" containerName="kube-rbac-proxy" containerID="cri-o://0bc70654b45d64c460f889af801cc89832d5d374d0f01eaa27186a160334b532" gracePeriod=30
Jun 02 16:44:45.862530 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:45.862496    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-vft25"]
Jun 02 16:44:45.862799 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:45.862787    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d22107e-40c7-421d-b9d5-fa40ff1657dd" containerName="kube-rbac-proxy"
Jun 02 16:44:45.862858 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:45.862801    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d22107e-40c7-421d-b9d5-fa40ff1657dd" containerName="kube-rbac-proxy"
Jun 02 16:44:45.862858 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:45.862810    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d22107e-40c7-421d-b9d5-fa40ff1657dd" containerName="storage-initializer"
Jun 02 16:44:45.862858 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:45.862816    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d22107e-40c7-421d-b9d5-fa40ff1657dd" containerName="storage-initializer"
Jun 02 16:44:45.862858 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:45.862836    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d22107e-40c7-421d-b9d5-fa40ff1657dd" containerName="kserve-container"
Jun 02 16:44:45.862858 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:45.862842    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d22107e-40c7-421d-b9d5-fa40ff1657dd" containerName="kserve-container"
Jun 02 16:44:45.862995 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:45.862897    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="9d22107e-40c7-421d-b9d5-fa40ff1657dd" containerName="kube-rbac-proxy"
Jun 02 16:44:45.862995 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:45.862907    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="9d22107e-40c7-421d-b9d5-fa40ff1657dd" containerName="kserve-container"
Jun 02 16:44:45.865768 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:45.865748    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-vft25"
Jun 02 16:44:45.867887 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:45.867864    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Jun 02 16:44:45.868012 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:45.867907    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Jun 02 16:44:45.876982 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:45.876955    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-vft25"]
Jun 02 16:44:45.973217 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:45.973123    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6b8q\" (UniqueName: \"kubernetes.io/projected/70dc6c21-474e-488b-b145-07a571ce4f2a-kube-api-access-m6b8q\") pod \"isvc-triton-predictor-84bb65d94b-vft25\" (UID: \"70dc6c21-474e-488b-b145-07a571ce4f2a\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-vft25"
Jun 02 16:44:45.973217 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:45.973187    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70dc6c21-474e-488b-b145-07a571ce4f2a-kserve-provision-location\") pod \"isvc-triton-predictor-84bb65d94b-vft25\" (UID: \"70dc6c21-474e-488b-b145-07a571ce4f2a\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-vft25"
Jun 02 16:44:45.973437 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:45.973236    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70dc6c21-474e-488b-b145-07a571ce4f2a-proxy-tls\") pod \"isvc-triton-predictor-84bb65d94b-vft25\" (UID: \"70dc6c21-474e-488b-b145-07a571ce4f2a\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-vft25"
Jun 02 16:44:45.973437 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:45.973310    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70dc6c21-474e-488b-b145-07a571ce4f2a-isvc-triton-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-84bb65d94b-vft25\" (UID: \"70dc6c21-474e-488b-b145-07a571ce4f2a\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-vft25"
Jun 02 16:44:46.074641 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:46.074604    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70dc6c21-474e-488b-b145-07a571ce4f2a-kserve-provision-location\") pod \"isvc-triton-predictor-84bb65d94b-vft25\" (UID: \"70dc6c21-474e-488b-b145-07a571ce4f2a\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-vft25"
Jun 02 16:44:46.074641 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:46.074646    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70dc6c21-474e-488b-b145-07a571ce4f2a-proxy-tls\") pod \"isvc-triton-predictor-84bb65d94b-vft25\" (UID: \"70dc6c21-474e-488b-b145-07a571ce4f2a\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-vft25"
Jun 02 16:44:46.074913 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:46.074770    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70dc6c21-474e-488b-b145-07a571ce4f2a-isvc-triton-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-84bb65d94b-vft25\" (UID: \"70dc6c21-474e-488b-b145-07a571ce4f2a\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-vft25"
Jun 02 16:44:46.074913 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:46.074818    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-m6b8q\" (UniqueName: \"kubernetes.io/projected/70dc6c21-474e-488b-b145-07a571ce4f2a-kube-api-access-m6b8q\") pod \"isvc-triton-predictor-84bb65d94b-vft25\" (UID: \"70dc6c21-474e-488b-b145-07a571ce4f2a\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-vft25"
Jun 02 16:44:46.075080 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:46.075062    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70dc6c21-474e-488b-b145-07a571ce4f2a-kserve-provision-location\") pod \"isvc-triton-predictor-84bb65d94b-vft25\" (UID: \"70dc6c21-474e-488b-b145-07a571ce4f2a\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-vft25"
Jun 02 16:44:46.075503 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:46.075487    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70dc6c21-474e-488b-b145-07a571ce4f2a-isvc-triton-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-84bb65d94b-vft25\" (UID: \"70dc6c21-474e-488b-b145-07a571ce4f2a\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-vft25"
Jun 02 16:44:46.077410 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:46.077381    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70dc6c21-474e-488b-b145-07a571ce4f2a-proxy-tls\") pod \"isvc-triton-predictor-84bb65d94b-vft25\" (UID: \"70dc6c21-474e-488b-b145-07a571ce4f2a\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-vft25"
Jun 02 16:44:46.082966 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:46.082930    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6b8q\" (UniqueName: \"kubernetes.io/projected/70dc6c21-474e-488b-b145-07a571ce4f2a-kube-api-access-m6b8q\") pod \"isvc-triton-predictor-84bb65d94b-vft25\" (UID: \"70dc6c21-474e-488b-b145-07a571ce4f2a\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-vft25"
Jun 02 16:44:46.175746 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:46.175702    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-vft25"
Jun 02 16:44:46.305924 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:46.305896    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-vft25"]
Jun 02 16:44:46.308485 ip-10-0-129-156 kubenswrapper[2579]: W0602 16:44:46.308460    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70dc6c21_474e_488b_b145_07a571ce4f2a.slice/crio-a5e0c36ada056603faa2630eb34fa0f1e8bc96f0967d8b7c616bc70bea242049 WatchSource:0}: Error finding container a5e0c36ada056603faa2630eb34fa0f1e8bc96f0967d8b7c616bc70bea242049: Status 404 returned error can't find the container with id a5e0c36ada056603faa2630eb34fa0f1e8bc96f0967d8b7c616bc70bea242049
Jun 02 16:44:46.709952 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:46.709902    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j" podUID="6bbb1e76-f916-4edb-9f9f-8a3dc4075950" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jun 02 16:44:46.818501 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:46.818463    2579 generic.go:358] "Generic (PLEG): container finished" podID="6bbb1e76-f916-4edb-9f9f-8a3dc4075950" containerID="0bc70654b45d64c460f889af801cc89832d5d374d0f01eaa27186a160334b532" exitCode=2
Jun 02 16:44:46.818932 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:46.818547    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j" event={"ID":"6bbb1e76-f916-4edb-9f9f-8a3dc4075950","Type":"ContainerDied","Data":"0bc70654b45d64c460f889af801cc89832d5d374d0f01eaa27186a160334b532"}
Jun 02 16:44:46.819918 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:46.819894    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-vft25" event={"ID":"70dc6c21-474e-488b-b145-07a571ce4f2a","Type":"ContainerStarted","Data":"5ec5b7673b090453d8025969f734ca12573a85bb83870767e0bd5eb10d2a5541"}
Jun 02 16:44:46.820018 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:46.819925    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-vft25" event={"ID":"70dc6c21-474e-488b-b145-07a571ce4f2a","Type":"ContainerStarted","Data":"a5e0c36ada056603faa2630eb34fa0f1e8bc96f0967d8b7c616bc70bea242049"}
Jun 02 16:44:50.833049 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:50.833014    2579 generic.go:358] "Generic (PLEG): container finished" podID="70dc6c21-474e-488b-b145-07a571ce4f2a" containerID="5ec5b7673b090453d8025969f734ca12573a85bb83870767e0bd5eb10d2a5541" exitCode=0
Jun 02 16:44:50.833461 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:50.833059    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-vft25" event={"ID":"70dc6c21-474e-488b-b145-07a571ce4f2a","Type":"ContainerDied","Data":"5ec5b7673b090453d8025969f734ca12573a85bb83870767e0bd5eb10d2a5541"}
Jun 02 16:44:51.709917 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:51.709875    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j" podUID="6bbb1e76-f916-4edb-9f9f-8a3dc4075950" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jun 02 16:44:56.710456 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:56.709821    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j" podUID="6bbb1e76-f916-4edb-9f9f-8a3dc4075950" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jun 02 16:44:56.710456 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:44:56.710066    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j"
Jun 02 16:45:01.709200 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:45:01.709062    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j" podUID="6bbb1e76-f916-4edb-9f9f-8a3dc4075950" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jun 02 16:45:06.709930 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:45:06.709874    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j" podUID="6bbb1e76-f916-4edb-9f9f-8a3dc4075950" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jun 02 16:45:11.709888 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:45:11.709822    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j" podUID="6bbb1e76-f916-4edb-9f9f-8a3dc4075950" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jun 02 16:45:15.942479 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:45:15.942436    2579 generic.go:358] "Generic (PLEG): container finished" podID="6bbb1e76-f916-4edb-9f9f-8a3dc4075950" containerID="5c5758da356a2b745f52df88ef218892b684066e4ed4a14948d9028f8a96dc75" exitCode=137
Jun 02 16:45:15.943012 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:45:15.942522    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j" event={"ID":"6bbb1e76-f916-4edb-9f9f-8a3dc4075950","Type":"ContainerDied","Data":"5c5758da356a2b745f52df88ef218892b684066e4ed4a14948d9028f8a96dc75"}
Jun 02 16:45:16.423078 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:45:16.423051    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j"
Jun 02 16:45:16.461971 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:45:16.461934    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6bbb1e76-f916-4edb-9f9f-8a3dc4075950-proxy-tls\") pod \"6bbb1e76-f916-4edb-9f9f-8a3dc4075950\" (UID: \"6bbb1e76-f916-4edb-9f9f-8a3dc4075950\") "
Jun 02 16:45:16.462155 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:45:16.462051    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6bbb1e76-f916-4edb-9f9f-8a3dc4075950-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") pod \"6bbb1e76-f916-4edb-9f9f-8a3dc4075950\" (UID: \"6bbb1e76-f916-4edb-9f9f-8a3dc4075950\") "
Jun 02 16:45:16.462155 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:45:16.462091    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6bbb1e76-f916-4edb-9f9f-8a3dc4075950-kserve-provision-location\") pod \"6bbb1e76-f916-4edb-9f9f-8a3dc4075950\" (UID: \"6bbb1e76-f916-4edb-9f9f-8a3dc4075950\") "
Jun 02 16:45:16.462155 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:45:16.462121    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phcvv\" (UniqueName: \"kubernetes.io/projected/6bbb1e76-f916-4edb-9f9f-8a3dc4075950-kube-api-access-phcvv\") pod \"6bbb1e76-f916-4edb-9f9f-8a3dc4075950\" (UID: \"6bbb1e76-f916-4edb-9f9f-8a3dc4075950\") "
Jun 02 16:45:16.462634 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:45:16.462606    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bbb1e76-f916-4edb-9f9f-8a3dc4075950-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-tensorflow-runtime-kube-rbac-proxy-sar-config") pod "6bbb1e76-f916-4edb-9f9f-8a3dc4075950" (UID: "6bbb1e76-f916-4edb-9f9f-8a3dc4075950"). InnerVolumeSpecName "isvc-tensorflow-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:45:16.464941 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:45:16.464912    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bbb1e76-f916-4edb-9f9f-8a3dc4075950-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6bbb1e76-f916-4edb-9f9f-8a3dc4075950" (UID: "6bbb1e76-f916-4edb-9f9f-8a3dc4075950"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:45:16.465890 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:45:16.465863    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bbb1e76-f916-4edb-9f9f-8a3dc4075950-kube-api-access-phcvv" (OuterVolumeSpecName: "kube-api-access-phcvv") pod "6bbb1e76-f916-4edb-9f9f-8a3dc4075950" (UID: "6bbb1e76-f916-4edb-9f9f-8a3dc4075950"). InnerVolumeSpecName "kube-api-access-phcvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:45:16.477219 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:45:16.477186    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bbb1e76-f916-4edb-9f9f-8a3dc4075950-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6bbb1e76-f916-4edb-9f9f-8a3dc4075950" (UID: "6bbb1e76-f916-4edb-9f9f-8a3dc4075950"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:45:16.563642 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:45:16.563544    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6bbb1e76-f916-4edb-9f9f-8a3dc4075950-proxy-tls\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:45:16.563642 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:45:16.563580    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6bbb1e76-f916-4edb-9f9f-8a3dc4075950-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:45:16.563642 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:45:16.563591    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6bbb1e76-f916-4edb-9f9f-8a3dc4075950-kserve-provision-location\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:45:16.563642 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:45:16.563601    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phcvv\" (UniqueName: \"kubernetes.io/projected/6bbb1e76-f916-4edb-9f9f-8a3dc4075950-kube-api-access-phcvv\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:45:16.948196 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:45:16.948110    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j" event={"ID":"6bbb1e76-f916-4edb-9f9f-8a3dc4075950","Type":"ContainerDied","Data":"3a0b70daa032eda7c244cd02c4cec14a107634a35177a6387512f7dfd7b3c9f0"}
Jun 02 16:45:16.948196 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:45:16.948130    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j"
Jun 02 16:45:16.948196 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:45:16.948167    2579 scope.go:117] "RemoveContainer" containerID="0bc70654b45d64c460f889af801cc89832d5d374d0f01eaa27186a160334b532"
Jun 02 16:45:16.959100 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:45:16.959072    2579 scope.go:117] "RemoveContainer" containerID="5c5758da356a2b745f52df88ef218892b684066e4ed4a14948d9028f8a96dc75"
Jun 02 16:45:16.969196 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:45:16.969177    2579 scope.go:117] "RemoveContainer" containerID="ae3e170003e61cfc1597ab0621613ad5670ec32ff1fde40cadb845e4975f06a0"
Jun 02 16:45:16.972439 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:45:16.972394    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j"]
Jun 02 16:45:16.976792 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:45:16.976767    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-gqf4j"]
Jun 02 16:45:18.064433 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:45:18.064179    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bbb1e76-f916-4edb-9f9f-8a3dc4075950" path="/var/lib/kubelet/pods/6bbb1e76-f916-4edb-9f9f-8a3dc4075950/volumes"
Jun 02 16:46:43.055880 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:43.055847    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lqx9k_6127e33c-aa77-437e-b443-e956037afedb/ovn-acl-logging/0.log"
Jun 02 16:46:43.055880 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:43.055847    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lqx9k_6127e33c-aa77-437e-b443-e956037afedb/ovn-acl-logging/0.log"
Jun 02 16:46:44.223512 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:44.223421    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-vft25" event={"ID":"70dc6c21-474e-488b-b145-07a571ce4f2a","Type":"ContainerStarted","Data":"63bc3e2be2e8341ee7d7868ecb2774c1a72474e59e5c9fe3c6a894c321b72269"}
Jun 02 16:46:44.223512 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:44.223467    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-vft25" event={"ID":"70dc6c21-474e-488b-b145-07a571ce4f2a","Type":"ContainerStarted","Data":"1b18d8bd9663f97cd8987316fc73357f9b8d79a59cc0a0aa69707a8ef1293a31"}
Jun 02 16:46:44.223975 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:44.223560    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-vft25"
Jun 02 16:46:44.243441 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:44.243368    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-vft25" podStartSLOduration=6.242368039 podStartE2EDuration="1m59.243352914s" podCreationTimestamp="2026-06-02 16:44:45 +0000 UTC" firstStartedPulling="2026-06-02 16:44:50.834105805 +0000 UTC m=+3499.360014213" lastFinishedPulling="2026-06-02 16:46:43.835090665 +0000 UTC m=+3612.360999088" observedRunningTime="2026-06-02 16:46:44.240977783 +0000 UTC m=+3612.766886213" watchObservedRunningTime="2026-06-02 16:46:44.243352914 +0000 UTC m=+3612.769261343"
Jun 02 16:46:45.226652 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:45.226621    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-vft25"
Jun 02 16:46:45.228035 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:45.228008    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-vft25" podUID="70dc6c21-474e-488b-b145-07a571ce4f2a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jun 02 16:46:46.228799 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:46.228760    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-vft25" podUID="70dc6c21-474e-488b-b145-07a571ce4f2a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jun 02 16:46:51.232823 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:51.232794    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-vft25"
Jun 02 16:46:51.233495 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:51.233471    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-vft25"
Jun 02 16:46:58.351090 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:58.351050    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-vft25"]
Jun 02 16:46:58.351596 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:58.351486    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-vft25" podUID="70dc6c21-474e-488b-b145-07a571ce4f2a" containerName="kserve-container" containerID="cri-o://1b18d8bd9663f97cd8987316fc73357f9b8d79a59cc0a0aa69707a8ef1293a31" gracePeriod=30
Jun 02 16:46:58.351596 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:58.351534    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-vft25" podUID="70dc6c21-474e-488b-b145-07a571ce4f2a" containerName="kube-rbac-proxy" containerID="cri-o://63bc3e2be2e8341ee7d7868ecb2774c1a72474e59e5c9fe3c6a894c321b72269" gracePeriod=30
Jun 02 16:46:58.454191 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:58.454152    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn"]
Jun 02 16:46:58.454453 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:58.454440    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bbb1e76-f916-4edb-9f9f-8a3dc4075950" containerName="kube-rbac-proxy"
Jun 02 16:46:58.454510 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:58.454455    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bbb1e76-f916-4edb-9f9f-8a3dc4075950" containerName="kube-rbac-proxy"
Jun 02 16:46:58.454510 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:58.454476    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bbb1e76-f916-4edb-9f9f-8a3dc4075950" containerName="kserve-container"
Jun 02 16:46:58.454510 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:58.454482    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bbb1e76-f916-4edb-9f9f-8a3dc4075950" containerName="kserve-container"
Jun 02 16:46:58.454510 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:58.454491    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bbb1e76-f916-4edb-9f9f-8a3dc4075950" containerName="storage-initializer"
Jun 02 16:46:58.454510 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:58.454496    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bbb1e76-f916-4edb-9f9f-8a3dc4075950" containerName="storage-initializer"
Jun 02 16:46:58.454662 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:58.454537    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="6bbb1e76-f916-4edb-9f9f-8a3dc4075950" containerName="kserve-container"
Jun 02 16:46:58.454662 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:58.454546    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="6bbb1e76-f916-4edb-9f9f-8a3dc4075950" containerName="kube-rbac-proxy"
Jun 02 16:46:58.471907 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:58.471877    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn"
Jun 02 16:46:58.474093 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:58.474071    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Jun 02 16:46:58.474202 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:58.474071    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Jun 02 16:46:58.476977 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:58.476950    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn"]
Jun 02 16:46:58.522840 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:58.522787    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dec8e82b-740c-430b-af85-593d3b35d298-kserve-provision-location\") pod \"isvc-xgboost-predictor-8689c4cfcc-mfkxn\" (UID: \"dec8e82b-740c-430b-af85-593d3b35d298\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn"
Jun 02 16:46:58.522840 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:58.522844    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dec8e82b-740c-430b-af85-593d3b35d298-proxy-tls\") pod \"isvc-xgboost-predictor-8689c4cfcc-mfkxn\" (UID: \"dec8e82b-740c-430b-af85-593d3b35d298\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn"
Jun 02 16:46:58.523048 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:58.522915    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dec8e82b-740c-430b-af85-593d3b35d298-isvc-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-8689c4cfcc-mfkxn\" (UID: \"dec8e82b-740c-430b-af85-593d3b35d298\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn"
Jun 02 16:46:58.523048 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:58.522969    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8f2g\" (UniqueName: \"kubernetes.io/projected/dec8e82b-740c-430b-af85-593d3b35d298-kube-api-access-l8f2g\") pod \"isvc-xgboost-predictor-8689c4cfcc-mfkxn\" (UID: \"dec8e82b-740c-430b-af85-593d3b35d298\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn"
Jun 02 16:46:58.624255 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:58.624168    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dec8e82b-740c-430b-af85-593d3b35d298-kserve-provision-location\") pod \"isvc-xgboost-predictor-8689c4cfcc-mfkxn\" (UID: \"dec8e82b-740c-430b-af85-593d3b35d298\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn"
Jun 02 16:46:58.624255 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:58.624212    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dec8e82b-740c-430b-af85-593d3b35d298-proxy-tls\") pod \"isvc-xgboost-predictor-8689c4cfcc-mfkxn\" (UID: \"dec8e82b-740c-430b-af85-593d3b35d298\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn"
Jun 02 16:46:58.624508 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:58.624330    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dec8e82b-740c-430b-af85-593d3b35d298-isvc-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-8689c4cfcc-mfkxn\" (UID: \"dec8e82b-740c-430b-af85-593d3b35d298\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn"
Jun 02 16:46:58.624508 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:58.624425    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-l8f2g\" (UniqueName: \"kubernetes.io/projected/dec8e82b-740c-430b-af85-593d3b35d298-kube-api-access-l8f2g\") pod \"isvc-xgboost-predictor-8689c4cfcc-mfkxn\" (UID: \"dec8e82b-740c-430b-af85-593d3b35d298\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn"
Jun 02 16:46:58.624609 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:58.624556    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dec8e82b-740c-430b-af85-593d3b35d298-kserve-provision-location\") pod \"isvc-xgboost-predictor-8689c4cfcc-mfkxn\" (UID: \"dec8e82b-740c-430b-af85-593d3b35d298\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn"
Jun 02 16:46:58.625024 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:58.625003    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dec8e82b-740c-430b-af85-593d3b35d298-isvc-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-8689c4cfcc-mfkxn\" (UID: \"dec8e82b-740c-430b-af85-593d3b35d298\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn"
Jun 02 16:46:58.626810 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:58.626792    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dec8e82b-740c-430b-af85-593d3b35d298-proxy-tls\") pod \"isvc-xgboost-predictor-8689c4cfcc-mfkxn\" (UID: \"dec8e82b-740c-430b-af85-593d3b35d298\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn"
Jun 02 16:46:58.633914 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:58.633893    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8f2g\" (UniqueName: \"kubernetes.io/projected/dec8e82b-740c-430b-af85-593d3b35d298-kube-api-access-l8f2g\") pod \"isvc-xgboost-predictor-8689c4cfcc-mfkxn\" (UID: \"dec8e82b-740c-430b-af85-593d3b35d298\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn"
Jun 02 16:46:58.782829 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:58.782781    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn"
Jun 02 16:46:58.911049 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:58.911023    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn"]
Jun 02 16:46:58.913496 ip-10-0-129-156 kubenswrapper[2579]: W0602 16:46:58.913464    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddec8e82b_740c_430b_af85_593d3b35d298.slice/crio-d84cb681b34eef4f75339089560777378975d682c8aad9f157e579287d1408c1 WatchSource:0}: Error finding container d84cb681b34eef4f75339089560777378975d682c8aad9f157e579287d1408c1: Status 404 returned error can't find the container with id d84cb681b34eef4f75339089560777378975d682c8aad9f157e579287d1408c1
Jun 02 16:46:59.263892 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:59.263857    2579 generic.go:358] "Generic (PLEG): container finished" podID="70dc6c21-474e-488b-b145-07a571ce4f2a" containerID="63bc3e2be2e8341ee7d7868ecb2774c1a72474e59e5c9fe3c6a894c321b72269" exitCode=2
Jun 02 16:46:59.264080 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:59.263942    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-vft25" event={"ID":"70dc6c21-474e-488b-b145-07a571ce4f2a","Type":"ContainerDied","Data":"63bc3e2be2e8341ee7d7868ecb2774c1a72474e59e5c9fe3c6a894c321b72269"}
Jun 02 16:46:59.265247 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:59.265225    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn" event={"ID":"dec8e82b-740c-430b-af85-593d3b35d298","Type":"ContainerStarted","Data":"d25b52d960783fb416948ed9fbdb428d2e5fcfafe519628012286e11f55f9556"}
Jun 02 16:46:59.265353 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:46:59.265256    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn" event={"ID":"dec8e82b-740c-430b-af85-593d3b35d298","Type":"ContainerStarted","Data":"d84cb681b34eef4f75339089560777378975d682c8aad9f157e579287d1408c1"}
Jun 02 16:47:00.902166 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:47:00.902139    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-vft25"
Jun 02 16:47:00.941556 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:47:00.941467    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6b8q\" (UniqueName: \"kubernetes.io/projected/70dc6c21-474e-488b-b145-07a571ce4f2a-kube-api-access-m6b8q\") pod \"70dc6c21-474e-488b-b145-07a571ce4f2a\" (UID: \"70dc6c21-474e-488b-b145-07a571ce4f2a\") "
Jun 02 16:47:00.941556 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:47:00.941519    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70dc6c21-474e-488b-b145-07a571ce4f2a-isvc-triton-kube-rbac-proxy-sar-config\") pod \"70dc6c21-474e-488b-b145-07a571ce4f2a\" (UID: \"70dc6c21-474e-488b-b145-07a571ce4f2a\") "
Jun 02 16:47:00.941802 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:47:00.941577    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70dc6c21-474e-488b-b145-07a571ce4f2a-kserve-provision-location\") pod \"70dc6c21-474e-488b-b145-07a571ce4f2a\" (UID: \"70dc6c21-474e-488b-b145-07a571ce4f2a\") "
Jun 02 16:47:00.941802 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:47:00.941622    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70dc6c21-474e-488b-b145-07a571ce4f2a-proxy-tls\") pod \"70dc6c21-474e-488b-b145-07a571ce4f2a\" (UID: \"70dc6c21-474e-488b-b145-07a571ce4f2a\") "
Jun 02 16:47:00.941944 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:47:00.941907    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70dc6c21-474e-488b-b145-07a571ce4f2a-isvc-triton-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-triton-kube-rbac-proxy-sar-config") pod "70dc6c21-474e-488b-b145-07a571ce4f2a" (UID: "70dc6c21-474e-488b-b145-07a571ce4f2a"). InnerVolumeSpecName "isvc-triton-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:47:00.942090 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:47:00.942063    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70dc6c21-474e-488b-b145-07a571ce4f2a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "70dc6c21-474e-488b-b145-07a571ce4f2a" (UID: "70dc6c21-474e-488b-b145-07a571ce4f2a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:47:00.943776 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:47:00.943754    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70dc6c21-474e-488b-b145-07a571ce4f2a-kube-api-access-m6b8q" (OuterVolumeSpecName: "kube-api-access-m6b8q") pod "70dc6c21-474e-488b-b145-07a571ce4f2a" (UID: "70dc6c21-474e-488b-b145-07a571ce4f2a"). InnerVolumeSpecName "kube-api-access-m6b8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:47:00.943856 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:47:00.943839    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70dc6c21-474e-488b-b145-07a571ce4f2a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "70dc6c21-474e-488b-b145-07a571ce4f2a" (UID: "70dc6c21-474e-488b-b145-07a571ce4f2a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:47:01.042656 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:47:01.042605    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6b8q\" (UniqueName: \"kubernetes.io/projected/70dc6c21-474e-488b-b145-07a571ce4f2a-kube-api-access-m6b8q\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:47:01.042656 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:47:01.042653    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70dc6c21-474e-488b-b145-07a571ce4f2a-isvc-triton-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:47:01.042656 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:47:01.042664    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70dc6c21-474e-488b-b145-07a571ce4f2a-kserve-provision-location\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:47:01.042656 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:47:01.042675    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70dc6c21-474e-488b-b145-07a571ce4f2a-proxy-tls\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:47:01.277589 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:47:01.277555    2579 generic.go:358] "Generic (PLEG): container finished" podID="70dc6c21-474e-488b-b145-07a571ce4f2a" containerID="1b18d8bd9663f97cd8987316fc73357f9b8d79a59cc0a0aa69707a8ef1293a31" exitCode=0
Jun 02 16:47:01.277787 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:47:01.277601    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-vft25" event={"ID":"70dc6c21-474e-488b-b145-07a571ce4f2a","Type":"ContainerDied","Data":"1b18d8bd9663f97cd8987316fc73357f9b8d79a59cc0a0aa69707a8ef1293a31"}
Jun 02 16:47:01.277787 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:47:01.277641    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-vft25"
Jun 02 16:47:01.277787 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:47:01.277656    2579 scope.go:117] "RemoveContainer" containerID="63bc3e2be2e8341ee7d7868ecb2774c1a72474e59e5c9fe3c6a894c321b72269"
Jun 02 16:47:01.277787 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:47:01.277645    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-vft25" event={"ID":"70dc6c21-474e-488b-b145-07a571ce4f2a","Type":"ContainerDied","Data":"a5e0c36ada056603faa2630eb34fa0f1e8bc96f0967d8b7c616bc70bea242049"}
Jun 02 16:47:01.286467 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:47:01.286445    2579 scope.go:117] "RemoveContainer" containerID="1b18d8bd9663f97cd8987316fc73357f9b8d79a59cc0a0aa69707a8ef1293a31"
Jun 02 16:47:01.293657 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:47:01.293637    2579 scope.go:117] "RemoveContainer" containerID="5ec5b7673b090453d8025969f734ca12573a85bb83870767e0bd5eb10d2a5541"
Jun 02 16:47:01.301188 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:47:01.301160    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-vft25"]
Jun 02 16:47:01.302216 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:47:01.302191    2579 scope.go:117] "RemoveContainer" containerID="63bc3e2be2e8341ee7d7868ecb2774c1a72474e59e5c9fe3c6a894c321b72269"
Jun 02 16:47:01.302796 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:47:01.302766    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63bc3e2be2e8341ee7d7868ecb2774c1a72474e59e5c9fe3c6a894c321b72269\": container with ID starting with 63bc3e2be2e8341ee7d7868ecb2774c1a72474e59e5c9fe3c6a894c321b72269 not found: ID does not exist" containerID="63bc3e2be2e8341ee7d7868ecb2774c1a72474e59e5c9fe3c6a894c321b72269"
Jun 02 16:47:01.303010 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:47:01.302809    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63bc3e2be2e8341ee7d7868ecb2774c1a72474e59e5c9fe3c6a894c321b72269"} err="failed to get container status \"63bc3e2be2e8341ee7d7868ecb2774c1a72474e59e5c9fe3c6a894c321b72269\": rpc error: code = NotFound desc = could not find container \"63bc3e2be2e8341ee7d7868ecb2774c1a72474e59e5c9fe3c6a894c321b72269\": container with ID starting with 63bc3e2be2e8341ee7d7868ecb2774c1a72474e59e5c9fe3c6a894c321b72269 not found: ID does not exist"
Jun 02 16:47:01.303010 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:47:01.302853    2579 scope.go:117] "RemoveContainer" containerID="1b18d8bd9663f97cd8987316fc73357f9b8d79a59cc0a0aa69707a8ef1293a31"
Jun 02 16:47:01.303172 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:47:01.303148    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b18d8bd9663f97cd8987316fc73357f9b8d79a59cc0a0aa69707a8ef1293a31\": container with ID starting with 1b18d8bd9663f97cd8987316fc73357f9b8d79a59cc0a0aa69707a8ef1293a31 not found: ID does not exist" containerID="1b18d8bd9663f97cd8987316fc73357f9b8d79a59cc0a0aa69707a8ef1293a31"
Jun 02 16:47:01.303229 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:47:01.303184    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b18d8bd9663f97cd8987316fc73357f9b8d79a59cc0a0aa69707a8ef1293a31"} err="failed to get container status \"1b18d8bd9663f97cd8987316fc73357f9b8d79a59cc0a0aa69707a8ef1293a31\": rpc error: code = NotFound desc = could not find container \"1b18d8bd9663f97cd8987316fc73357f9b8d79a59cc0a0aa69707a8ef1293a31\": container with ID starting with 1b18d8bd9663f97cd8987316fc73357f9b8d79a59cc0a0aa69707a8ef1293a31 not found: ID does not exist"
Jun 02 16:47:01.303229 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:47:01.303209    2579 scope.go:117] "RemoveContainer" containerID="5ec5b7673b090453d8025969f734ca12573a85bb83870767e0bd5eb10d2a5541"
Jun 02 16:47:01.303523 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:47:01.303503    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ec5b7673b090453d8025969f734ca12573a85bb83870767e0bd5eb10d2a5541\": container with ID starting with 5ec5b7673b090453d8025969f734ca12573a85bb83870767e0bd5eb10d2a5541 not found: ID does not exist" containerID="5ec5b7673b090453d8025969f734ca12573a85bb83870767e0bd5eb10d2a5541"
Jun 02 16:47:01.303615 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:47:01.303533    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ec5b7673b090453d8025969f734ca12573a85bb83870767e0bd5eb10d2a5541"} err="failed to get container status \"5ec5b7673b090453d8025969f734ca12573a85bb83870767e0bd5eb10d2a5541\": rpc error: code = NotFound desc = could not find container \"5ec5b7673b090453d8025969f734ca12573a85bb83870767e0bd5eb10d2a5541\": container with ID starting with 5ec5b7673b090453d8025969f734ca12573a85bb83870767e0bd5eb10d2a5541 not found: ID does not exist"
Jun 02 16:47:01.307029 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:47:01.306982    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-vft25"]
Jun 02 16:47:02.063421 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:47:02.063378    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70dc6c21-474e-488b-b145-07a571ce4f2a" path="/var/lib/kubelet/pods/70dc6c21-474e-488b-b145-07a571ce4f2a/volumes"
Jun 02 16:47:03.284497 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:47:03.284461    2579 generic.go:358] "Generic (PLEG): container finished" podID="dec8e82b-740c-430b-af85-593d3b35d298" containerID="d25b52d960783fb416948ed9fbdb428d2e5fcfafe519628012286e11f55f9556" exitCode=0
Jun 02 16:47:03.284837 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:47:03.284541    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn" event={"ID":"dec8e82b-740c-430b-af85-593d3b35d298","Type":"ContainerDied","Data":"d25b52d960783fb416948ed9fbdb428d2e5fcfafe519628012286e11f55f9556"}
Jun 02 16:47:23.346509 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:47:23.346473    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn" event={"ID":"dec8e82b-740c-430b-af85-593d3b35d298","Type":"ContainerStarted","Data":"3b1bde6333b6d459d779177e619e781afa122b4c8b4e5f6c5316458d819e619e"}
Jun 02 16:47:23.346509 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:47:23.346515    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn" event={"ID":"dec8e82b-740c-430b-af85-593d3b35d298","Type":"ContainerStarted","Data":"350887f4c9f6649bde5329562c0c6390ddf0d6fa779deb2dea50b2fc8a4bd902"}
Jun 02 16:47:23.347001 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:47:23.346722    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn"
Jun 02 16:47:24.349149 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:47:24.349106    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn"
Jun 02 16:47:24.350512 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:47:24.350488    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn" podUID="dec8e82b-740c-430b-af85-593d3b35d298" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jun 02 16:47:25.352338 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:47:25.352294    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn" podUID="dec8e82b-740c-430b-af85-593d3b35d298" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jun 02 16:47:30.356725 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:47:30.356695    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn"
Jun 02 16:47:30.357276 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:47:30.357250    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn" podUID="dec8e82b-740c-430b-af85-593d3b35d298" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jun 02 16:47:30.375300 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:47:30.375236    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn" podStartSLOduration=13.243457207 podStartE2EDuration="32.375220659s" podCreationTimestamp="2026-06-02 16:46:58 +0000 UTC" firstStartedPulling="2026-06-02 16:47:03.285843653 +0000 UTC m=+3631.811752062" lastFinishedPulling="2026-06-02 16:47:22.417607103 +0000 UTC m=+3650.943515514" observedRunningTime="2026-06-02 16:47:23.374735414 +0000 UTC m=+3651.900643844" watchObservedRunningTime="2026-06-02 16:47:30.375220659 +0000 UTC m=+3658.901129088"
Jun 02 16:47:40.357850 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:47:40.357808    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn" podUID="dec8e82b-740c-430b-af85-593d3b35d298" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jun 02 16:47:50.357923 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:47:50.357838    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn" podUID="dec8e82b-740c-430b-af85-593d3b35d298" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jun 02 16:48:00.357680 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:48:00.357634    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn" podUID="dec8e82b-740c-430b-af85-593d3b35d298" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jun 02 16:48:10.357581 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:48:10.357532    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn" podUID="dec8e82b-740c-430b-af85-593d3b35d298" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jun 02 16:48:20.357897 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:48:20.357857    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn" podUID="dec8e82b-740c-430b-af85-593d3b35d298" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jun 02 16:48:30.358568 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:48:30.358533    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn"
Jun 02 16:48:38.559303 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:48:38.559268    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn"]
Jun 02 16:48:38.559803 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:48:38.559715    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn" podUID="dec8e82b-740c-430b-af85-593d3b35d298" containerName="kserve-container" containerID="cri-o://350887f4c9f6649bde5329562c0c6390ddf0d6fa779deb2dea50b2fc8a4bd902" gracePeriod=30
Jun 02 16:48:38.559803 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:48:38.559760    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn" podUID="dec8e82b-740c-430b-af85-593d3b35d298" containerName="kube-rbac-proxy" containerID="cri-o://3b1bde6333b6d459d779177e619e781afa122b4c8b4e5f6c5316458d819e619e" gracePeriod=30
Jun 02 16:48:39.554982 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:48:39.554946    2579 generic.go:358] "Generic (PLEG): container finished" podID="dec8e82b-740c-430b-af85-593d3b35d298" containerID="3b1bde6333b6d459d779177e619e781afa122b4c8b4e5f6c5316458d819e619e" exitCode=2
Jun 02 16:48:39.555162 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:48:39.555023    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn" event={"ID":"dec8e82b-740c-430b-af85-593d3b35d298","Type":"ContainerDied","Data":"3b1bde6333b6d459d779177e619e781afa122b4c8b4e5f6c5316458d819e619e"}
Jun 02 16:48:40.353632 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:48:40.353578    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn" podUID="dec8e82b-740c-430b-af85-593d3b35d298" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jun 02 16:48:40.358012 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:48:40.357983    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn" podUID="dec8e82b-740c-430b-af85-593d3b35d298" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jun 02 16:48:42.565910 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:48:42.565873    2579 generic.go:358] "Generic (PLEG): container finished" podID="dec8e82b-740c-430b-af85-593d3b35d298" containerID="350887f4c9f6649bde5329562c0c6390ddf0d6fa779deb2dea50b2fc8a4bd902" exitCode=0
Jun 02 16:48:42.566284 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:48:42.565941    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn" event={"ID":"dec8e82b-740c-430b-af85-593d3b35d298","Type":"ContainerDied","Data":"350887f4c9f6649bde5329562c0c6390ddf0d6fa779deb2dea50b2fc8a4bd902"}
Jun 02 16:48:42.598865 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:48:42.598837    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn"
Jun 02 16:48:42.669127 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:48:42.669027    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dec8e82b-740c-430b-af85-593d3b35d298-kserve-provision-location\") pod \"dec8e82b-740c-430b-af85-593d3b35d298\" (UID: \"dec8e82b-740c-430b-af85-593d3b35d298\") "
Jun 02 16:48:42.669127 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:48:42.669068    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dec8e82b-740c-430b-af85-593d3b35d298-proxy-tls\") pod \"dec8e82b-740c-430b-af85-593d3b35d298\" (UID: \"dec8e82b-740c-430b-af85-593d3b35d298\") "
Jun 02 16:48:42.669307 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:48:42.669136    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8f2g\" (UniqueName: \"kubernetes.io/projected/dec8e82b-740c-430b-af85-593d3b35d298-kube-api-access-l8f2g\") pod \"dec8e82b-740c-430b-af85-593d3b35d298\" (UID: \"dec8e82b-740c-430b-af85-593d3b35d298\") "
Jun 02 16:48:42.669307 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:48:42.669170    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dec8e82b-740c-430b-af85-593d3b35d298-isvc-xgboost-kube-rbac-proxy-sar-config\") pod \"dec8e82b-740c-430b-af85-593d3b35d298\" (UID: \"dec8e82b-740c-430b-af85-593d3b35d298\") "
Jun 02 16:48:42.669432 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:48:42.669382    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec8e82b-740c-430b-af85-593d3b35d298-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dec8e82b-740c-430b-af85-593d3b35d298" (UID: "dec8e82b-740c-430b-af85-593d3b35d298"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:48:42.669595 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:48:42.669571    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dec8e82b-740c-430b-af85-593d3b35d298-isvc-xgboost-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-kube-rbac-proxy-sar-config") pod "dec8e82b-740c-430b-af85-593d3b35d298" (UID: "dec8e82b-740c-430b-af85-593d3b35d298"). InnerVolumeSpecName "isvc-xgboost-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:48:42.671435 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:48:42.671413    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dec8e82b-740c-430b-af85-593d3b35d298-kube-api-access-l8f2g" (OuterVolumeSpecName: "kube-api-access-l8f2g") pod "dec8e82b-740c-430b-af85-593d3b35d298" (UID: "dec8e82b-740c-430b-af85-593d3b35d298"). InnerVolumeSpecName "kube-api-access-l8f2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:48:42.671435 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:48:42.671424    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dec8e82b-740c-430b-af85-593d3b35d298-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dec8e82b-740c-430b-af85-593d3b35d298" (UID: "dec8e82b-740c-430b-af85-593d3b35d298"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:48:42.770736 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:48:42.770674    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dec8e82b-740c-430b-af85-593d3b35d298-kserve-provision-location\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:48:42.770736 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:48:42.770727    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dec8e82b-740c-430b-af85-593d3b35d298-proxy-tls\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:48:42.770736 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:48:42.770741    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8f2g\" (UniqueName: \"kubernetes.io/projected/dec8e82b-740c-430b-af85-593d3b35d298-kube-api-access-l8f2g\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:48:42.770996 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:48:42.770755    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dec8e82b-740c-430b-af85-593d3b35d298-isvc-xgboost-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:48:43.570558 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:48:43.570522    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn" event={"ID":"dec8e82b-740c-430b-af85-593d3b35d298","Type":"ContainerDied","Data":"d84cb681b34eef4f75339089560777378975d682c8aad9f157e579287d1408c1"}
Jun 02 16:48:43.571013 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:48:43.570571    2579 scope.go:117] "RemoveContainer" containerID="3b1bde6333b6d459d779177e619e781afa122b4c8b4e5f6c5316458d819e619e"
Jun 02 16:48:43.571013 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:48:43.570591    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn"
Jun 02 16:48:43.578702 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:48:43.578686    2579 scope.go:117] "RemoveContainer" containerID="350887f4c9f6649bde5329562c0c6390ddf0d6fa779deb2dea50b2fc8a4bd902"
Jun 02 16:48:43.585872 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:48:43.585850    2579 scope.go:117] "RemoveContainer" containerID="d25b52d960783fb416948ed9fbdb428d2e5fcfafe519628012286e11f55f9556"
Jun 02 16:48:43.595031 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:48:43.594999    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn"]
Jun 02 16:48:43.601540 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:48:43.601510    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-mfkxn"]
Jun 02 16:48:44.063511 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:48:44.063473    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dec8e82b-740c-430b-af85-593d3b35d298" path="/var/lib/kubelet/pods/dec8e82b-740c-430b-af85-593d3b35d298/volumes"
Jun 02 16:50:19.066667 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:19.066626    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz"]
Jun 02 16:50:19.068934 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:19.066883    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dec8e82b-740c-430b-af85-593d3b35d298" containerName="storage-initializer"
Jun 02 16:50:19.068934 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:19.066895    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dec8e82b-740c-430b-af85-593d3b35d298" containerName="storage-initializer"
Jun 02 16:50:19.068934 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:19.066907    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70dc6c21-474e-488b-b145-07a571ce4f2a" containerName="storage-initializer"
Jun 02 16:50:19.068934 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:19.066915    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="70dc6c21-474e-488b-b145-07a571ce4f2a" containerName="storage-initializer"
Jun 02 16:50:19.068934 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:19.066923    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dec8e82b-740c-430b-af85-593d3b35d298" containerName="kube-rbac-proxy"
Jun 02 16:50:19.068934 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:19.066930    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dec8e82b-740c-430b-af85-593d3b35d298" containerName="kube-rbac-proxy"
Jun 02 16:50:19.068934 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:19.066939    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70dc6c21-474e-488b-b145-07a571ce4f2a" containerName="kserve-container"
Jun 02 16:50:19.068934 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:19.066944    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="70dc6c21-474e-488b-b145-07a571ce4f2a" containerName="kserve-container"
Jun 02 16:50:19.068934 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:19.066950    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dec8e82b-740c-430b-af85-593d3b35d298" containerName="kserve-container"
Jun 02 16:50:19.068934 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:19.066955    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dec8e82b-740c-430b-af85-593d3b35d298" containerName="kserve-container"
Jun 02 16:50:19.068934 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:19.066965    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70dc6c21-474e-488b-b145-07a571ce4f2a" containerName="kube-rbac-proxy"
Jun 02 16:50:19.068934 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:19.066970    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="70dc6c21-474e-488b-b145-07a571ce4f2a" containerName="kube-rbac-proxy"
Jun 02 16:50:19.068934 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:19.067007    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="dec8e82b-740c-430b-af85-593d3b35d298" containerName="kube-rbac-proxy"
Jun 02 16:50:19.068934 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:19.067016    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="70dc6c21-474e-488b-b145-07a571ce4f2a" containerName="kserve-container"
Jun 02 16:50:19.068934 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:19.067022    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="70dc6c21-474e-488b-b145-07a571ce4f2a" containerName="kube-rbac-proxy"
Jun 02 16:50:19.068934 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:19.067027    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="dec8e82b-740c-430b-af85-593d3b35d298" containerName="kserve-container"
Jun 02 16:50:19.069949 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:19.069930    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz"
Jun 02 16:50:19.074187 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:19.074162    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Jun 02 16:50:19.074395 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:19.074383    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Jun 02 16:50:19.075190 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:19.075165    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jun 02 16:50:19.075190 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:19.075183    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ztrcr\""
Jun 02 16:50:19.075381 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:19.075262    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jun 02 16:50:19.092789 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:19.092758    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz"]
Jun 02 16:50:19.204279 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:19.204244    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abb8bcdb-1e70-49c8-84fa-7e2ade733280-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-779db84d9-bv7dz\" (UID: \"abb8bcdb-1e70-49c8-84fa-7e2ade733280\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz"
Jun 02 16:50:19.204516 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:19.204298    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abb8bcdb-1e70-49c8-84fa-7e2ade733280-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-779db84d9-bv7dz\" (UID: \"abb8bcdb-1e70-49c8-84fa-7e2ade733280\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz"
Jun 02 16:50:19.204516 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:19.204344    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c54k9\" (UniqueName: \"kubernetes.io/projected/abb8bcdb-1e70-49c8-84fa-7e2ade733280-kube-api-access-c54k9\") pod \"isvc-xgboost-runtime-predictor-779db84d9-bv7dz\" (UID: \"abb8bcdb-1e70-49c8-84fa-7e2ade733280\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz"
Jun 02 16:50:19.204516 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:19.204382    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abb8bcdb-1e70-49c8-84fa-7e2ade733280-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-779db84d9-bv7dz\" (UID: \"abb8bcdb-1e70-49c8-84fa-7e2ade733280\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz"
Jun 02 16:50:19.304909 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:19.304876    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abb8bcdb-1e70-49c8-84fa-7e2ade733280-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-779db84d9-bv7dz\" (UID: \"abb8bcdb-1e70-49c8-84fa-7e2ade733280\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz"
Jun 02 16:50:19.305092 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:19.304918    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abb8bcdb-1e70-49c8-84fa-7e2ade733280-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-779db84d9-bv7dz\" (UID: \"abb8bcdb-1e70-49c8-84fa-7e2ade733280\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz"
Jun 02 16:50:19.305092 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:19.304957    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abb8bcdb-1e70-49c8-84fa-7e2ade733280-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-779db84d9-bv7dz\" (UID: \"abb8bcdb-1e70-49c8-84fa-7e2ade733280\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz"
Jun 02 16:50:19.305092 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:19.304980    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c54k9\" (UniqueName: \"kubernetes.io/projected/abb8bcdb-1e70-49c8-84fa-7e2ade733280-kube-api-access-c54k9\") pod \"isvc-xgboost-runtime-predictor-779db84d9-bv7dz\" (UID: \"abb8bcdb-1e70-49c8-84fa-7e2ade733280\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz"
Jun 02 16:50:19.305451 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:19.305432    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abb8bcdb-1e70-49c8-84fa-7e2ade733280-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-779db84d9-bv7dz\" (UID: \"abb8bcdb-1e70-49c8-84fa-7e2ade733280\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz"
Jun 02 16:50:19.305699 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:19.305672    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abb8bcdb-1e70-49c8-84fa-7e2ade733280-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-779db84d9-bv7dz\" (UID: \"abb8bcdb-1e70-49c8-84fa-7e2ade733280\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz"
Jun 02 16:50:19.307575 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:19.307560    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abb8bcdb-1e70-49c8-84fa-7e2ade733280-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-779db84d9-bv7dz\" (UID: \"abb8bcdb-1e70-49c8-84fa-7e2ade733280\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz"
Jun 02 16:50:19.313979 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:19.313951    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c54k9\" (UniqueName: \"kubernetes.io/projected/abb8bcdb-1e70-49c8-84fa-7e2ade733280-kube-api-access-c54k9\") pod \"isvc-xgboost-runtime-predictor-779db84d9-bv7dz\" (UID: \"abb8bcdb-1e70-49c8-84fa-7e2ade733280\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz"
Jun 02 16:50:19.380674 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:19.380591    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz"
Jun 02 16:50:19.510815 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:19.510775    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz"]
Jun 02 16:50:19.514500 ip-10-0-129-156 kubenswrapper[2579]: W0602 16:50:19.514467    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabb8bcdb_1e70_49c8_84fa_7e2ade733280.slice/crio-243c7b070bcd260bec6353fcef85f57d3fe09e9c504d5795e6fd723312dbaa8a WatchSource:0}: Error finding container 243c7b070bcd260bec6353fcef85f57d3fe09e9c504d5795e6fd723312dbaa8a: Status 404 returned error can't find the container with id 243c7b070bcd260bec6353fcef85f57d3fe09e9c504d5795e6fd723312dbaa8a
Jun 02 16:50:19.516255 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:19.516238    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 16:50:19.832630 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:19.832589    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz" event={"ID":"abb8bcdb-1e70-49c8-84fa-7e2ade733280","Type":"ContainerStarted","Data":"f55a4e1ea580508ea5be4b092dba12e1592850cccc6ea6e4dba08e99f1652e54"}
Jun 02 16:50:19.832807 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:19.832636    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz" event={"ID":"abb8bcdb-1e70-49c8-84fa-7e2ade733280","Type":"ContainerStarted","Data":"243c7b070bcd260bec6353fcef85f57d3fe09e9c504d5795e6fd723312dbaa8a"}
Jun 02 16:50:23.843749 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:23.843713    2579 generic.go:358] "Generic (PLEG): container finished" podID="abb8bcdb-1e70-49c8-84fa-7e2ade733280" containerID="f55a4e1ea580508ea5be4b092dba12e1592850cccc6ea6e4dba08e99f1652e54" exitCode=0
Jun 02 16:50:23.844161 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:23.843790    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz" event={"ID":"abb8bcdb-1e70-49c8-84fa-7e2ade733280","Type":"ContainerDied","Data":"f55a4e1ea580508ea5be4b092dba12e1592850cccc6ea6e4dba08e99f1652e54"}
Jun 02 16:50:24.848795 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:24.848761    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz" event={"ID":"abb8bcdb-1e70-49c8-84fa-7e2ade733280","Type":"ContainerStarted","Data":"32dafea86b831f257392ba105ee1ad8e84a8e2d37254a27043e2c097e0db2c23"}
Jun 02 16:50:24.849195 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:24.848804    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz" event={"ID":"abb8bcdb-1e70-49c8-84fa-7e2ade733280","Type":"ContainerStarted","Data":"1396a2deb451624ab999dccec4d367c361e14d07fa05db49b9305caaffe4ad9e"}
Jun 02 16:50:24.849195 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:24.849034    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz"
Jun 02 16:50:24.869167 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:24.869117    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz" podStartSLOduration=5.869100679 podStartE2EDuration="5.869100679s" podCreationTimestamp="2026-06-02 16:50:19 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:50:24.867199643 +0000 UTC m=+3833.393108073" watchObservedRunningTime="2026-06-02 16:50:24.869100679 +0000 UTC m=+3833.395009109"
Jun 02 16:50:25.851468 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:25.851432    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz"
Jun 02 16:50:25.852816 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:25.852786    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz" podUID="abb8bcdb-1e70-49c8-84fa-7e2ade733280" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jun 02 16:50:26.854134 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:26.854100    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz" podUID="abb8bcdb-1e70-49c8-84fa-7e2ade733280" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jun 02 16:50:31.858854 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:31.858823    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz"
Jun 02 16:50:31.859445 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:31.859395    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz" podUID="abb8bcdb-1e70-49c8-84fa-7e2ade733280" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jun 02 16:50:41.859843 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:41.859799    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz" podUID="abb8bcdb-1e70-49c8-84fa-7e2ade733280" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jun 02 16:50:51.859867 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:50:51.859770    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz" podUID="abb8bcdb-1e70-49c8-84fa-7e2ade733280" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jun 02 16:51:01.859972 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:51:01.859917    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz" podUID="abb8bcdb-1e70-49c8-84fa-7e2ade733280" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jun 02 16:51:11.860474 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:51:11.860430    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz" podUID="abb8bcdb-1e70-49c8-84fa-7e2ade733280" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jun 02 16:51:21.860319 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:51:21.860269    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz" podUID="abb8bcdb-1e70-49c8-84fa-7e2ade733280" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jun 02 16:51:31.860323 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:51:31.860292    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz"
Jun 02 16:51:39.118690 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:51:39.118642    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz"]
Jun 02 16:51:39.119162 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:51:39.119102    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz" podUID="abb8bcdb-1e70-49c8-84fa-7e2ade733280" containerName="kserve-container" containerID="cri-o://1396a2deb451624ab999dccec4d367c361e14d07fa05db49b9305caaffe4ad9e" gracePeriod=30
Jun 02 16:51:39.119311 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:51:39.119173    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz" podUID="abb8bcdb-1e70-49c8-84fa-7e2ade733280" containerName="kube-rbac-proxy" containerID="cri-o://32dafea86b831f257392ba105ee1ad8e84a8e2d37254a27043e2c097e0db2c23" gracePeriod=30
Jun 02 16:51:40.061638 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:51:40.061605    2579 generic.go:358] "Generic (PLEG): container finished" podID="abb8bcdb-1e70-49c8-84fa-7e2ade733280" containerID="32dafea86b831f257392ba105ee1ad8e84a8e2d37254a27043e2c097e0db2c23" exitCode=2
Jun 02 16:51:40.063105 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:51:40.063080    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz" event={"ID":"abb8bcdb-1e70-49c8-84fa-7e2ade733280","Type":"ContainerDied","Data":"32dafea86b831f257392ba105ee1ad8e84a8e2d37254a27043e2c097e0db2c23"}
Jun 02 16:51:41.854621 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:51:41.854569    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz" podUID="abb8bcdb-1e70-49c8-84fa-7e2ade733280" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jun 02 16:51:41.859996 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:51:41.859965    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz" podUID="abb8bcdb-1e70-49c8-84fa-7e2ade733280" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jun 02 16:51:43.085305 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:51:43.085271    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lqx9k_6127e33c-aa77-437e-b443-e956037afedb/ovn-acl-logging/0.log"
Jun 02 16:51:43.086029 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:51:43.086010    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lqx9k_6127e33c-aa77-437e-b443-e956037afedb/ovn-acl-logging/0.log"
Jun 02 16:51:43.376379 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:51:43.376351    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz"
Jun 02 16:51:43.457419 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:51:43.457373    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abb8bcdb-1e70-49c8-84fa-7e2ade733280-kserve-provision-location\") pod \"abb8bcdb-1e70-49c8-84fa-7e2ade733280\" (UID: \"abb8bcdb-1e70-49c8-84fa-7e2ade733280\") "
Jun 02 16:51:43.457602 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:51:43.457469    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abb8bcdb-1e70-49c8-84fa-7e2ade733280-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") pod \"abb8bcdb-1e70-49c8-84fa-7e2ade733280\" (UID: \"abb8bcdb-1e70-49c8-84fa-7e2ade733280\") "
Jun 02 16:51:43.457602 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:51:43.457530    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abb8bcdb-1e70-49c8-84fa-7e2ade733280-proxy-tls\") pod \"abb8bcdb-1e70-49c8-84fa-7e2ade733280\" (UID: \"abb8bcdb-1e70-49c8-84fa-7e2ade733280\") "
Jun 02 16:51:43.457602 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:51:43.457567    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c54k9\" (UniqueName: \"kubernetes.io/projected/abb8bcdb-1e70-49c8-84fa-7e2ade733280-kube-api-access-c54k9\") pod \"abb8bcdb-1e70-49c8-84fa-7e2ade733280\" (UID: \"abb8bcdb-1e70-49c8-84fa-7e2ade733280\") "
Jun 02 16:51:43.457817 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:51:43.457785    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abb8bcdb-1e70-49c8-84fa-7e2ade733280-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "abb8bcdb-1e70-49c8-84fa-7e2ade733280" (UID: "abb8bcdb-1e70-49c8-84fa-7e2ade733280"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:51:43.457817 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:51:43.457801    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abb8bcdb-1e70-49c8-84fa-7e2ade733280-isvc-xgboost-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-runtime-kube-rbac-proxy-sar-config") pod "abb8bcdb-1e70-49c8-84fa-7e2ade733280" (UID: "abb8bcdb-1e70-49c8-84fa-7e2ade733280"). InnerVolumeSpecName "isvc-xgboost-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:51:43.459888 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:51:43.459843    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abb8bcdb-1e70-49c8-84fa-7e2ade733280-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "abb8bcdb-1e70-49c8-84fa-7e2ade733280" (UID: "abb8bcdb-1e70-49c8-84fa-7e2ade733280"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:51:43.460044 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:51:43.459904    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abb8bcdb-1e70-49c8-84fa-7e2ade733280-kube-api-access-c54k9" (OuterVolumeSpecName: "kube-api-access-c54k9") pod "abb8bcdb-1e70-49c8-84fa-7e2ade733280" (UID: "abb8bcdb-1e70-49c8-84fa-7e2ade733280"). InnerVolumeSpecName "kube-api-access-c54k9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:51:43.558575 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:51:43.558530    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c54k9\" (UniqueName: \"kubernetes.io/projected/abb8bcdb-1e70-49c8-84fa-7e2ade733280-kube-api-access-c54k9\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:51:43.558575 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:51:43.558563    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abb8bcdb-1e70-49c8-84fa-7e2ade733280-kserve-provision-location\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:51:43.558575 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:51:43.558574    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abb8bcdb-1e70-49c8-84fa-7e2ade733280-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:51:43.558575 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:51:43.558587    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abb8bcdb-1e70-49c8-84fa-7e2ade733280-proxy-tls\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:51:44.075010 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:51:44.074978    2579 generic.go:358] "Generic (PLEG): container finished" podID="abb8bcdb-1e70-49c8-84fa-7e2ade733280" containerID="1396a2deb451624ab999dccec4d367c361e14d07fa05db49b9305caaffe4ad9e" exitCode=0
Jun 02 16:51:44.075176 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:51:44.075061    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz"
Jun 02 16:51:44.075176 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:51:44.075057    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz" event={"ID":"abb8bcdb-1e70-49c8-84fa-7e2ade733280","Type":"ContainerDied","Data":"1396a2deb451624ab999dccec4d367c361e14d07fa05db49b9305caaffe4ad9e"}
Jun 02 16:51:44.075176 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:51:44.075167    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz" event={"ID":"abb8bcdb-1e70-49c8-84fa-7e2ade733280","Type":"ContainerDied","Data":"243c7b070bcd260bec6353fcef85f57d3fe09e9c504d5795e6fd723312dbaa8a"}
Jun 02 16:51:44.075317 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:51:44.075184    2579 scope.go:117] "RemoveContainer" containerID="32dafea86b831f257392ba105ee1ad8e84a8e2d37254a27043e2c097e0db2c23"
Jun 02 16:51:44.083690 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:51:44.083669    2579 scope.go:117] "RemoveContainer" containerID="1396a2deb451624ab999dccec4d367c361e14d07fa05db49b9305caaffe4ad9e"
Jun 02 16:51:44.092697 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:51:44.092668    2579 scope.go:117] "RemoveContainer" containerID="f55a4e1ea580508ea5be4b092dba12e1592850cccc6ea6e4dba08e99f1652e54"
Jun 02 16:51:44.093120 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:51:44.092820    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz"]
Jun 02 16:51:44.096890 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:51:44.096861    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-bv7dz"]
Jun 02 16:51:44.101112 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:51:44.101091    2579 scope.go:117] "RemoveContainer" containerID="32dafea86b831f257392ba105ee1ad8e84a8e2d37254a27043e2c097e0db2c23"
Jun 02 16:51:44.101513 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:51:44.101490    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32dafea86b831f257392ba105ee1ad8e84a8e2d37254a27043e2c097e0db2c23\": container with ID starting with 32dafea86b831f257392ba105ee1ad8e84a8e2d37254a27043e2c097e0db2c23 not found: ID does not exist" containerID="32dafea86b831f257392ba105ee1ad8e84a8e2d37254a27043e2c097e0db2c23"
Jun 02 16:51:44.101594 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:51:44.101520    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32dafea86b831f257392ba105ee1ad8e84a8e2d37254a27043e2c097e0db2c23"} err="failed to get container status \"32dafea86b831f257392ba105ee1ad8e84a8e2d37254a27043e2c097e0db2c23\": rpc error: code = NotFound desc = could not find container \"32dafea86b831f257392ba105ee1ad8e84a8e2d37254a27043e2c097e0db2c23\": container with ID starting with 32dafea86b831f257392ba105ee1ad8e84a8e2d37254a27043e2c097e0db2c23 not found: ID does not exist"
Jun 02 16:51:44.101594 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:51:44.101540    2579 scope.go:117] "RemoveContainer" containerID="1396a2deb451624ab999dccec4d367c361e14d07fa05db49b9305caaffe4ad9e"
Jun 02 16:51:44.101824 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:51:44.101805    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1396a2deb451624ab999dccec4d367c361e14d07fa05db49b9305caaffe4ad9e\": container with ID starting with 1396a2deb451624ab999dccec4d367c361e14d07fa05db49b9305caaffe4ad9e not found: ID does not exist" containerID="1396a2deb451624ab999dccec4d367c361e14d07fa05db49b9305caaffe4ad9e"
Jun 02 16:51:44.101883 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:51:44.101835    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1396a2deb451624ab999dccec4d367c361e14d07fa05db49b9305caaffe4ad9e"} err="failed to get container status \"1396a2deb451624ab999dccec4d367c361e14d07fa05db49b9305caaffe4ad9e\": rpc error: code = NotFound desc = could not find container \"1396a2deb451624ab999dccec4d367c361e14d07fa05db49b9305caaffe4ad9e\": container with ID starting with 1396a2deb451624ab999dccec4d367c361e14d07fa05db49b9305caaffe4ad9e not found: ID does not exist"
Jun 02 16:51:44.101883 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:51:44.101853    2579 scope.go:117] "RemoveContainer" containerID="f55a4e1ea580508ea5be4b092dba12e1592850cccc6ea6e4dba08e99f1652e54"
Jun 02 16:51:44.102110 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:51:44.102092    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f55a4e1ea580508ea5be4b092dba12e1592850cccc6ea6e4dba08e99f1652e54\": container with ID starting with f55a4e1ea580508ea5be4b092dba12e1592850cccc6ea6e4dba08e99f1652e54 not found: ID does not exist" containerID="f55a4e1ea580508ea5be4b092dba12e1592850cccc6ea6e4dba08e99f1652e54"
Jun 02 16:51:44.102153 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:51:44.102117    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f55a4e1ea580508ea5be4b092dba12e1592850cccc6ea6e4dba08e99f1652e54"} err="failed to get container status \"f55a4e1ea580508ea5be4b092dba12e1592850cccc6ea6e4dba08e99f1652e54\": rpc error: code = NotFound desc = could not find container \"f55a4e1ea580508ea5be4b092dba12e1592850cccc6ea6e4dba08e99f1652e54\": container with ID starting with f55a4e1ea580508ea5be4b092dba12e1592850cccc6ea6e4dba08e99f1652e54 not found: ID does not exist"
Jun 02 16:51:46.063672 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:51:46.063638    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abb8bcdb-1e70-49c8-84fa-7e2ade733280" path="/var/lib/kubelet/pods/abb8bcdb-1e70-49c8-84fa-7e2ade733280/volumes"
Jun 02 16:52:39.371692 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:52:39.371645    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws"]
Jun 02 16:52:39.372163 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:52:39.371913    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abb8bcdb-1e70-49c8-84fa-7e2ade733280" containerName="kube-rbac-proxy"
Jun 02 16:52:39.372163 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:52:39.371923    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="abb8bcdb-1e70-49c8-84fa-7e2ade733280" containerName="kube-rbac-proxy"
Jun 02 16:52:39.372163 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:52:39.371937    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abb8bcdb-1e70-49c8-84fa-7e2ade733280" containerName="kserve-container"
Jun 02 16:52:39.372163 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:52:39.371942    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="abb8bcdb-1e70-49c8-84fa-7e2ade733280" containerName="kserve-container"
Jun 02 16:52:39.372163 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:52:39.371949    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abb8bcdb-1e70-49c8-84fa-7e2ade733280" containerName="storage-initializer"
Jun 02 16:52:39.372163 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:52:39.371956    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="abb8bcdb-1e70-49c8-84fa-7e2ade733280" containerName="storage-initializer"
Jun 02 16:52:39.372163 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:52:39.372006    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="abb8bcdb-1e70-49c8-84fa-7e2ade733280" containerName="kube-rbac-proxy"
Jun 02 16:52:39.372163 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:52:39.372015    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="abb8bcdb-1e70-49c8-84fa-7e2ade733280" containerName="kserve-container"
Jun 02 16:52:39.376324 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:52:39.376297    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws"
Jun 02 16:52:39.378737 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:52:39.378693    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jun 02 16:52:39.378897 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:52:39.378693    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Jun 02 16:52:39.378897 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:52:39.378718    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Jun 02 16:52:39.379443 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:52:39.379424    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jun 02 16:52:39.379532 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:52:39.379434    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ztrcr\""
Jun 02 16:52:39.390233 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:52:39.390192    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws"]
Jun 02 16:52:39.492961 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:52:39.492908    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b1bf1c7-199d-4711-8e3e-8ba13481e036-proxy-tls\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-rbgws\" (UID: \"2b1bf1c7-199d-4711-8e3e-8ba13481e036\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws"
Jun 02 16:52:39.493157 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:52:39.492976    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whrrd\" (UniqueName: \"kubernetes.io/projected/2b1bf1c7-199d-4711-8e3e-8ba13481e036-kube-api-access-whrrd\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-rbgws\" (UID: \"2b1bf1c7-199d-4711-8e3e-8ba13481e036\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws"
Jun 02 16:52:39.493157 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:52:39.493008    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b1bf1c7-199d-4711-8e3e-8ba13481e036-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-rbgws\" (UID: \"2b1bf1c7-199d-4711-8e3e-8ba13481e036\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws"
Jun 02 16:52:39.493157 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:52:39.493029    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b1bf1c7-199d-4711-8e3e-8ba13481e036-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-rbgws\" (UID: \"2b1bf1c7-199d-4711-8e3e-8ba13481e036\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws"
Jun 02 16:52:39.594136 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:52:39.594094    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b1bf1c7-199d-4711-8e3e-8ba13481e036-proxy-tls\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-rbgws\" (UID: \"2b1bf1c7-199d-4711-8e3e-8ba13481e036\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws"
Jun 02 16:52:39.594286 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:52:39.594152    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-whrrd\" (UniqueName: \"kubernetes.io/projected/2b1bf1c7-199d-4711-8e3e-8ba13481e036-kube-api-access-whrrd\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-rbgws\" (UID: \"2b1bf1c7-199d-4711-8e3e-8ba13481e036\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws"
Jun 02 16:52:39.594286 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:52:39.594177    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b1bf1c7-199d-4711-8e3e-8ba13481e036-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-rbgws\" (UID: \"2b1bf1c7-199d-4711-8e3e-8ba13481e036\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws"
Jun 02 16:52:39.594286 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:52:39.594198    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b1bf1c7-199d-4711-8e3e-8ba13481e036-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-rbgws\" (UID: \"2b1bf1c7-199d-4711-8e3e-8ba13481e036\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws"
Jun 02 16:52:39.594598 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:52:39.594579    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b1bf1c7-199d-4711-8e3e-8ba13481e036-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-rbgws\" (UID: \"2b1bf1c7-199d-4711-8e3e-8ba13481e036\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws"
Jun 02 16:52:39.594960 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:52:39.594930    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b1bf1c7-199d-4711-8e3e-8ba13481e036-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-rbgws\" (UID: \"2b1bf1c7-199d-4711-8e3e-8ba13481e036\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws"
Jun 02 16:52:39.596983 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:52:39.596962    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b1bf1c7-199d-4711-8e3e-8ba13481e036-proxy-tls\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-rbgws\" (UID: \"2b1bf1c7-199d-4711-8e3e-8ba13481e036\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws"
Jun 02 16:52:39.602727 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:52:39.602694    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whrrd\" (UniqueName: \"kubernetes.io/projected/2b1bf1c7-199d-4711-8e3e-8ba13481e036-kube-api-access-whrrd\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-rbgws\" (UID: \"2b1bf1c7-199d-4711-8e3e-8ba13481e036\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws"
Jun 02 16:52:39.696485 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:52:39.696351    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws"
Jun 02 16:52:39.835712 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:52:39.835672    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws"]
Jun 02 16:52:39.839488 ip-10-0-129-156 kubenswrapper[2579]: W0602 16:52:39.839437    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b1bf1c7_199d_4711_8e3e_8ba13481e036.slice/crio-8d544e1a3ae02848eff24cc57c8daa51fb8664e90f04a0e1e501b654fbeff111 WatchSource:0}: Error finding container 8d544e1a3ae02848eff24cc57c8daa51fb8664e90f04a0e1e501b654fbeff111: Status 404 returned error can't find the container with id 8d544e1a3ae02848eff24cc57c8daa51fb8664e90f04a0e1e501b654fbeff111
Jun 02 16:52:40.238813 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:52:40.238776    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws" event={"ID":"2b1bf1c7-199d-4711-8e3e-8ba13481e036","Type":"ContainerStarted","Data":"828952b1d16e8d24aeee2b1a78492820835678e57fa11e40c8418ad27f8c0301"}
Jun 02 16:52:40.239020 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:52:40.238820    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws" event={"ID":"2b1bf1c7-199d-4711-8e3e-8ba13481e036","Type":"ContainerStarted","Data":"8d544e1a3ae02848eff24cc57c8daa51fb8664e90f04a0e1e501b654fbeff111"}
Jun 02 16:52:44.250759 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:52:44.250718    2579 generic.go:358] "Generic (PLEG): container finished" podID="2b1bf1c7-199d-4711-8e3e-8ba13481e036" containerID="828952b1d16e8d24aeee2b1a78492820835678e57fa11e40c8418ad27f8c0301" exitCode=0
Jun 02 16:52:44.251201 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:52:44.250790    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws" event={"ID":"2b1bf1c7-199d-4711-8e3e-8ba13481e036","Type":"ContainerDied","Data":"828952b1d16e8d24aeee2b1a78492820835678e57fa11e40c8418ad27f8c0301"}
Jun 02 16:52:45.256711 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:52:45.256669    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws" event={"ID":"2b1bf1c7-199d-4711-8e3e-8ba13481e036","Type":"ContainerStarted","Data":"555f5f15aecd1a9c61280739de1e132224c97cac687957e2db2f1fa4d8fecbf1"}
Jun 02 16:52:45.257342 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:52:45.256720    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws" event={"ID":"2b1bf1c7-199d-4711-8e3e-8ba13481e036","Type":"ContainerStarted","Data":"1f47f5e986d7440637a514a8520b38758d320bb9aa4cd361746b3d87caa51d55"}
Jun 02 16:52:45.257342 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:52:45.257015    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws"
Jun 02 16:52:45.257342 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:52:45.257121    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws"
Jun 02 16:52:45.258240 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:52:45.258217    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws" podUID="2b1bf1c7-199d-4711-8e3e-8ba13481e036" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jun 02 16:52:45.274804 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:52:45.274105    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws" podStartSLOduration=6.274083613 podStartE2EDuration="6.274083613s" podCreationTimestamp="2026-06-02 16:52:39 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:52:45.273097046 +0000 UTC m=+3973.799005479" watchObservedRunningTime="2026-06-02 16:52:45.274083613 +0000 UTC m=+3973.799992056"
Jun 02 16:52:46.260222 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:52:46.260175    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws" podUID="2b1bf1c7-199d-4711-8e3e-8ba13481e036" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jun 02 16:52:51.265040 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:52:51.265008    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws"
Jun 02 16:52:51.265630 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:52:51.265596    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws" podUID="2b1bf1c7-199d-4711-8e3e-8ba13481e036" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jun 02 16:53:01.265834 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:53:01.265792    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws" podUID="2b1bf1c7-199d-4711-8e3e-8ba13481e036" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jun 02 16:53:11.266366 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:53:11.266325    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws" podUID="2b1bf1c7-199d-4711-8e3e-8ba13481e036" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jun 02 16:53:21.266025 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:53:21.265981    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws" podUID="2b1bf1c7-199d-4711-8e3e-8ba13481e036" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jun 02 16:53:31.265636 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:53:31.265491    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws" podUID="2b1bf1c7-199d-4711-8e3e-8ba13481e036" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jun 02 16:53:41.266171 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:53:41.266130    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws" podUID="2b1bf1c7-199d-4711-8e3e-8ba13481e036" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jun 02 16:53:51.266641 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:53:51.266547    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws"
Jun 02 16:53:59.473923 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:53:59.473882    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws"]
Jun 02 16:53:59.474559 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:53:59.474257    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws" podUID="2b1bf1c7-199d-4711-8e3e-8ba13481e036" containerName="kserve-container" containerID="cri-o://1f47f5e986d7440637a514a8520b38758d320bb9aa4cd361746b3d87caa51d55" gracePeriod=30
Jun 02 16:53:59.474559 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:53:59.474320    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws" podUID="2b1bf1c7-199d-4711-8e3e-8ba13481e036" containerName="kube-rbac-proxy" containerID="cri-o://555f5f15aecd1a9c61280739de1e132224c97cac687957e2db2f1fa4d8fecbf1" gracePeriod=30
Jun 02 16:54:00.471228 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:54:00.471190    2579 generic.go:358] "Generic (PLEG): container finished" podID="2b1bf1c7-199d-4711-8e3e-8ba13481e036" containerID="555f5f15aecd1a9c61280739de1e132224c97cac687957e2db2f1fa4d8fecbf1" exitCode=2
Jun 02 16:54:00.471430 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:54:00.471265    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws" event={"ID":"2b1bf1c7-199d-4711-8e3e-8ba13481e036","Type":"ContainerDied","Data":"555f5f15aecd1a9c61280739de1e132224c97cac687957e2db2f1fa4d8fecbf1"}
Jun 02 16:54:01.261253 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:54:01.261202    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws" podUID="2b1bf1c7-199d-4711-8e3e-8ba13481e036" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Jun 02 16:54:01.266523 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:54:01.266490    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws" podUID="2b1bf1c7-199d-4711-8e3e-8ba13481e036" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jun 02 16:54:03.625694 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:54:03.625653    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws"
Jun 02 16:54:03.750818 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:54:03.750784    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b1bf1c7-199d-4711-8e3e-8ba13481e036-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"2b1bf1c7-199d-4711-8e3e-8ba13481e036\" (UID: \"2b1bf1c7-199d-4711-8e3e-8ba13481e036\") "
Jun 02 16:54:03.751020 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:54:03.750828    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b1bf1c7-199d-4711-8e3e-8ba13481e036-proxy-tls\") pod \"2b1bf1c7-199d-4711-8e3e-8ba13481e036\" (UID: \"2b1bf1c7-199d-4711-8e3e-8ba13481e036\") "
Jun 02 16:54:03.751020 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:54:03.750853    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whrrd\" (UniqueName: \"kubernetes.io/projected/2b1bf1c7-199d-4711-8e3e-8ba13481e036-kube-api-access-whrrd\") pod \"2b1bf1c7-199d-4711-8e3e-8ba13481e036\" (UID: \"2b1bf1c7-199d-4711-8e3e-8ba13481e036\") "
Jun 02 16:54:03.751020 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:54:03.750875    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b1bf1c7-199d-4711-8e3e-8ba13481e036-kserve-provision-location\") pod \"2b1bf1c7-199d-4711-8e3e-8ba13481e036\" (UID: \"2b1bf1c7-199d-4711-8e3e-8ba13481e036\") "
Jun 02 16:54:03.751217 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:54:03.751180    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b1bf1c7-199d-4711-8e3e-8ba13481e036-isvc-xgboost-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-v2-kube-rbac-proxy-sar-config") pod "2b1bf1c7-199d-4711-8e3e-8ba13481e036" (UID: "2b1bf1c7-199d-4711-8e3e-8ba13481e036"). InnerVolumeSpecName "isvc-xgboost-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:54:03.751282 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:54:03.751243    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b1bf1c7-199d-4711-8e3e-8ba13481e036-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2b1bf1c7-199d-4711-8e3e-8ba13481e036" (UID: "2b1bf1c7-199d-4711-8e3e-8ba13481e036"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:54:03.753292 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:54:03.753265    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b1bf1c7-199d-4711-8e3e-8ba13481e036-kube-api-access-whrrd" (OuterVolumeSpecName: "kube-api-access-whrrd") pod "2b1bf1c7-199d-4711-8e3e-8ba13481e036" (UID: "2b1bf1c7-199d-4711-8e3e-8ba13481e036"). InnerVolumeSpecName "kube-api-access-whrrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:54:03.753397 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:54:03.753260    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b1bf1c7-199d-4711-8e3e-8ba13481e036-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2b1bf1c7-199d-4711-8e3e-8ba13481e036" (UID: "2b1bf1c7-199d-4711-8e3e-8ba13481e036"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:54:03.852022 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:54:03.851979    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b1bf1c7-199d-4711-8e3e-8ba13481e036-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:54:03.852022 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:54:03.852012    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b1bf1c7-199d-4711-8e3e-8ba13481e036-proxy-tls\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:54:03.852022 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:54:03.852022    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whrrd\" (UniqueName: \"kubernetes.io/projected/2b1bf1c7-199d-4711-8e3e-8ba13481e036-kube-api-access-whrrd\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:54:03.852267 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:54:03.852034    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b1bf1c7-199d-4711-8e3e-8ba13481e036-kserve-provision-location\") on node \"ip-10-0-129-156.ec2.internal\" DevicePath \"\""
Jun 02 16:54:04.486038 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:54:04.486001    2579 generic.go:358] "Generic (PLEG): container finished" podID="2b1bf1c7-199d-4711-8e3e-8ba13481e036" containerID="1f47f5e986d7440637a514a8520b38758d320bb9aa4cd361746b3d87caa51d55" exitCode=0
Jun 02 16:54:04.486227 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:54:04.486064    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws" event={"ID":"2b1bf1c7-199d-4711-8e3e-8ba13481e036","Type":"ContainerDied","Data":"1f47f5e986d7440637a514a8520b38758d320bb9aa4cd361746b3d87caa51d55"}
Jun 02 16:54:04.486227 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:54:04.486076    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws"
Jun 02 16:54:04.486227 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:54:04.486089    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws" event={"ID":"2b1bf1c7-199d-4711-8e3e-8ba13481e036","Type":"ContainerDied","Data":"8d544e1a3ae02848eff24cc57c8daa51fb8664e90f04a0e1e501b654fbeff111"}
Jun 02 16:54:04.486227 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:54:04.486110    2579 scope.go:117] "RemoveContainer" containerID="555f5f15aecd1a9c61280739de1e132224c97cac687957e2db2f1fa4d8fecbf1"
Jun 02 16:54:04.496008 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:54:04.495681    2579 scope.go:117] "RemoveContainer" containerID="1f47f5e986d7440637a514a8520b38758d320bb9aa4cd361746b3d87caa51d55"
Jun 02 16:54:04.506649 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:54:04.506610    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws"]
Jun 02 16:54:04.508169 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:54:04.508146    2579 scope.go:117] "RemoveContainer" containerID="828952b1d16e8d24aeee2b1a78492820835678e57fa11e40c8418ad27f8c0301"
Jun 02 16:54:04.509868 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:54:04.509838    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-rbgws"]
Jun 02 16:54:04.516395 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:54:04.516370    2579 scope.go:117] "RemoveContainer" containerID="555f5f15aecd1a9c61280739de1e132224c97cac687957e2db2f1fa4d8fecbf1"
Jun 02 16:54:04.516757 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:54:04.516737    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"555f5f15aecd1a9c61280739de1e132224c97cac687957e2db2f1fa4d8fecbf1\": container with ID starting with 555f5f15aecd1a9c61280739de1e132224c97cac687957e2db2f1fa4d8fecbf1 not found: ID does not exist" containerID="555f5f15aecd1a9c61280739de1e132224c97cac687957e2db2f1fa4d8fecbf1"
Jun 02 16:54:04.516843 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:54:04.516765    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"555f5f15aecd1a9c61280739de1e132224c97cac687957e2db2f1fa4d8fecbf1"} err="failed to get container status \"555f5f15aecd1a9c61280739de1e132224c97cac687957e2db2f1fa4d8fecbf1\": rpc error: code = NotFound desc = could not find container \"555f5f15aecd1a9c61280739de1e132224c97cac687957e2db2f1fa4d8fecbf1\": container with ID starting with 555f5f15aecd1a9c61280739de1e132224c97cac687957e2db2f1fa4d8fecbf1 not found: ID does not exist"
Jun 02 16:54:04.516843 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:54:04.516785    2579 scope.go:117] "RemoveContainer" containerID="1f47f5e986d7440637a514a8520b38758d320bb9aa4cd361746b3d87caa51d55"
Jun 02 16:54:04.517060 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:54:04.517044    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f47f5e986d7440637a514a8520b38758d320bb9aa4cd361746b3d87caa51d55\": container with ID starting with 1f47f5e986d7440637a514a8520b38758d320bb9aa4cd361746b3d87caa51d55 not found: ID does not exist" containerID="1f47f5e986d7440637a514a8520b38758d320bb9aa4cd361746b3d87caa51d55"
Jun 02 16:54:04.517101 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:54:04.517067    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f47f5e986d7440637a514a8520b38758d320bb9aa4cd361746b3d87caa51d55"} err="failed to get container status \"1f47f5e986d7440637a514a8520b38758d320bb9aa4cd361746b3d87caa51d55\": rpc error: code = NotFound desc = could not find container \"1f47f5e986d7440637a514a8520b38758d320bb9aa4cd361746b3d87caa51d55\": container with ID starting with 1f47f5e986d7440637a514a8520b38758d320bb9aa4cd361746b3d87caa51d55 not found: ID does not exist"
Jun 02 16:54:04.517101 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:54:04.517082    2579 scope.go:117] "RemoveContainer" containerID="828952b1d16e8d24aeee2b1a78492820835678e57fa11e40c8418ad27f8c0301"
Jun 02 16:54:04.517336 ip-10-0-129-156 kubenswrapper[2579]: E0602 16:54:04.517315    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"828952b1d16e8d24aeee2b1a78492820835678e57fa11e40c8418ad27f8c0301\": container with ID starting with 828952b1d16e8d24aeee2b1a78492820835678e57fa11e40c8418ad27f8c0301 not found: ID does not exist" containerID="828952b1d16e8d24aeee2b1a78492820835678e57fa11e40c8418ad27f8c0301"
Jun 02 16:54:04.517384 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:54:04.517345    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"828952b1d16e8d24aeee2b1a78492820835678e57fa11e40c8418ad27f8c0301"} err="failed to get container status \"828952b1d16e8d24aeee2b1a78492820835678e57fa11e40c8418ad27f8c0301\": rpc error: code = NotFound desc = could not find container \"828952b1d16e8d24aeee2b1a78492820835678e57fa11e40c8418ad27f8c0301\": container with ID starting with 828952b1d16e8d24aeee2b1a78492820835678e57fa11e40c8418ad27f8c0301 not found: ID does not exist"
Jun 02 16:54:06.063538 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:54:06.063504    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b1bf1c7-199d-4711-8e3e-8ba13481e036" path="/var/lib/kubelet/pods/2b1bf1c7-199d-4711-8e3e-8ba13481e036/volumes"
Jun 02 16:56:43.104102 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:56:43.104072    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lqx9k_6127e33c-aa77-437e-b443-e956037afedb/ovn-acl-logging/0.log"
Jun 02 16:56:43.106018 ip-10-0-129-156 kubenswrapper[2579]: I0602 16:56:43.105994    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lqx9k_6127e33c-aa77-437e-b443-e956037afedb/ovn-acl-logging/0.log"
Jun 02 17:00:18.899988 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:18.899947    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zpxbm/must-gather-zmm4l"]
Jun 02 17:00:18.900568 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:18.900290    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b1bf1c7-199d-4711-8e3e-8ba13481e036" containerName="storage-initializer"
Jun 02 17:00:18.900568 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:18.900309    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b1bf1c7-199d-4711-8e3e-8ba13481e036" containerName="storage-initializer"
Jun 02 17:00:18.900568 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:18.900331    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b1bf1c7-199d-4711-8e3e-8ba13481e036" containerName="kube-rbac-proxy"
Jun 02 17:00:18.900568 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:18.900340    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b1bf1c7-199d-4711-8e3e-8ba13481e036" containerName="kube-rbac-proxy"
Jun 02 17:00:18.900568 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:18.900359    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b1bf1c7-199d-4711-8e3e-8ba13481e036" containerName="kserve-container"
Jun 02 17:00:18.900568 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:18.900368    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b1bf1c7-199d-4711-8e3e-8ba13481e036" containerName="kserve-container"
Jun 02 17:00:18.900568 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:18.900452    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="2b1bf1c7-199d-4711-8e3e-8ba13481e036" containerName="kserve-container"
Jun 02 17:00:18.900568 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:18.900465    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="2b1bf1c7-199d-4711-8e3e-8ba13481e036" containerName="kube-rbac-proxy"
Jun 02 17:00:18.903363 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:18.903343    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zpxbm/must-gather-zmm4l"
Jun 02 17:00:18.905713 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:18.905695    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zpxbm\"/\"kube-root-ca.crt\""
Jun 02 17:00:18.906556 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:18.906535    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zpxbm\"/\"default-dockercfg-kvlpp\""
Jun 02 17:00:18.906664 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:18.906540    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zpxbm\"/\"openshift-service-ca.crt\""
Jun 02 17:00:18.908611 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:18.908590    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zpxbm/must-gather-zmm4l"]
Jun 02 17:00:19.003699 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:19.003670    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgcd6\" (UniqueName: \"kubernetes.io/projected/781bfb10-5f8d-48eb-8806-0705d45cfdcd-kube-api-access-mgcd6\") pod \"must-gather-zmm4l\" (UID: \"781bfb10-5f8d-48eb-8806-0705d45cfdcd\") " pod="openshift-must-gather-zpxbm/must-gather-zmm4l"
Jun 02 17:00:19.003823 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:19.003709    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/781bfb10-5f8d-48eb-8806-0705d45cfdcd-must-gather-output\") pod \"must-gather-zmm4l\" (UID: \"781bfb10-5f8d-48eb-8806-0705d45cfdcd\") " pod="openshift-must-gather-zpxbm/must-gather-zmm4l"
Jun 02 17:00:19.104423 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:19.104372    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mgcd6\" (UniqueName: \"kubernetes.io/projected/781bfb10-5f8d-48eb-8806-0705d45cfdcd-kube-api-access-mgcd6\") pod \"must-gather-zmm4l\" (UID: \"781bfb10-5f8d-48eb-8806-0705d45cfdcd\") " pod="openshift-must-gather-zpxbm/must-gather-zmm4l"
Jun 02 17:00:19.104570 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:19.104548    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/781bfb10-5f8d-48eb-8806-0705d45cfdcd-must-gather-output\") pod \"must-gather-zmm4l\" (UID: \"781bfb10-5f8d-48eb-8806-0705d45cfdcd\") " pod="openshift-must-gather-zpxbm/must-gather-zmm4l"
Jun 02 17:00:19.104821 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:19.104806    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/781bfb10-5f8d-48eb-8806-0705d45cfdcd-must-gather-output\") pod \"must-gather-zmm4l\" (UID: \"781bfb10-5f8d-48eb-8806-0705d45cfdcd\") " pod="openshift-must-gather-zpxbm/must-gather-zmm4l"
Jun 02 17:00:19.111770 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:19.111750    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgcd6\" (UniqueName: \"kubernetes.io/projected/781bfb10-5f8d-48eb-8806-0705d45cfdcd-kube-api-access-mgcd6\") pod \"must-gather-zmm4l\" (UID: \"781bfb10-5f8d-48eb-8806-0705d45cfdcd\") " pod="openshift-must-gather-zpxbm/must-gather-zmm4l"
Jun 02 17:00:19.212200 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:19.212147    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zpxbm/must-gather-zmm4l"
Jun 02 17:00:19.332780 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:19.332749    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zpxbm/must-gather-zmm4l"]
Jun 02 17:00:19.336111 ip-10-0-129-156 kubenswrapper[2579]: W0602 17:00:19.336063    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod781bfb10_5f8d_48eb_8806_0705d45cfdcd.slice/crio-7ed2864f9d220e7b00cb9329ac47e3295f5b26b61d349de455f815f73f05d951 WatchSource:0}: Error finding container 7ed2864f9d220e7b00cb9329ac47e3295f5b26b61d349de455f815f73f05d951: Status 404 returned error can't find the container with id 7ed2864f9d220e7b00cb9329ac47e3295f5b26b61d349de455f815f73f05d951
Jun 02 17:00:19.339778 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:19.339759    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 17:00:19.486779 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:19.486743    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zpxbm/must-gather-zmm4l" event={"ID":"781bfb10-5f8d-48eb-8806-0705d45cfdcd","Type":"ContainerStarted","Data":"7ed2864f9d220e7b00cb9329ac47e3295f5b26b61d349de455f815f73f05d951"}
Jun 02 17:00:20.497909 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:20.497873    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zpxbm/must-gather-zmm4l" event={"ID":"781bfb10-5f8d-48eb-8806-0705d45cfdcd","Type":"ContainerStarted","Data":"70d8681b787075c2050d08e08dc46d3d6ab1a3f847712210ae871e5200346d5f"}
Jun 02 17:00:20.497909 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:20.497914    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zpxbm/must-gather-zmm4l" event={"ID":"781bfb10-5f8d-48eb-8806-0705d45cfdcd","Type":"ContainerStarted","Data":"22109cede21a4cb0e6589f16e2fbfd82003cd4b9d17a17538790207d89db179e"}
Jun 02 17:00:20.513430 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:20.513354    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-zpxbm/must-gather-zmm4l" podStartSLOduration=1.706876588 podStartE2EDuration="2.513336806s" podCreationTimestamp="2026-06-02 17:00:18 +0000 UTC" firstStartedPulling="2026-06-02 17:00:19.339913674 +0000 UTC m=+4427.865822083" lastFinishedPulling="2026-06-02 17:00:20.146373891 +0000 UTC m=+4428.672282301" observedRunningTime="2026-06-02 17:00:20.511898471 +0000 UTC m=+4429.037806902" watchObservedRunningTime="2026-06-02 17:00:20.513336806 +0000 UTC m=+4429.039245239"
Jun 02 17:00:21.711105 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:21.711075    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vh7pj_32e4470f-53cd-4fda-b78b-404f9e5d45c3/global-pull-secret-syncer/0.log"
Jun 02 17:00:21.879120 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:21.879083    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zt7mm_7b0e93e2-a6b5-4b5a-be51-e09f43184379/konnectivity-agent/0.log"
Jun 02 17:00:21.896475 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:21.896431    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-156.ec2.internal_42688cc70769c2fbb9ca611c4f39c7ef/haproxy/0.log"
Jun 02 17:00:25.909203 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:25.909181    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6rlgc_29f69b2e-fd61-45ea-a3a9-3680485eb5fa/node-exporter/0.log"
Jun 02 17:00:25.930219 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:25.930193    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6rlgc_29f69b2e-fd61-45ea-a3a9-3680485eb5fa/kube-rbac-proxy/0.log"
Jun 02 17:00:25.948034 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:25.948006    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6rlgc_29f69b2e-fd61-45ea-a3a9-3680485eb5fa/init-textfile/0.log"
Jun 02 17:00:28.756778 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:28.756737    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zpxbm/perf-node-gather-daemonset-br78b"]
Jun 02 17:00:28.759851 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:28.759828    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-br78b"
Jun 02 17:00:28.768904 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:28.768880    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zpxbm/perf-node-gather-daemonset-br78b"]
Jun 02 17:00:28.889302 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:28.889264    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4da0be5-0c09-469f-aa12-79597e2117e9-sys\") pod \"perf-node-gather-daemonset-br78b\" (UID: \"f4da0be5-0c09-469f-aa12-79597e2117e9\") " pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-br78b"
Jun 02 17:00:28.889498 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:28.889311    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dknps\" (UniqueName: \"kubernetes.io/projected/f4da0be5-0c09-469f-aa12-79597e2117e9-kube-api-access-dknps\") pod \"perf-node-gather-daemonset-br78b\" (UID: \"f4da0be5-0c09-469f-aa12-79597e2117e9\") " pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-br78b"
Jun 02 17:00:28.889498 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:28.889344    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4da0be5-0c09-469f-aa12-79597e2117e9-lib-modules\") pod \"perf-node-gather-daemonset-br78b\" (UID: \"f4da0be5-0c09-469f-aa12-79597e2117e9\") " pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-br78b"
Jun 02 17:00:28.889498 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:28.889486    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f4da0be5-0c09-469f-aa12-79597e2117e9-proc\") pod \"perf-node-gather-daemonset-br78b\" (UID: \"f4da0be5-0c09-469f-aa12-79597e2117e9\") " pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-br78b"
Jun 02 17:00:28.889640 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:28.889524    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f4da0be5-0c09-469f-aa12-79597e2117e9-podres\") pod \"perf-node-gather-daemonset-br78b\" (UID: \"f4da0be5-0c09-469f-aa12-79597e2117e9\") " pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-br78b"
Jun 02 17:00:28.990098 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:28.990064    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4da0be5-0c09-469f-aa12-79597e2117e9-sys\") pod \"perf-node-gather-daemonset-br78b\" (UID: \"f4da0be5-0c09-469f-aa12-79597e2117e9\") " pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-br78b"
Jun 02 17:00:28.990258 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:28.990113    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dknps\" (UniqueName: \"kubernetes.io/projected/f4da0be5-0c09-469f-aa12-79597e2117e9-kube-api-access-dknps\") pod \"perf-node-gather-daemonset-br78b\" (UID: \"f4da0be5-0c09-469f-aa12-79597e2117e9\") " pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-br78b"
Jun 02 17:00:28.990258 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:28.990136    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4da0be5-0c09-469f-aa12-79597e2117e9-lib-modules\") pod \"perf-node-gather-daemonset-br78b\" (UID: \"f4da0be5-0c09-469f-aa12-79597e2117e9\") " pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-br78b"
Jun 02 17:00:28.990258 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:28.990179    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f4da0be5-0c09-469f-aa12-79597e2117e9-proc\") pod \"perf-node-gather-daemonset-br78b\" (UID: \"f4da0be5-0c09-469f-aa12-79597e2117e9\") " pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-br78b"
Jun 02 17:00:28.990258 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:28.990191    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4da0be5-0c09-469f-aa12-79597e2117e9-sys\") pod \"perf-node-gather-daemonset-br78b\" (UID: \"f4da0be5-0c09-469f-aa12-79597e2117e9\") " pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-br78b"
Jun 02 17:00:28.990258 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:28.990256    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f4da0be5-0c09-469f-aa12-79597e2117e9-proc\") pod \"perf-node-gather-daemonset-br78b\" (UID: \"f4da0be5-0c09-469f-aa12-79597e2117e9\") " pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-br78b"
Jun 02 17:00:28.990512 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:28.990201    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f4da0be5-0c09-469f-aa12-79597e2117e9-podres\") pod \"perf-node-gather-daemonset-br78b\" (UID: \"f4da0be5-0c09-469f-aa12-79597e2117e9\") " pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-br78b"
Jun 02 17:00:28.990512 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:28.990295    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f4da0be5-0c09-469f-aa12-79597e2117e9-podres\") pod \"perf-node-gather-daemonset-br78b\" (UID: \"f4da0be5-0c09-469f-aa12-79597e2117e9\") " pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-br78b"
Jun 02 17:00:28.990512 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:28.990379    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4da0be5-0c09-469f-aa12-79597e2117e9-lib-modules\") pod \"perf-node-gather-daemonset-br78b\" (UID: \"f4da0be5-0c09-469f-aa12-79597e2117e9\") " pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-br78b"
Jun 02 17:00:28.998229 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:28.998194    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dknps\" (UniqueName: \"kubernetes.io/projected/f4da0be5-0c09-469f-aa12-79597e2117e9-kube-api-access-dknps\") pod \"perf-node-gather-daemonset-br78b\" (UID: \"f4da0be5-0c09-469f-aa12-79597e2117e9\") " pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-br78b"
Jun 02 17:00:29.070293 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:29.070155    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-br78b"
Jun 02 17:00:29.222080 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:29.222047    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zpxbm/perf-node-gather-daemonset-br78b"]
Jun 02 17:00:29.225105 ip-10-0-129-156 kubenswrapper[2579]: W0602 17:00:29.225070    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf4da0be5_0c09_469f_aa12_79597e2117e9.slice/crio-0a1f09ad123c8d149af0b0ba0dedb6a30e48925a3cc91efeee813f37530e09d7 WatchSource:0}: Error finding container 0a1f09ad123c8d149af0b0ba0dedb6a30e48925a3cc91efeee813f37530e09d7: Status 404 returned error can't find the container with id 0a1f09ad123c8d149af0b0ba0dedb6a30e48925a3cc91efeee813f37530e09d7
Jun 02 17:00:29.402566 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:29.402504    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bgsrq_666f2157-8d74-421e-af17-78b99e918795/dns/0.log"
Jun 02 17:00:29.420725 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:29.420700    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bgsrq_666f2157-8d74-421e-af17-78b99e918795/kube-rbac-proxy/0.log"
Jun 02 17:00:29.528290 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:29.528248    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-br78b" event={"ID":"f4da0be5-0c09-469f-aa12-79597e2117e9","Type":"ContainerStarted","Data":"8675ded049d7882477844c17eb28a7cc7e39b7dbbea0e19e31d5bf64ecc47dfe"}
Jun 02 17:00:29.528290 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:29.528282    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-br78b" event={"ID":"f4da0be5-0c09-469f-aa12-79597e2117e9","Type":"ContainerStarted","Data":"0a1f09ad123c8d149af0b0ba0dedb6a30e48925a3cc91efeee813f37530e09d7"}
Jun 02 17:00:29.528501 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:29.528366    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-br78b"
Jun 02 17:00:29.543160 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:29.543113    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-br78b" podStartSLOduration=1.5430998489999999 podStartE2EDuration="1.543099849s" podCreationTimestamp="2026-06-02 17:00:28 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 17:00:29.541843682 +0000 UTC m=+4438.067752233" watchObservedRunningTime="2026-06-02 17:00:29.543099849 +0000 UTC m=+4438.069008280"
Jun 02 17:00:29.564083 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:29.564064    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-th978_4290cf74-a174-4c3f-88b4-66b05bbe4750/dns-node-resolver/0.log"
Jun 02 17:00:29.952716 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:29.952677    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-77b6f9cc7b-d8g9c_00f0d738-668c-4e32-99d0-137724dca252/registry/0.log"
Jun 02 17:00:29.974925 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:29.974894    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cwjmt_3bee98d4-d583-4ecb-b07b-9f422a9810da/node-ca/0.log"
Jun 02 17:00:30.997274 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:30.997249    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cqh44_5a9d0209-8692-4052-bb84-c3b8bb9d383e/serve-healthcheck-canary/0.log"
Jun 02 17:00:31.442715 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:31.442632    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-82hzh_b959e92c-6548-4cb4-945c-e229516c5e9c/kube-rbac-proxy/0.log"
Jun 02 17:00:31.460394 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:31.460370    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-82hzh_b959e92c-6548-4cb4-945c-e229516c5e9c/exporter/0.log"
Jun 02 17:00:31.478147 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:31.478118    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-82hzh_b959e92c-6548-4cb4-945c-e229516c5e9c/extractor/0.log"
Jun 02 17:00:33.787743 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:33.787701    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-696fc77849-hj7x7_4c713f6d-8f54-42fa-83c4-0fba73803c43/manager/0.log"
Jun 02 17:00:33.866526 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:33.866484    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-79f457656b-r48bb_f3c112f5-381e-4196-85fc-fbeda63c6b43/seaweedfs/0.log"
Jun 02 17:00:35.541613 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:35.541581    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-br78b"
Jun 02 17:00:38.849860 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:38.849733    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dc6wj_810b8e3e-5836-4214-a141-c340d2220639/kube-multus-additional-cni-plugins/0.log"
Jun 02 17:00:38.877083 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:38.877051    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dc6wj_810b8e3e-5836-4214-a141-c340d2220639/egress-router-binary-copy/0.log"
Jun 02 17:00:38.894027 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:38.893994    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dc6wj_810b8e3e-5836-4214-a141-c340d2220639/cni-plugins/0.log"
Jun 02 17:00:38.911826 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:38.911800    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dc6wj_810b8e3e-5836-4214-a141-c340d2220639/bond-cni-plugin/0.log"
Jun 02 17:00:38.932477 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:38.932448    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dc6wj_810b8e3e-5836-4214-a141-c340d2220639/routeoverride-cni/0.log"
Jun 02 17:00:38.959474 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:38.959435    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dc6wj_810b8e3e-5836-4214-a141-c340d2220639/whereabouts-cni-bincopy/0.log"
Jun 02 17:00:38.978599 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:38.978572    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dc6wj_810b8e3e-5836-4214-a141-c340d2220639/whereabouts-cni/0.log"
Jun 02 17:00:39.305952 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:39.305911    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-lg8nx_73b94606-f892-4c47-9022-5d42f71bb77d/kube-multus/0.log"
Jun 02 17:00:39.382267 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:39.382238    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6lvc8_eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca/network-metrics-daemon/0.log"
Jun 02 17:00:39.398601 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:39.398567    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6lvc8_eb58a64a-e8f6-4492-9aa2-9c9b6bf926ca/kube-rbac-proxy/0.log"
Jun 02 17:00:40.564030 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:40.563995    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lqx9k_6127e33c-aa77-437e-b443-e956037afedb/ovn-controller/0.log"
Jun 02 17:00:40.578134 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:40.578102    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lqx9k_6127e33c-aa77-437e-b443-e956037afedb/ovn-acl-logging/0.log"
Jun 02 17:00:40.597640 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:40.597610    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lqx9k_6127e33c-aa77-437e-b443-e956037afedb/ovn-acl-logging/1.log"
Jun 02 17:00:40.614279 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:40.614252    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lqx9k_6127e33c-aa77-437e-b443-e956037afedb/kube-rbac-proxy-node/0.log"
Jun 02 17:00:40.632324 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:40.632292    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lqx9k_6127e33c-aa77-437e-b443-e956037afedb/kube-rbac-proxy-ovn-metrics/0.log"
Jun 02 17:00:40.647795 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:40.647768    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lqx9k_6127e33c-aa77-437e-b443-e956037afedb/northd/0.log"
Jun 02 17:00:40.666080 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:40.666056    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lqx9k_6127e33c-aa77-437e-b443-e956037afedb/nbdb/0.log"
Jun 02 17:00:40.684020 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:40.683991    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lqx9k_6127e33c-aa77-437e-b443-e956037afedb/sbdb/0.log"
Jun 02 17:00:40.791339 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:40.791256    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lqx9k_6127e33c-aa77-437e-b443-e956037afedb/ovnkube-controller/0.log"
Jun 02 17:00:42.021894 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:42.021865    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-zzt45_43370289-88af-4e83-b409-4e8a98e09857/network-check-target-container/0.log"
Jun 02 17:00:42.927699 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:42.927664    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-zqsll_c0c551c2-7ea0-4de3-b24a-735cdb0c536d/iptables-alerter/0.log"
Jun 02 17:00:43.594889 ip-10-0-129-156 kubenswrapper[2579]: I0602 17:00:43.594860    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-mllz2_498f7354-865b-4ad1-9e9f-fabd57febd4a/tuned/0.log"