Apr 16 16:47:40.804546 ip-10-0-142-37 systemd[1]: Starting Kubernetes Kubelet... Apr 16 16:47:41.305386 ip-10-0-142-37 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 16 16:47:41.305386 ip-10-0-142-37 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Apr 16 16:47:41.305386 ip-10-0-142-37 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 16 16:47:41.305386 ip-10-0-142-37 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Apr 16 16:47:41.305386 ip-10-0-142-37 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 16 16:47:41.307263 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.307164 2576 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Apr 16 16:47:41.313008 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.312986 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Apr 16 16:47:41.313008 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313004 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Apr 16 16:47:41.313008 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313008 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Apr 16 16:47:41.313008 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313012 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Apr 16 16:47:41.313171 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313015 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Apr 16 16:47:41.313171 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313019 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Apr 16 16:47:41.313171 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313021 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Apr 16 16:47:41.313171 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313024 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Apr 16 16:47:41.313171 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313027 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Apr 16 16:47:41.313171 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313029 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Apr 16 16:47:41.313171 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313032 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Apr 16 16:47:41.313171 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313035 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Apr 16 16:47:41.313171 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313037 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Apr 16 16:47:41.313171 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313039 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Apr 16 16:47:41.313171 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313042 2576 feature_gate.go:328] unrecognized feature gate: Example Apr 16 16:47:41.313171 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313045 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Apr 16 16:47:41.313171 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313047 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Apr 16 16:47:41.313171 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313050 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS Apr 16 16:47:41.313171 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313052 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Apr 16 16:47:41.313171 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313055 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Apr 16 16:47:41.313171 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313057 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Apr 16 16:47:41.313171 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313060 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Apr 16 16:47:41.313171 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313062 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Apr 16 16:47:41.313171 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313065 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Apr 16 16:47:41.313667 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313067 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Apr 16 16:47:41.313667 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313070 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Apr 16 16:47:41.313667 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313073 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Apr 16 16:47:41.313667 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313076 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Apr 16 16:47:41.313667 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313079 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Apr 16 16:47:41.313667 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313082 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Apr 16 16:47:41.313667 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313084 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations Apr 16 16:47:41.313667 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313088 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Apr 16 16:47:41.313667 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313090 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Apr 16 16:47:41.313667 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313093 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Apr 16 16:47:41.313667 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313095 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Apr 16 16:47:41.313667 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313098 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Apr 16 16:47:41.313667 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313101 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Apr 16 16:47:41.313667 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313104 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Apr 16 16:47:41.313667 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313107 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Apr 16 16:47:41.313667 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313110 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Apr 16 16:47:41.313667 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313112 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Apr 16 16:47:41.313667 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313115 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Apr 16 16:47:41.313667 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313119 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Apr 16 16:47:41.313667 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313121 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Apr 16 16:47:41.314169 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313124 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Apr 16 16:47:41.314169 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313126 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Apr 16 16:47:41.314169 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313129 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Apr 16 16:47:41.314169 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313131 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Apr 16 16:47:41.314169 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313134 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Apr 16 16:47:41.314169 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313137 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Apr 16 16:47:41.314169 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313139 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Apr 16 16:47:41.314169 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313142 2576 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController Apr 16 16:47:41.314169 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313145 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Apr 16 16:47:41.314169 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313147 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Apr 16 16:47:41.314169 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313150 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Apr 16 16:47:41.314169 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313153 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Apr 16 16:47:41.314169 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313155 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Apr 16 16:47:41.314169 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313159 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Apr 16 16:47:41.314169 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313164 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Apr 16 16:47:41.314169 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313167 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Apr 16 16:47:41.314169 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313169 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Apr 16 16:47:41.314169 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313172 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Apr 16 16:47:41.314169 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313175 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Apr 16 16:47:41.314681 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313179 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Apr 16 16:47:41.314681 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313182 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Apr 16 16:47:41.314681 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313184 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Apr 16 16:47:41.314681 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313188 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Apr 16 16:47:41.314681 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313191 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Apr 16 16:47:41.314681 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313194 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Apr 16 16:47:41.314681 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313197 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Apr 16 16:47:41.314681 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313200 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Apr 16 16:47:41.314681 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313203 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Apr 16 16:47:41.314681 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313205 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Apr 16 16:47:41.314681 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313208 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Apr 16 16:47:41.314681 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313210 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Apr 16 16:47:41.314681 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313213 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Apr 16 16:47:41.314681 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313216 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Apr 16 16:47:41.314681 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313218 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Apr 16 16:47:41.314681 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313222 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Apr 16 16:47:41.314681 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313224 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Apr 16 16:47:41.314681 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313227 2576 feature_gate.go:328] unrecognized feature gate: Example2 Apr 16 16:47:41.314681 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313229 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI Apr 16 16:47:41.315208 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313232 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Apr 16 16:47:41.315208 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313235 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Apr 16 16:47:41.315208 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313237 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Apr 16 16:47:41.315208 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313240 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Apr 16 16:47:41.315208 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313667 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Apr 16 16:47:41.315208 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313673 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Apr 16 16:47:41.315208 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313676 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Apr 16 16:47:41.315208 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313679 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Apr 16 16:47:41.315208 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313682 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Apr 16 16:47:41.315208 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313685 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Apr 16 16:47:41.315208 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313688 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Apr 16 16:47:41.315208 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313691 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Apr 16 16:47:41.315208 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313693 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Apr 16 16:47:41.315208 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313696 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Apr 16 16:47:41.315208 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313698 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI Apr 16 16:47:41.315208 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313701 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Apr 16 16:47:41.315208 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313704 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Apr 16 16:47:41.315208 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313706 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Apr 16 16:47:41.315208 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313709 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Apr 16 16:47:41.315208 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313712 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Apr 16 16:47:41.315706 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313714 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Apr 16 16:47:41.315706 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313718 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Apr 16 16:47:41.315706 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313722 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Apr 16 16:47:41.315706 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313725 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Apr 16 16:47:41.315706 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313728 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Apr 16 16:47:41.315706 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313730 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Apr 16 16:47:41.315706 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313733 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Apr 16 16:47:41.315706 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313736 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Apr 16 16:47:41.315706 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313739 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Apr 16 16:47:41.315706 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313741 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Apr 16 16:47:41.315706 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313744 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Apr 16 16:47:41.315706 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313746 2576 feature_gate.go:328] unrecognized feature gate: Example2 Apr 16 16:47:41.315706 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313749 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Apr 16 16:47:41.315706 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313751 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Apr 16 16:47:41.315706 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313754 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Apr 16 16:47:41.315706 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313756 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Apr 16 16:47:41.315706 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313758 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Apr 16 16:47:41.315706 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313761 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Apr 16 16:47:41.315706 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313763 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Apr 16 16:47:41.316170 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313765 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations Apr 16 16:47:41.316170 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313768 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Apr 16 16:47:41.316170 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313770 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Apr 16 16:47:41.316170 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313773 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Apr 16 16:47:41.316170 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313776 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Apr 16 16:47:41.316170 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313779 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Apr 16 16:47:41.316170 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313782 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Apr 16 16:47:41.316170 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313785 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Apr 16 16:47:41.316170 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313787 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Apr 16 16:47:41.316170 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313790 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Apr 16 16:47:41.316170 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313792 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Apr 16 16:47:41.316170 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313795 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Apr 16 16:47:41.316170 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313797 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Apr 16 16:47:41.316170 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313799 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Apr 16 16:47:41.316170 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313802 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Apr 16 16:47:41.316170 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313804 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Apr 16 16:47:41.316170 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313807 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Apr 16 16:47:41.316170 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313809 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Apr 16 16:47:41.316170 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313812 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Apr 16 16:47:41.316170 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313814 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Apr 16 16:47:41.316683 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313817 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Apr 16 16:47:41.316683 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313819 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Apr 16 16:47:41.316683 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313822 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Apr 16 16:47:41.316683 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313824 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Apr 16 16:47:41.316683 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313827 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Apr 16 16:47:41.316683 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313829 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Apr 16 16:47:41.316683 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313832 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Apr 16 16:47:41.316683 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313834 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Apr 16 16:47:41.316683 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313837 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS Apr 16 16:47:41.316683 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313840 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Apr 16 16:47:41.316683 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313843 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Apr 16 16:47:41.316683 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313845 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Apr 16 16:47:41.316683 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313848 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Apr 16 16:47:41.316683 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313850 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Apr 16 16:47:41.316683 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313854 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Apr 16 16:47:41.316683 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313856 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Apr 16 16:47:41.316683 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313859 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Apr 16 16:47:41.316683 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313861 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Apr 16 16:47:41.316683 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313864 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Apr 16 16:47:41.316683 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313866 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Apr 16 16:47:41.317162 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313871 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Apr 16 16:47:41.317162 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313874 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Apr 16 16:47:41.317162 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313877 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Apr 16 16:47:41.317162 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313879 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Apr 16 16:47:41.317162 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313882 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Apr 16 16:47:41.317162 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313884 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Apr 16 16:47:41.317162 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313887 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Apr 16 16:47:41.317162 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313889 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Apr 16 16:47:41.317162 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313891 2576 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController Apr 16 16:47:41.317162 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313894 2576 feature_gate.go:328] unrecognized feature gate: Example Apr 16 16:47:41.317162 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.313896 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Apr 16 16:47:41.317162 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.313986 2576 flags.go:64] FLAG: --address="0.0.0.0" Apr 16 16:47:41.317162 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.313994 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Apr 16 16:47:41.317162 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314000 2576 flags.go:64] FLAG: --anonymous-auth="true" Apr 16 16:47:41.317162 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314004 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Apr 16 16:47:41.317162 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314010 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Apr 16 16:47:41.317162 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314013 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Apr 16 16:47:41.317162 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314017 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Apr 16 16:47:41.317162 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314022 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Apr 16 16:47:41.317162 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314025 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Apr 16 16:47:41.317662 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314027 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Apr 16 16:47:41.317662 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314031 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Apr 16 16:47:41.317662 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314034 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Apr 16 16:47:41.317662 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314037 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Apr 16 16:47:41.317662 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314040 2576 flags.go:64] FLAG: --cgroup-root="" Apr 16 16:47:41.317662 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314043 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Apr 16 16:47:41.317662 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314046 2576 flags.go:64] FLAG: --client-ca-file="" Apr 16 16:47:41.317662 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314049 2576 flags.go:64] FLAG: --cloud-config="" Apr 16 16:47:41.317662 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314052 2576 flags.go:64] FLAG: --cloud-provider="external" Apr 16 16:47:41.317662 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314055 2576 flags.go:64] FLAG: --cluster-dns="[]" Apr 16 16:47:41.317662 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314060 2576 flags.go:64] FLAG: --cluster-domain="" Apr 16 16:47:41.317662 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314062 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Apr 16 16:47:41.317662 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314065 2576 flags.go:64] FLAG: --config-dir="" Apr 16 16:47:41.317662 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314068 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Apr 16 16:47:41.317662 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314071 2576 flags.go:64] FLAG: --container-log-max-files="5" Apr 16 16:47:41.317662 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314075 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Apr 16 16:47:41.317662 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314078 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Apr 16 16:47:41.317662 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314081 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Apr 16 16:47:41.317662 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314084 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Apr 16 16:47:41.317662 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314087 2576 flags.go:64] FLAG: --contention-profiling="false" Apr 16 16:47:41.317662 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314090 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Apr 16 16:47:41.317662 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314093 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Apr 16 16:47:41.317662 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314096 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Apr 16 16:47:41.317662 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314098 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Apr 16 16:47:41.317662 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314107 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Apr 16 16:47:41.318285 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314110 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Apr 16 16:47:41.318285 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314113 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Apr 16 16:47:41.318285 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314116 2576 flags.go:64] FLAG: --enable-load-reader="false" Apr 16 16:47:41.318285 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314120 2576 flags.go:64] FLAG: --enable-server="true" Apr 16 16:47:41.318285 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314123 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Apr 16 16:47:41.318285 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314127 2576 flags.go:64] FLAG: --event-burst="100" Apr 16 16:47:41.318285 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314130 2576 flags.go:64] FLAG: --event-qps="50" Apr 16 16:47:41.318285 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314133 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Apr 16 16:47:41.318285 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314136 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Apr 16 16:47:41.318285 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314139 2576 flags.go:64] FLAG: --eviction-hard="" Apr 16 16:47:41.318285 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314143 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Apr 16 16:47:41.318285 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314146 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Apr 16 16:47:41.318285 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314149 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Apr 16 16:47:41.318285 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314152 2576 flags.go:64] FLAG: --eviction-soft="" Apr 16 16:47:41.318285 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314156 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Apr 16 16:47:41.318285 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314159 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Apr 16 16:47:41.318285 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314162 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Apr 16 16:47:41.318285 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314166 2576 flags.go:64] FLAG: --experimental-mounter-path="" Apr 16 16:47:41.318285 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314169 2576 flags.go:64] FLAG: --fail-cgroupv1="false" Apr 16 16:47:41.318285 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314171 2576 flags.go:64] FLAG: --fail-swap-on="true" Apr 16 16:47:41.318285 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314175 2576 flags.go:64] FLAG: --feature-gates="" Apr 16 16:47:41.318285 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314179 2576 flags.go:64] FLAG: --file-check-frequency="20s" Apr 16 16:47:41.318285 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314182 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Apr 16 16:47:41.318285 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314185 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Apr 16 16:47:41.318285 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314188 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Apr 16 16:47:41.318892 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314191 2576 flags.go:64] FLAG: --healthz-port="10248" Apr 16 16:47:41.318892 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314194 2576 flags.go:64] FLAG: --help="false" Apr 16 16:47:41.318892 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314197 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-142-37.ec2.internal" Apr 16 16:47:41.318892 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314200 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Apr 16 16:47:41.318892 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314203 2576 flags.go:64] FLAG: --http-check-frequency="20s" Apr 16 16:47:41.318892 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314205 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Apr 16 16:47:41.318892 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314209 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Apr 16 16:47:41.318892 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314212 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Apr 16 16:47:41.318892 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314215 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Apr 16 16:47:41.318892 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314217 2576 flags.go:64] FLAG: --image-service-endpoint="" Apr 16 16:47:41.318892 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314220 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Apr 16 16:47:41.318892 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314223 2576 flags.go:64] FLAG: --kube-api-burst="100" Apr 16 16:47:41.318892 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314226 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Apr 16 16:47:41.318892 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314229 2576 flags.go:64] FLAG: --kube-api-qps="50" Apr 16 16:47:41.318892 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314232 2576 flags.go:64] FLAG: --kube-reserved="" Apr 16 16:47:41.318892 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314234 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Apr 16 16:47:41.318892 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314237 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Apr 16 16:47:41.318892 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314240 2576 flags.go:64] FLAG: --kubelet-cgroups="" Apr 16 16:47:41.318892 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314260 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Apr 16 16:47:41.318892 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314263 2576 flags.go:64] FLAG: --lock-file="" Apr 16 16:47:41.318892 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314267 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Apr 16 16:47:41.318892 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314270 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Apr 16 16:47:41.318892 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314273 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Apr 16 16:47:41.318892 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314278 2576 flags.go:64] FLAG: --log-json-split-stream="false" Apr 16 16:47:41.319547 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314281 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Apr 16 16:47:41.319547 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314283 2576 flags.go:64] FLAG: --log-text-split-stream="false" Apr 16 16:47:41.319547 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314286 2576 flags.go:64] FLAG: --logging-format="text" Apr 16 16:47:41.319547 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314289 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Apr 16 16:47:41.319547 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314292 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Apr 16 16:47:41.319547 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314295 2576 flags.go:64] FLAG: --manifest-url="" Apr 16 16:47:41.319547 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314297 2576 flags.go:64] FLAG: --manifest-url-header="" Apr 16 16:47:41.319547 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314302 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Apr 16 16:47:41.319547 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314304 2576 flags.go:64] FLAG: --max-open-files="1000000" Apr 16 16:47:41.319547 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314309 2576 flags.go:64] FLAG: --max-pods="110" Apr 16 16:47:41.319547 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314311 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Apr 16 16:47:41.319547 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314314 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Apr 16 16:47:41.319547 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314317 2576 flags.go:64] FLAG: --memory-manager-policy="None" Apr 16 16:47:41.319547 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314320 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Apr 16 16:47:41.319547 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314323 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Apr 16 16:47:41.319547 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314326 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Apr 16 16:47:41.319547 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314328 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Apr 16 16:47:41.319547 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314336 2576 flags.go:64] FLAG: --node-status-max-images="50" Apr 16 16:47:41.319547 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314339 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Apr 16 16:47:41.319547 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314341 2576 flags.go:64] FLAG: --oom-score-adj="-999" Apr 16 16:47:41.319547 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314345 2576 flags.go:64] FLAG: --pod-cidr="" Apr 16 16:47:41.319547 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314348 2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dc76bab72f320de3d4105c90d73c4fb139c09e20ce0fa8dcbc0cb59920d27dec" Apr 16 16:47:41.319547 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314353 2576 flags.go:64] FLAG: --pod-manifest-path="" Apr 16 16:47:41.320097 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314355 2576 flags.go:64] FLAG: --pod-max-pids="-1" Apr 16 16:47:41.320097 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314358 2576 flags.go:64] FLAG: --pods-per-core="0" Apr 16 16:47:41.320097 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314364 2576 flags.go:64] FLAG: --port="10250" Apr 16 16:47:41.320097 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314366 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Apr 16 16:47:41.320097 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314369 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07be6b0a0de208418" Apr 16 16:47:41.320097 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314372 2576 flags.go:64] FLAG: --qos-reserved="" Apr 16 16:47:41.320097 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314375 2576 flags.go:64] FLAG: --read-only-port="10255" Apr 16 16:47:41.320097 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314378 2576 flags.go:64] FLAG: --register-node="true" Apr 16 16:47:41.320097 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314381 2576 flags.go:64] FLAG: --register-schedulable="true" Apr 16 16:47:41.320097 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314384 2576 flags.go:64] FLAG: --register-with-taints="" Apr 16 16:47:41.320097 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314388 2576 flags.go:64] FLAG: --registry-burst="10" Apr 16 16:47:41.320097 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314391 2576 flags.go:64] FLAG: --registry-qps="5" Apr 16 16:47:41.320097 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314393 2576 flags.go:64] FLAG: --reserved-cpus="" Apr 16 16:47:41.320097 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314396 2576 flags.go:64] FLAG: --reserved-memory="" Apr 16 16:47:41.320097 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314400 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Apr 16 16:47:41.320097 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314403 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Apr 16 16:47:41.320097 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314405 2576 flags.go:64] FLAG: --rotate-certificates="false" Apr 16 16:47:41.320097 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314408 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Apr 16 16:47:41.320097 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314412 2576 flags.go:64] FLAG: --runonce="false" Apr 16 16:47:41.320097 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314414 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Apr 16 16:47:41.320097 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314417 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Apr 16 16:47:41.320097 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314420 2576 flags.go:64] FLAG: --seccomp-default="false" Apr 16 16:47:41.320097 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314423 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Apr 16 16:47:41.320097 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314426 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Apr 16 16:47:41.320097 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314428 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Apr 16 16:47:41.320097 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314432 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Apr 16 16:47:41.320729 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314434 2576 flags.go:64] FLAG: --storage-driver-password="root" Apr 16 16:47:41.320729 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314437 2576 flags.go:64] FLAG: --storage-driver-secure="false" Apr 16 16:47:41.320729 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314440 2576 flags.go:64] FLAG: --storage-driver-table="stats" Apr 16 16:47:41.320729 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314443 2576 flags.go:64] FLAG: --storage-driver-user="root" Apr 16 16:47:41.320729 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314447 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Apr 16 16:47:41.320729 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314450 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Apr 16 16:47:41.320729 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314453 2576 flags.go:64] FLAG: --system-cgroups="" Apr 16 16:47:41.320729 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314455 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Apr 16 16:47:41.320729 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314462 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Apr 16 16:47:41.320729 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314465 2576 flags.go:64] FLAG: --tls-cert-file="" Apr 16 16:47:41.320729 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314469 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Apr 16 16:47:41.320729 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314474 2576 flags.go:64] FLAG: --tls-min-version="" Apr 16 16:47:41.320729 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314477 2576 flags.go:64] FLAG: --tls-private-key-file="" Apr 16 16:47:41.320729 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314480 2576 flags.go:64] FLAG: --topology-manager-policy="none" Apr 16 16:47:41.320729 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314482 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Apr 16 16:47:41.320729 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314485 2576 flags.go:64] FLAG: --topology-manager-scope="container" Apr 16 16:47:41.320729 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314488 2576 flags.go:64] FLAG: --v="2" Apr 16 16:47:41.320729 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314492 2576 flags.go:64] FLAG: --version="false" Apr 16 16:47:41.320729 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314497 2576 flags.go:64] FLAG: --vmodule="" Apr 16 16:47:41.320729 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314501 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Apr 16 16:47:41.320729 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.314504 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Apr 16 16:47:41.320729 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314592 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Apr 16 16:47:41.320729 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314595 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations Apr 16 16:47:41.320729 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314598 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Apr 16 16:47:41.321336 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314601 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Apr 16 16:47:41.321336 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314604 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Apr 16 16:47:41.321336 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314607 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Apr 16 16:47:41.321336 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314610 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Apr 16 16:47:41.321336 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314614 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Apr 16 16:47:41.321336 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314617 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Apr 16 16:47:41.321336 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314620 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Apr 16 16:47:41.321336 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314622 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Apr 16 16:47:41.321336 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314625 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Apr 16 16:47:41.321336 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314627 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Apr 16 16:47:41.321336 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314630 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Apr 16 16:47:41.321336 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314633 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Apr 16 16:47:41.321336 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314636 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Apr 16 16:47:41.321336 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314639 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Apr 16 16:47:41.321336 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314642 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Apr 16 16:47:41.321336 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314644 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Apr 16 16:47:41.321336 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314648 2576 feature_gate.go:328] unrecognized feature gate: Example Apr 16 16:47:41.321336 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314651 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Apr 16 16:47:41.321336 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314654 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Apr 16 16:47:41.321336 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314657 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Apr 16 16:47:41.321828 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314659 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Apr 16 16:47:41.321828 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314662 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Apr 16 16:47:41.321828 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314664 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Apr 16 16:47:41.321828 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314667 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Apr 16 16:47:41.321828 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314669 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Apr 16 16:47:41.321828 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314672 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Apr 16 16:47:41.321828 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314674 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Apr 16 16:47:41.321828 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314676 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Apr 16 16:47:41.321828 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314679 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Apr 16 16:47:41.321828 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314681 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Apr 16 16:47:41.321828 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314684 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Apr 16 16:47:41.321828 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314686 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Apr 16 16:47:41.321828 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314689 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Apr 16 16:47:41.321828 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314692 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Apr 16 16:47:41.321828 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314694 2576 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController Apr 16 16:47:41.321828 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314697 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Apr 16 16:47:41.321828 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314700 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Apr 16 16:47:41.321828 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314702 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Apr 16 16:47:41.321828 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314705 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Apr 16 16:47:41.322302 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314707 2576 feature_gate.go:328] unrecognized feature gate: Example2 Apr 16 16:47:41.322302 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314709 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Apr 16 16:47:41.322302 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314712 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Apr 16 16:47:41.322302 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314714 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Apr 16 16:47:41.322302 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314717 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Apr 16 16:47:41.322302 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314719 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Apr 16 16:47:41.322302 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314721 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Apr 16 16:47:41.322302 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314724 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Apr 16 16:47:41.322302 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314727 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Apr 16 16:47:41.322302 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314730 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Apr 16 16:47:41.322302 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314733 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Apr 16 16:47:41.322302 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314737 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Apr 16 16:47:41.322302 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314740 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Apr 16 16:47:41.322302 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314742 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Apr 16 16:47:41.322302 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314744 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Apr 16 16:47:41.322302 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314747 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Apr 16 16:47:41.322302 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314749 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Apr 16 16:47:41.322302 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314752 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Apr 16 16:47:41.322302 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314754 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Apr 16 16:47:41.322302 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314757 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI Apr 16 16:47:41.322795 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314759 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Apr 16 16:47:41.322795 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314763 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Apr 16 16:47:41.322795 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314766 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Apr 16 16:47:41.322795 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314769 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS Apr 16 16:47:41.322795 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314771 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Apr 16 16:47:41.322795 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314774 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Apr 16 16:47:41.322795 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314777 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Apr 16 16:47:41.322795 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314779 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Apr 16 16:47:41.322795 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314782 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Apr 16 16:47:41.322795 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314784 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Apr 16 16:47:41.322795 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314787 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Apr 16 16:47:41.322795 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314789 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Apr 16 16:47:41.322795 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314792 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Apr 16 16:47:41.322795 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314794 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Apr 16 16:47:41.322795 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314797 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Apr 16 16:47:41.322795 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314800 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Apr 16 16:47:41.322795 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314802 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Apr 16 16:47:41.322795 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314804 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Apr 16 16:47:41.322795 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314807 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Apr 16 16:47:41.323310 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314809 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Apr 16 16:47:41.323310 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314812 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Apr 16 16:47:41.323310 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314816 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Apr 16 16:47:41.323310 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314819 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Apr 16 16:47:41.323310 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.314822 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Apr 16 16:47:41.323310 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.315566 2576 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} Apr 16 16:47:41.323310 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.322567 2576 server.go:530] "Kubelet version" kubeletVersion="v1.33.9" Apr 16 16:47:41.323310 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.322583 2576 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Apr 16 16:47:41.323310 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322629 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Apr 16 16:47:41.323310 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322634 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Apr 16 16:47:41.323310 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322637 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Apr 16 16:47:41.323310 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322640 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Apr 16 16:47:41.323310 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322643 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Apr 16 16:47:41.323310 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322645 2576 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController Apr 16 16:47:41.323310 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322648 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Apr 16 16:47:41.323310 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322651 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Apr 16 16:47:41.323712 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322653 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Apr 16 16:47:41.323712 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322656 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Apr 16 16:47:41.323712 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322659 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Apr 16 16:47:41.323712 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322661 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Apr 16 16:47:41.323712 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322664 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Apr 16 16:47:41.323712 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322666 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Apr 16 16:47:41.323712 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322669 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Apr 16 16:47:41.323712 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322671 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Apr 16 16:47:41.323712 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322674 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Apr 16 16:47:41.323712 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322676 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Apr 16 16:47:41.323712 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322678 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Apr 16 16:47:41.323712 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322683 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Apr 16 16:47:41.323712 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322687 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Apr 16 16:47:41.323712 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322690 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Apr 16 16:47:41.323712 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322693 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Apr 16 16:47:41.323712 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322696 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Apr 16 16:47:41.323712 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322699 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Apr 16 16:47:41.323712 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322702 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Apr 16 16:47:41.323712 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322705 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Apr 16 16:47:41.324164 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322708 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Apr 16 16:47:41.324164 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322711 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Apr 16 16:47:41.324164 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322714 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Apr 16 16:47:41.324164 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322717 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Apr 16 16:47:41.324164 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322720 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Apr 16 16:47:41.324164 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322723 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Apr 16 16:47:41.324164 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322725 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Apr 16 16:47:41.324164 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322728 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Apr 16 16:47:41.324164 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322730 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Apr 16 16:47:41.324164 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322733 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Apr 16 16:47:41.324164 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322736 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Apr 16 16:47:41.324164 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322738 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Apr 16 16:47:41.324164 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322741 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Apr 16 16:47:41.324164 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322743 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Apr 16 16:47:41.324164 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322746 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Apr 16 16:47:41.324164 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322748 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Apr 16 16:47:41.324164 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322752 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Apr 16 16:47:41.324164 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322756 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Apr 16 16:47:41.324164 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322759 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Apr 16 16:47:41.324164 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322761 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Apr 16 16:47:41.324663 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322764 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Apr 16 16:47:41.324663 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322766 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Apr 16 16:47:41.324663 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322769 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI Apr 16 16:47:41.324663 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322771 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Apr 16 16:47:41.324663 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322774 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Apr 16 16:47:41.324663 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322776 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Apr 16 16:47:41.324663 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322779 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Apr 16 16:47:41.324663 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322781 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Apr 16 16:47:41.324663 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322784 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Apr 16 16:47:41.324663 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322787 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Apr 16 16:47:41.324663 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322790 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Apr 16 16:47:41.324663 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322793 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Apr 16 16:47:41.324663 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322796 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Apr 16 16:47:41.324663 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322798 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Apr 16 16:47:41.324663 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322801 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Apr 16 16:47:41.324663 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322804 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Apr 16 16:47:41.324663 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322807 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Apr 16 16:47:41.324663 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322810 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Apr 16 16:47:41.324663 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322812 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Apr 16 16:47:41.324663 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322815 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations Apr 16 16:47:41.325129 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322817 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Apr 16 16:47:41.325129 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322820 2576 feature_gate.go:328] unrecognized feature gate: Example Apr 16 16:47:41.325129 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322822 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Apr 16 16:47:41.325129 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322825 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Apr 16 16:47:41.325129 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322827 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Apr 16 16:47:41.325129 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322830 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS Apr 16 16:47:41.325129 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322832 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Apr 16 16:47:41.325129 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322835 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Apr 16 16:47:41.325129 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322837 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Apr 16 16:47:41.325129 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322839 2576 feature_gate.go:328] unrecognized feature gate: Example2 Apr 16 16:47:41.325129 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322842 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Apr 16 16:47:41.325129 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322844 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Apr 16 16:47:41.325129 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322847 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Apr 16 16:47:41.325129 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322850 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Apr 16 16:47:41.325129 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322853 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Apr 16 16:47:41.325129 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322855 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Apr 16 16:47:41.325129 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322858 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Apr 16 16:47:41.325129 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322860 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Apr 16 16:47:41.325129 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322863 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Apr 16 16:47:41.325616 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.322868 2576 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} Apr 16 16:47:41.325616 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322959 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Apr 16 16:47:41.325616 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322963 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Apr 16 16:47:41.325616 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322966 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Apr 16 16:47:41.325616 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322968 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Apr 16 16:47:41.325616 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322972 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Apr 16 16:47:41.325616 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322975 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Apr 16 16:47:41.325616 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322977 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Apr 16 16:47:41.325616 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322979 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Apr 16 16:47:41.325616 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322982 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Apr 16 16:47:41.325616 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322984 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Apr 16 16:47:41.325616 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322987 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Apr 16 16:47:41.325616 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322989 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Apr 16 16:47:41.325616 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322992 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS Apr 16 16:47:41.325616 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322994 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Apr 16 16:47:41.325976 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322997 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Apr 16 16:47:41.325976 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.322999 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Apr 16 16:47:41.325976 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323001 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Apr 16 16:47:41.325976 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323005 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Apr 16 16:47:41.325976 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323008 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Apr 16 16:47:41.325976 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323011 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Apr 16 16:47:41.325976 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323014 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Apr 16 16:47:41.325976 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323017 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Apr 16 16:47:41.325976 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323019 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Apr 16 16:47:41.325976 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323022 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Apr 16 16:47:41.325976 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323025 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Apr 16 16:47:41.325976 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323028 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Apr 16 16:47:41.325976 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323031 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Apr 16 16:47:41.325976 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323033 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Apr 16 16:47:41.325976 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323036 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Apr 16 16:47:41.325976 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323039 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Apr 16 16:47:41.325976 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323042 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Apr 16 16:47:41.325976 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323044 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Apr 16 16:47:41.325976 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323047 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Apr 16 16:47:41.325976 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323049 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Apr 16 16:47:41.326505 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323051 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Apr 16 16:47:41.326505 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323054 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Apr 16 16:47:41.326505 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323056 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Apr 16 16:47:41.326505 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323059 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Apr 16 16:47:41.326505 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323061 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Apr 16 16:47:41.326505 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323064 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Apr 16 16:47:41.326505 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323066 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Apr 16 16:47:41.326505 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323069 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Apr 16 16:47:41.326505 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323071 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Apr 16 16:47:41.326505 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323074 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Apr 16 16:47:41.326505 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323076 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Apr 16 16:47:41.326505 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323079 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Apr 16 16:47:41.326505 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323081 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Apr 16 16:47:41.326505 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323083 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Apr 16 16:47:41.326505 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323086 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Apr 16 16:47:41.326505 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323088 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Apr 16 16:47:41.326505 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323090 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Apr 16 16:47:41.326505 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323093 2576 feature_gate.go:328] unrecognized feature gate: Example2 Apr 16 16:47:41.326505 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323096 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Apr 16 16:47:41.326958 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323098 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Apr 16 16:47:41.326958 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323100 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Apr 16 16:47:41.326958 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323103 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Apr 16 16:47:41.326958 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323105 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Apr 16 16:47:41.326958 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323108 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Apr 16 16:47:41.326958 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323110 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Apr 16 16:47:41.326958 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323112 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Apr 16 16:47:41.326958 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323115 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Apr 16 16:47:41.326958 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323118 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Apr 16 16:47:41.326958 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323121 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Apr 16 16:47:41.326958 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323125 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Apr 16 16:47:41.326958 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323127 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Apr 16 16:47:41.326958 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323130 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Apr 16 16:47:41.326958 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323132 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Apr 16 16:47:41.326958 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323135 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Apr 16 16:47:41.326958 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323137 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Apr 16 16:47:41.326958 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323139 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Apr 16 16:47:41.326958 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323142 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Apr 16 16:47:41.326958 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323144 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Apr 16 16:47:41.326958 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323147 2576 feature_gate.go:328] unrecognized feature gate: Example Apr 16 16:47:41.327450 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323150 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Apr 16 16:47:41.327450 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323152 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Apr 16 16:47:41.327450 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323155 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI Apr 16 16:47:41.327450 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323157 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Apr 16 16:47:41.327450 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323160 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Apr 16 16:47:41.327450 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323162 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Apr 16 16:47:41.327450 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323164 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Apr 16 16:47:41.327450 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323167 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Apr 16 16:47:41.327450 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323169 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Apr 16 16:47:41.327450 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323171 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Apr 16 16:47:41.327450 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323174 2576 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController Apr 16 16:47:41.327450 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323176 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Apr 16 16:47:41.327450 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:41.323179 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations Apr 16 16:47:41.327450 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.323183 2576 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} Apr 16 16:47:41.327450 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.324017 2576 server.go:962] "Client rotation is on, will bootstrap in background" Apr 16 16:47:41.327806 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.326391 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Apr 16 16:47:41.327806 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.327501 2576 server.go:1019] "Starting client certificate rotation" Apr 16 16:47:41.327806 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.327598 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Apr 16 16:47:41.327806 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.327638 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Apr 16 16:47:41.356256 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.356226 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Apr 16 16:47:41.361599 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.361567 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Apr 16 16:47:41.378259 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.378230 2576 log.go:25] "Validated CRI v1 runtime API" Apr 16 16:47:41.384378 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.384226 2576 log.go:25] "Validated CRI v1 image API" Apr 16 16:47:41.386085 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.386066 2576 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Apr 16 16:47:41.386650 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.386635 2576 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Apr 16 16:47:41.394091 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.394069 2576 fs.go:135] Filesystem UUIDs: map[3d596d6b-3dcc-4e15-8444-2447c95e2c26:/dev/nvme0n1p4 3d97eb2f-7e63-4cac-9834-82fadd5cb0a1:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Apr 16 16:47:41.394157 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.394092 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Apr 16 16:47:41.400373 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.400238 2576 manager.go:217] Machine: {Timestamp:2026-04-16 16:47:41.398411674 +0000 UTC m=+0.459561200 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099476 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20d1f3866a109a18e3f86ccc1af5a6 SystemUUID:ec20d1f3-866a-109a-18e3-f86ccc1af5a6 BootID:1e69f4aa-2ce8-4bc7-8ca6-daec909f09c4 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6098944 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:0f:df:36:47:51 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:0f:df:36:47:51 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7e:c9:20:6b:e1:45 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Apr 16 16:47:41.400373 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.400367 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Apr 16 16:47:41.400478 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.400452 2576 manager.go:233] Version: {KernelVersion:5.14.0-570.104.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260401-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Apr 16 16:47:41.401657 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.401632 2576 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Apr 16 16:47:41.401800 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.401660 2576 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-37.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Apr 16 16:47:41.401841 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.401806 2576 topology_manager.go:138] "Creating topology manager with none policy" Apr 16 16:47:41.401841 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.401815 2576 container_manager_linux.go:306] "Creating device plugin manager" Apr 16 16:47:41.401841 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.401827 2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Apr 16 16:47:41.403862 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.403851 2576 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Apr 16 16:47:41.405687 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.405676 2576 state_mem.go:36] "Initialized new in-memory state store" Apr 16 16:47:41.405828 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.405818 2576 server.go:1267] "Using root directory" path="/var/lib/kubelet" Apr 16 16:47:41.408540 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.408530 2576 kubelet.go:491] "Attempting to sync node with API server" Apr 16 16:47:41.408577 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.408550 2576 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests" Apr 16 16:47:41.408577 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.408565 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Apr 16 16:47:41.408577 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.408575 2576 kubelet.go:397] "Adding apiserver pod source" Apr 16 16:47:41.408694 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.408588 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Apr 16 16:47:41.409848 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.409830 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Apr 16 16:47:41.409944 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.409857 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Apr 16 16:47:41.413220 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.413203 2576 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.10-2.rhaos4.20.gita4d0894.el9" apiVersion="v1" Apr 16 16:47:41.419684 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.418831 2576 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Apr 16 16:47:41.420574 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.420561 2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Apr 16 16:47:41.421417 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.421400 2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Apr 16 16:47:41.421464 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.421420 2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Apr 16 16:47:41.421464 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.421426 2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Apr 16 16:47:41.421464 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.421438 2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Apr 16 16:47:41.421464 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.421445 2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret" Apr 16 16:47:41.421464 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.421451 2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Apr 16 16:47:41.421464 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.421456 2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Apr 16 16:47:41.421464 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.421465 2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc" Apr 16 16:47:41.421648 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.421471 2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Apr 16 16:47:41.421648 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.421481 2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected" Apr 16 16:47:41.421648 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.421489 2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Apr 16 16:47:41.421648 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.421514 2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi" Apr 16 16:47:41.421648 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.421587 2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image" Apr 16 16:47:41.421953 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:41.421935 2576 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-37.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Apr 16 16:47:41.421995 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:41.421959 2576 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Apr 16 16:47:41.425585 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.425570 2576 watchdog_linux.go:99] "Systemd watchdog is not enabled" Apr 16 16:47:41.425642 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.425617 2576 server.go:1295] "Started kubelet" Apr 16 16:47:41.425748 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.425709 2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Apr 16 16:47:41.425823 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.425770 2576 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Apr 16 16:47:41.425869 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.425855 2576 server_v1.go:47] "podresources" method="list" useActivePods=true Apr 16 16:47:41.426431 ip-10-0-142-37 systemd[1]: Started Kubernetes Kubelet. Apr 16 16:47:41.427016 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.426675 2576 server.go:317] "Adding debug handlers to kubelet server" Apr 16 16:47:41.427016 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.426758 2576 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-37.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Apr 16 16:47:41.427892 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.427874 2576 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Apr 16 16:47:41.433905 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:41.432883 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-37.ec2.internal.18a6e43ecd2c64d9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-37.ec2.internal,UID:ip-10-0-142-37.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-37.ec2.internal,},FirstTimestamp:2026-04-16 16:47:41.425583321 +0000 UTC m=+0.486732842,LastTimestamp:2026-04-16 16:47:41.425583321 +0000 UTC m=+0.486732842,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-37.ec2.internal,}" Apr 16 16:47:41.434965 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.434948 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sfbkh" Apr 16 16:47:41.435871 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.435849 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Apr 16 16:47:41.435952 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:41.435924 2576 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Apr 16 16:47:41.436493 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.436474 2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Apr 16 16:47:41.437053 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.437039 2576 factory.go:55] Registering systemd factory Apr 16 16:47:41.437104 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.437057 2576 factory.go:223] Registration of the systemd container factory successfully Apr 16 16:47:41.437268 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.437228 2576 desired_state_of_world_populator.go:150] "Desired state populator starts to run" Apr 16 16:47:41.437316 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.437235 2576 volume_manager.go:295] "The desired_state_of_world populator starts" Apr 16 16:47:41.437316 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.437289 2576 volume_manager.go:297] "Starting Kubelet Volume Manager" Apr 16 16:47:41.437399 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.437340 2576 factory.go:153] Registering CRI-O factory Apr 16 16:47:41.437399 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.437398 2576 factory.go:223] Registration of the crio container factory successfully Apr 16 16:47:41.437478 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.437418 2576 reconstruct.go:97] "Volume reconstruction finished" Apr 16 16:47:41.437478 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.437428 2576 reconciler.go:26] "Reconciler: start to sync state" Apr 16 16:47:41.437478 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:41.437458 2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-142-37.ec2.internal\" not found" Apr 16 16:47:41.437588 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.437576 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Apr 16 16:47:41.437619 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.437599 2576 factory.go:103] Registering Raw factory Apr 16 16:47:41.437619 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.437613 2576 manager.go:1196] Started watching for new ooms in manager Apr 16 16:47:41.438017 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.438002 2576 manager.go:319] Starting recovery of all containers Apr 16 16:47:41.440199 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:41.440171 2576 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Apr 16 16:47:41.440199 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:41.440198 2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-37.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Apr 16 16:47:41.442434 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.442415 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sfbkh" Apr 16 16:47:41.448639 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.448620 2576 manager.go:324] Recovery completed Apr 16 16:47:41.451337 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:41.451316 2576 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Apr 16 16:47:41.454376 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.454361 2576 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" Apr 16 16:47:41.456857 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.456842 2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-37.ec2.internal" event="NodeHasSufficientMemory" Apr 16 16:47:41.456938 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.456872 2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-37.ec2.internal" event="NodeHasNoDiskPressure" Apr 16 16:47:41.456938 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.456883 2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-37.ec2.internal" event="NodeHasSufficientPID" Apr 16 16:47:41.457413 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.457400 2576 cpu_manager.go:222] "Starting CPU manager" policy="none" Apr 16 16:47:41.457413 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.457412 2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Apr 16 16:47:41.457490 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.457428 2576 state_mem.go:36] "Initialized new in-memory state store" Apr 16 16:47:41.461072 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.461061 2576 policy_none.go:49] "None policy: Start" Apr 16 16:47:41.461114 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.461077 2576 memory_manager.go:186] "Starting memorymanager" policy="None" Apr 16 16:47:41.461114 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.461086 2576 state_mem.go:35] "Initializing new in-memory state store" Apr 16 16:47:41.501842 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.501827 2576 manager.go:341] "Starting Device Plugin manager" Apr 16 16:47:41.522957 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:41.501860 2576 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Apr 16 16:47:41.522957 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.501874 2576 server.go:85] "Starting device plugin registration server" Apr 16 16:47:41.522957 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.502140 2576 eviction_manager.go:189] "Eviction manager: starting control loop" Apr 16 16:47:41.522957 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.502151 2576 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Apr 16 16:47:41.522957 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.502275 2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Apr 16 16:47:41.522957 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.502355 2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Apr 16 16:47:41.522957 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.502364 2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Apr 16 16:47:41.522957 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:41.502869 2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Apr 16 16:47:41.522957 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:41.502902 2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-37.ec2.internal\" not found" Apr 16 16:47:41.586590 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.586504 2576 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4" Apr 16 16:47:41.587830 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.587816 2576 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6" Apr 16 16:47:41.587897 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.587842 2576 status_manager.go:230] "Starting to sync pod status with apiserver" Apr 16 16:47:41.587897 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.587860 2576 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Apr 16 16:47:41.587897 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.587868 2576 kubelet.go:2451] "Starting kubelet main sync loop" Apr 16 16:47:41.588005 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:41.587900 2576 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Apr 16 16:47:41.595319 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.595302 2576 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Apr 16 16:47:41.602449 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.602436 2576 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" Apr 16 16:47:41.603261 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.603228 2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-37.ec2.internal" event="NodeHasSufficientMemory" Apr 16 16:47:41.603310 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.603277 2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-37.ec2.internal" event="NodeHasNoDiskPressure" Apr 16 16:47:41.603310 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.603288 2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-37.ec2.internal" event="NodeHasSufficientPID" Apr 16 16:47:41.603372 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.603311 2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-37.ec2.internal" Apr 16 16:47:41.612867 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.612853 2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-37.ec2.internal" Apr 16 16:47:41.612911 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:41.612876 2576 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-37.ec2.internal\": node \"ip-10-0-142-37.ec2.internal\" not found" Apr 16 16:47:41.628257 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:41.628211 2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-142-37.ec2.internal\" not found" Apr 16 16:47:41.688815 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.688771 2576 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-37.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-37.ec2.internal"] Apr 16 16:47:41.688962 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.688879 2576 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" Apr 16 16:47:41.689876 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.689861 2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-37.ec2.internal" event="NodeHasSufficientMemory" Apr 16 16:47:41.689965 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.689889 2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-37.ec2.internal" event="NodeHasNoDiskPressure" Apr 16 16:47:41.689965 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.689900 2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-37.ec2.internal" event="NodeHasSufficientPID" Apr 16 16:47:41.692226 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.692214 2576 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" Apr 16 16:47:41.692357 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.692343 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-37.ec2.internal" Apr 16 16:47:41.692398 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.692374 2576 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" Apr 16 16:47:41.692887 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.692864 2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-37.ec2.internal" event="NodeHasSufficientMemory" Apr 16 16:47:41.692887 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.692878 2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-37.ec2.internal" event="NodeHasSufficientMemory" Apr 16 16:47:41.693015 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.692892 2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-37.ec2.internal" event="NodeHasNoDiskPressure" Apr 16 16:47:41.693015 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.692899 2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-37.ec2.internal" event="NodeHasNoDiskPressure" Apr 16 16:47:41.693015 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.692904 2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-37.ec2.internal" event="NodeHasSufficientPID" Apr 16 16:47:41.693015 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.692911 2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-37.ec2.internal" event="NodeHasSufficientPID" Apr 16 16:47:41.695480 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.695465 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-37.ec2.internal" Apr 16 16:47:41.695546 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.695489 2576 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" Apr 16 16:47:41.696109 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.696093 2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-37.ec2.internal" event="NodeHasSufficientMemory" Apr 16 16:47:41.696177 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.696124 2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-37.ec2.internal" event="NodeHasNoDiskPressure" Apr 16 16:47:41.696177 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.696138 2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-37.ec2.internal" event="NodeHasSufficientPID" Apr 16 16:47:41.719199 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:41.719175 2576 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-37.ec2.internal\" not found" node="ip-10-0-142-37.ec2.internal" Apr 16 16:47:41.723825 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:41.723806 2576 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-37.ec2.internal\" not found" node="ip-10-0-142-37.ec2.internal" Apr 16 16:47:41.729045 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:41.729020 2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-142-37.ec2.internal\" not found" Apr 16 16:47:41.739366 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.739341 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8b1f1c157df36eece23646e8d91601bf-config\") pod \"kube-apiserver-proxy-ip-10-0-142-37.ec2.internal\" (UID: \"8b1f1c157df36eece23646e8d91601bf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-37.ec2.internal" Apr 16 16:47:41.739461 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.739373 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0f4d89b70f22123800295882f9c30018-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-37.ec2.internal\" (UID: \"0f4d89b70f22123800295882f9c30018\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-37.ec2.internal" Apr 16 16:47:41.739461 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.739394 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f4d89b70f22123800295882f9c30018-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-37.ec2.internal\" (UID: \"0f4d89b70f22123800295882f9c30018\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-37.ec2.internal" Apr 16 16:47:41.829139 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:41.829111 2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-142-37.ec2.internal\" not found" Apr 16 16:47:41.840547 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.840492 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0f4d89b70f22123800295882f9c30018-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-37.ec2.internal\" (UID: \"0f4d89b70f22123800295882f9c30018\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-37.ec2.internal" Apr 16 16:47:41.840547 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.840525 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f4d89b70f22123800295882f9c30018-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-37.ec2.internal\" (UID: \"0f4d89b70f22123800295882f9c30018\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-37.ec2.internal" Apr 16 16:47:41.840641 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.840546 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8b1f1c157df36eece23646e8d91601bf-config\") pod \"kube-apiserver-proxy-ip-10-0-142-37.ec2.internal\" (UID: \"8b1f1c157df36eece23646e8d91601bf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-37.ec2.internal" Apr 16 16:47:41.840641 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.840588 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8b1f1c157df36eece23646e8d91601bf-config\") pod \"kube-apiserver-proxy-ip-10-0-142-37.ec2.internal\" (UID: \"8b1f1c157df36eece23646e8d91601bf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-37.ec2.internal" Apr 16 16:47:41.840641 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.840588 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0f4d89b70f22123800295882f9c30018-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-37.ec2.internal\" (UID: \"0f4d89b70f22123800295882f9c30018\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-37.ec2.internal" Apr 16 16:47:41.840641 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:41.840590 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f4d89b70f22123800295882f9c30018-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-37.ec2.internal\" (UID: \"0f4d89b70f22123800295882f9c30018\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-37.ec2.internal" Apr 16 16:47:41.929941 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:41.929897 2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-142-37.ec2.internal\" not found" Apr 16 16:47:42.022523 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.022493 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-37.ec2.internal" Apr 16 16:47:42.026004 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.025988 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-37.ec2.internal" Apr 16 16:47:42.030594 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:42.030572 2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-142-37.ec2.internal\" not found" Apr 16 16:47:42.131163 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:42.131079 2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-142-37.ec2.internal\" not found" Apr 16 16:47:42.231671 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:42.231640 2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-142-37.ec2.internal\" not found" Apr 16 16:47:42.327234 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.327206 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Apr 16 16:47:42.327861 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.327370 2576 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Apr 16 16:47:42.332368 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:42.332348 2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-142-37.ec2.internal\" not found" Apr 16 16:47:42.346388 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.346365 2576 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Apr 16 16:47:42.348780 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.348765 2576 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Apr 16 16:47:42.409764 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.409700 2576 apiserver.go:52] "Watching apiserver" Apr 16 16:47:42.417842 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.417820 2576 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Apr 16 16:47:42.418160 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.418140 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-kwz5r","openshift-image-registry/node-ca-q6fvl","openshift-multus/multus-ndl5w","openshift-multus/network-metrics-daemon-hk5gr","openshift-network-operator/iptables-alerter-9vmmq","openshift-ovn-kubernetes/ovnkube-node-p7c8c","kube-system/global-pull-secret-syncer-pgvd2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9c8bl","openshift-cluster-node-tuning-operator/tuned-vh7lw","openshift-dns/node-resolver-42cdh","openshift-multus/multus-additional-cni-plugins-8zhpq","openshift-network-diagnostics/network-check-target-5kz89"] Apr 16 16:47:42.422794 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.422777 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kwz5r" Apr 16 16:47:42.424828 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.424807 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9cs84\"" Apr 16 16:47:42.424828 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.424823 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Apr 16 16:47:42.424962 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.424834 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Apr 16 16:47:42.425237 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.425211 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q6fvl" Apr 16 16:47:42.426746 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.426728 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Apr 16 16:47:42.426888 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.426875 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Apr 16 16:47:42.426957 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.426896 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Apr 16 16:47:42.427370 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.427354 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hnwcx\"" Apr 16 16:47:42.427460 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.427444 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.429206 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.429173 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Apr 16 16:47:42.429392 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.429378 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qg6jz\"" Apr 16 16:47:42.429452 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.429381 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Apr 16 16:47:42.429452 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.429424 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Apr 16 16:47:42.429452 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.429426 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Apr 16 16:47:42.429992 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.429970 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk5gr" Apr 16 16:47:42.430066 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.430054 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9vmmq" Apr 16 16:47:42.430104 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:42.430078 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk5gr" podUID="412718e2-2075-4c7b-a6a9-3b94ed57564c" Apr 16 16:47:42.431731 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.431715 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wlmjm\"" Apr 16 16:47:42.432003 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.431986 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Apr 16 16:47:42.432071 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.432015 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Apr 16 16:47:42.432071 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.432059 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Apr 16 16:47:42.432697 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.432676 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.434707 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.434689 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Apr 16 16:47:42.434832 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.434815 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Apr 16 16:47:42.435038 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.434993 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pgvd2" Apr 16 16:47:42.435131 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:42.435063 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pgvd2" podUID="98d7d9af-1c21-4f39-b605-21a11d06f2ae" Apr 16 16:47:42.435218 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.435201 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Apr 16 16:47:42.435218 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.435214 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Apr 16 16:47:42.435344 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.435217 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Apr 16 16:47:42.435344 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.435269 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Apr 16 16:47:42.435344 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.435288 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-p5pmh\"" Apr 16 16:47:42.435980 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.435966 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Apr 16 16:47:42.436708 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.436691 2576 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-37.ec2.internal" Apr 16 16:47:42.437735 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.437708 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9c8bl" Apr 16 16:47:42.439678 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.439347 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Apr 16 16:47:42.439678 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.439397 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Apr 16 16:47:42.439793 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.439708 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Apr 16 16:47:42.439793 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.439761 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-sf5tz\"" Apr 16 16:47:42.440197 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.440181 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.443195 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.442430 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Apr 16 16:47:42.443195 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.442578 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2c9dm\"" Apr 16 16:47:42.443195 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.442838 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Apr 16 16:47:42.443502 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.443485 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2fef7f9f-e604-47d1-877a-3dfcbbdebba0-iptables-alerter-script\") pod \"iptables-alerter-9vmmq\" (UID: \"2fef7f9f-e604-47d1-877a-3dfcbbdebba0\") " pod="openshift-network-operator/iptables-alerter-9vmmq" Apr 16 16:47:42.443573 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.443513 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c2fec5cf-333c-4342-bd77-642bb2581568-ovn-node-metrics-cert\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.443573 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.443531 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/145fe4b0-c23a-49dc-abe3-212e53cff730-etc-kubernetes\") pod \"tuned-vh7lw\" (UID: \"145fe4b0-c23a-49dc-abe3-212e53cff730\") " pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.443573 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.443545 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/145fe4b0-c23a-49dc-abe3-212e53cff730-sys\") pod \"tuned-vh7lw\" (UID: \"145fe4b0-c23a-49dc-abe3-212e53cff730\") " pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.443573 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.443561 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/145fe4b0-c23a-49dc-abe3-212e53cff730-etc-tuned\") pod \"tuned-vh7lw\" (UID: \"145fe4b0-c23a-49dc-abe3-212e53cff730\") " pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.443769 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.443614 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/643eef8b-e0e5-48fb-a732-6d880b6b62e7-socket-dir\") pod \"aws-ebs-csi-driver-node-9c8bl\" (UID: \"643eef8b-e0e5-48fb-a732-6d880b6b62e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9c8bl" Apr 16 16:47:42.443769 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.443661 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-host-run-netns\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.443769 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.443699 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8e078538-0666-4e1e-8b80-2f09583fbd00-host-run-multus-certs\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.443769 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.443720 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/145fe4b0-c23a-49dc-abe3-212e53cff730-run\") pod \"tuned-vh7lw\" (UID: \"145fe4b0-c23a-49dc-abe3-212e53cff730\") " pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.443769 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.443738 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f6b12331-2022-4c05-ac42-753274b499dc-agent-certs\") pod \"konnectivity-agent-kwz5r\" (UID: \"f6b12331-2022-4c05-ac42-753274b499dc\") " pod="kube-system/konnectivity-agent-kwz5r" Apr 16 16:47:42.444033 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.443788 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrvvc\" (UniqueName: \"kubernetes.io/projected/412718e2-2075-4c7b-a6a9-3b94ed57564c-kube-api-access-rrvvc\") pod \"network-metrics-daemon-hk5gr\" (UID: \"412718e2-2075-4c7b-a6a9-3b94ed57564c\") " pod="openshift-multus/network-metrics-daemon-hk5gr" Apr 16 16:47:42.444033 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.443828 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/145fe4b0-c23a-49dc-abe3-212e53cff730-etc-modprobe-d\") pod \"tuned-vh7lw\" (UID: \"145fe4b0-c23a-49dc-abe3-212e53cff730\") " pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.444033 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.443870 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e078538-0666-4e1e-8b80-2f09583fbd00-cnibin\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.444033 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.443894 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-systemd-units\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.444033 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.443946 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-run-openvswitch\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.444033 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.443972 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c2fec5cf-333c-4342-bd77-642bb2581568-env-overrides\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.444033 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.443994 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fce43675-3aee-453f-b5ad-a28fba84c810-serviceca\") pod \"node-ca-q6fvl\" (UID: \"fce43675-3aee-453f-b5ad-a28fba84c810\") " pod="openshift-image-registry/node-ca-q6fvl" Apr 16 16:47:42.444033 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444020 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e078538-0666-4e1e-8b80-2f09583fbd00-host-run-netns\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.444351 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444054 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e078538-0666-4e1e-8b80-2f09583fbd00-host-var-lib-cni-bin\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.444351 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444080 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.444351 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444123 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/98d7d9af-1c21-4f39-b605-21a11d06f2ae-dbus\") pod \"global-pull-secret-syncer-pgvd2\" (UID: \"98d7d9af-1c21-4f39-b605-21a11d06f2ae\") " pod="kube-system/global-pull-secret-syncer-pgvd2" Apr 16 16:47:42.444351 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444157 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rjsk\" (UniqueName: \"kubernetes.io/projected/fce43675-3aee-453f-b5ad-a28fba84c810-kube-api-access-4rjsk\") pod \"node-ca-q6fvl\" (UID: \"fce43675-3aee-453f-b5ad-a28fba84c810\") " pod="openshift-image-registry/node-ca-q6fvl" Apr 16 16:47:42.444351 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444177 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e078538-0666-4e1e-8b80-2f09583fbd00-host-var-lib-kubelet\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.444351 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444192 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-569qg\" (UniqueName: \"kubernetes.io/projected/2fef7f9f-e604-47d1-877a-3dfcbbdebba0-kube-api-access-569qg\") pod \"iptables-alerter-9vmmq\" (UID: \"2fef7f9f-e604-47d1-877a-3dfcbbdebba0\") " pod="openshift-network-operator/iptables-alerter-9vmmq" Apr 16 16:47:42.444351 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444207 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-node-log\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.444351 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444222 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-log-socket\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.444351 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444269 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-04-15 16:42:41 +0000 UTC" deadline="2027-09-24 21:17:25.18118432 +0000 UTC" Apr 16 16:47:42.444351 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444294 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12628h29m42.736892673s" Apr 16 16:47:42.444351 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444303 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-host-cni-netd\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.444351 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444335 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/145fe4b0-c23a-49dc-abe3-212e53cff730-lib-modules\") pod \"tuned-vh7lw\" (UID: \"145fe4b0-c23a-49dc-abe3-212e53cff730\") " pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.444351 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444350 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f6b12331-2022-4c05-ac42-753274b499dc-konnectivity-ca\") pod \"konnectivity-agent-kwz5r\" (UID: \"f6b12331-2022-4c05-ac42-753274b499dc\") " pod="kube-system/konnectivity-agent-kwz5r" Apr 16 16:47:42.444839 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444369 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/98d7d9af-1c21-4f39-b605-21a11d06f2ae-kubelet-config\") pod \"global-pull-secret-syncer-pgvd2\" (UID: \"98d7d9af-1c21-4f39-b605-21a11d06f2ae\") " pod="kube-system/global-pull-secret-syncer-pgvd2" Apr 16 16:47:42.444839 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444390 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8e078538-0666-4e1e-8b80-2f09583fbd00-os-release\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.444839 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444405 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8e078538-0666-4e1e-8b80-2f09583fbd00-host-run-k8s-cni-cncf-io\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.444839 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444437 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-var-lib-openvswitch\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.444839 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444460 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e078538-0666-4e1e-8b80-2f09583fbd00-cni-binary-copy\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.444839 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444475 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2fef7f9f-e604-47d1-877a-3dfcbbdebba0-host-slash\") pod \"iptables-alerter-9vmmq\" (UID: \"2fef7f9f-e604-47d1-877a-3dfcbbdebba0\") " pod="openshift-network-operator/iptables-alerter-9vmmq" Apr 16 16:47:42.444839 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444518 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-host-cni-bin\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.444839 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444541 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/412718e2-2075-4c7b-a6a9-3b94ed57564c-metrics-certs\") pod \"network-metrics-daemon-hk5gr\" (UID: \"412718e2-2075-4c7b-a6a9-3b94ed57564c\") " pod="openshift-multus/network-metrics-daemon-hk5gr" Apr 16 16:47:42.444839 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444566 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/145fe4b0-c23a-49dc-abe3-212e53cff730-etc-sysctl-d\") pod \"tuned-vh7lw\" (UID: \"145fe4b0-c23a-49dc-abe3-212e53cff730\") " pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.444839 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444585 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/145fe4b0-c23a-49dc-abe3-212e53cff730-host\") pod \"tuned-vh7lw\" (UID: \"145fe4b0-c23a-49dc-abe3-212e53cff730\") " pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.444839 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444602 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8e078538-0666-4e1e-8b80-2f09583fbd00-host-var-lib-cni-multus\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.444839 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444623 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e078538-0666-4e1e-8b80-2f09583fbd00-etc-kubernetes\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.444839 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444638 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8e078538-0666-4e1e-8b80-2f09583fbd00-multus-daemon-config\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.444839 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444657 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-host-slash\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.444839 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444673 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-run-systemd\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.444839 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444688 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c2fec5cf-333c-4342-bd77-642bb2581568-ovnkube-config\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.444839 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444706 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8e078538-0666-4e1e-8b80-2f09583fbd00-multus-socket-dir-parent\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.445385 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444721 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/145fe4b0-c23a-49dc-abe3-212e53cff730-var-lib-kubelet\") pod \"tuned-vh7lw\" (UID: \"145fe4b0-c23a-49dc-abe3-212e53cff730\") " pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.445385 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444746 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/643eef8b-e0e5-48fb-a732-6d880b6b62e7-registration-dir\") pod \"aws-ebs-csi-driver-node-9c8bl\" (UID: \"643eef8b-e0e5-48fb-a732-6d880b6b62e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9c8bl" Apr 16 16:47:42.445385 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444773 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fce43675-3aee-453f-b5ad-a28fba84c810-host\") pod \"node-ca-q6fvl\" (UID: \"fce43675-3aee-453f-b5ad-a28fba84c810\") " pod="openshift-image-registry/node-ca-q6fvl" Apr 16 16:47:42.445385 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444790 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-etc-openvswitch\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.445385 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444803 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-run-ovn\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.445385 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444822 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/643eef8b-e0e5-48fb-a732-6d880b6b62e7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9c8bl\" (UID: \"643eef8b-e0e5-48fb-a732-6d880b6b62e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9c8bl" Apr 16 16:47:42.445385 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444857 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/643eef8b-e0e5-48fb-a732-6d880b6b62e7-device-dir\") pod \"aws-ebs-csi-driver-node-9c8bl\" (UID: \"643eef8b-e0e5-48fb-a732-6d880b6b62e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9c8bl" Apr 16 16:47:42.445385 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444881 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxrps\" (UniqueName: \"kubernetes.io/projected/643eef8b-e0e5-48fb-a732-6d880b6b62e7-kube-api-access-qxrps\") pod \"aws-ebs-csi-driver-node-9c8bl\" (UID: \"643eef8b-e0e5-48fb-a732-6d880b6b62e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9c8bl" Apr 16 16:47:42.445385 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444906 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/145fe4b0-c23a-49dc-abe3-212e53cff730-etc-sysctl-conf\") pod \"tuned-vh7lw\" (UID: \"145fe4b0-c23a-49dc-abe3-212e53cff730\") " pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.445385 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444934 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwmjt\" (UniqueName: \"kubernetes.io/projected/145fe4b0-c23a-49dc-abe3-212e53cff730-kube-api-access-dwmjt\") pod \"tuned-vh7lw\" (UID: \"145fe4b0-c23a-49dc-abe3-212e53cff730\") " pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.445385 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444948 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e078538-0666-4e1e-8b80-2f09583fbd00-multus-cni-dir\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.445385 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444962 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8e078538-0666-4e1e-8b80-2f09583fbd00-hostroot\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.445385 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.444977 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xg5n\" (UniqueName: \"kubernetes.io/projected/8e078538-0666-4e1e-8b80-2f09583fbd00-kube-api-access-2xg5n\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.445385 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.445012 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/98d7d9af-1c21-4f39-b605-21a11d06f2ae-original-pull-secret\") pod \"global-pull-secret-syncer-pgvd2\" (UID: \"98d7d9af-1c21-4f39-b605-21a11d06f2ae\") " pod="kube-system/global-pull-secret-syncer-pgvd2" Apr 16 16:47:42.445385 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.445030 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/643eef8b-e0e5-48fb-a732-6d880b6b62e7-etc-selinux\") pod \"aws-ebs-csi-driver-node-9c8bl\" (UID: \"643eef8b-e0e5-48fb-a732-6d880b6b62e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9c8bl" Apr 16 16:47:42.445385 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.445043 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/643eef8b-e0e5-48fb-a732-6d880b6b62e7-sys-fs\") pod \"aws-ebs-csi-driver-node-9c8bl\" (UID: \"643eef8b-e0e5-48fb-a732-6d880b6b62e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9c8bl" Apr 16 16:47:42.445823 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.445066 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/145fe4b0-c23a-49dc-abe3-212e53cff730-etc-sysconfig\") pod \"tuned-vh7lw\" (UID: \"145fe4b0-c23a-49dc-abe3-212e53cff730\") " pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.445823 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.445108 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-host-kubelet\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.445823 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.445131 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c2fec5cf-333c-4342-bd77-642bb2581568-ovnkube-script-lib\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.445823 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.445146 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdgpm\" (UniqueName: \"kubernetes.io/projected/c2fec5cf-333c-4342-bd77-642bb2581568-kube-api-access-kdgpm\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.445823 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.445173 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e078538-0666-4e1e-8b80-2f09583fbd00-multus-conf-dir\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.445823 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.445190 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-host-run-ovn-kubernetes\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.445823 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.445204 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/145fe4b0-c23a-49dc-abe3-212e53cff730-etc-systemd\") pod \"tuned-vh7lw\" (UID: \"145fe4b0-c23a-49dc-abe3-212e53cff730\") " pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.445823 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.445219 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/145fe4b0-c23a-49dc-abe3-212e53cff730-tmp\") pod \"tuned-vh7lw\" (UID: \"145fe4b0-c23a-49dc-abe3-212e53cff730\") " pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.445823 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.445264 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e078538-0666-4e1e-8b80-2f09583fbd00-system-cni-dir\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.447256 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.447222 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-42cdh" Apr 16 16:47:42.447349 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.447328 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8zhpq" Apr 16 16:47:42.447737 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.447717 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Apr 16 16:47:42.447820 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.447769 2576 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-37.ec2.internal" Apr 16 16:47:42.449187 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.449170 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Apr 16 16:47:42.449311 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.449174 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Apr 16 16:47:42.449433 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.449315 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vbwx8\"" Apr 16 16:47:42.449433 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.449392 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vkzzw\"" Apr 16 16:47:42.449433 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.449391 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Apr 16 16:47:42.449549 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.449523 2576 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Apr 16 16:47:42.449582 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.449553 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-37.ec2.internal"] Apr 16 16:47:42.449628 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.449618 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kz89" Apr 16 16:47:42.449680 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:42.449667 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5kz89" podUID="e730d89c-85b3-4a44-883b-d65f347261ec" Apr 16 16:47:42.450011 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.449998 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Apr 16 16:47:42.458615 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.458597 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Apr 16 16:47:42.458740 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.458726 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-37.ec2.internal"] Apr 16 16:47:42.479458 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.479391 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wxx6x" Apr 16 16:47:42.487919 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.487901 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wxx6x" Apr 16 16:47:42.539101 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.539043 2576 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world" Apr 16 16:47:42.545864 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.545839 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/98d7d9af-1c21-4f39-b605-21a11d06f2ae-dbus\") pod \"global-pull-secret-syncer-pgvd2\" (UID: \"98d7d9af-1c21-4f39-b605-21a11d06f2ae\") " pod="kube-system/global-pull-secret-syncer-pgvd2" Apr 16 16:47:42.545963 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.545877 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4rjsk\" (UniqueName: \"kubernetes.io/projected/fce43675-3aee-453f-b5ad-a28fba84c810-kube-api-access-4rjsk\") pod \"node-ca-q6fvl\" (UID: \"fce43675-3aee-453f-b5ad-a28fba84c810\") " pod="openshift-image-registry/node-ca-q6fvl" Apr 16 16:47:42.545963 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.545905 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e078538-0666-4e1e-8b80-2f09583fbd00-host-var-lib-kubelet\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.545963 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.545931 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-569qg\" (UniqueName: \"kubernetes.io/projected/2fef7f9f-e604-47d1-877a-3dfcbbdebba0-kube-api-access-569qg\") pod \"iptables-alerter-9vmmq\" (UID: \"2fef7f9f-e604-47d1-877a-3dfcbbdebba0\") " pod="openshift-network-operator/iptables-alerter-9vmmq" Apr 16 16:47:42.545963 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.545957 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-node-log\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.546152 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.545980 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-log-socket\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.546152 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546017 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-host-cni-netd\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.546152 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546035 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/98d7d9af-1c21-4f39-b605-21a11d06f2ae-dbus\") pod \"global-pull-secret-syncer-pgvd2\" (UID: \"98d7d9af-1c21-4f39-b605-21a11d06f2ae\") " pod="kube-system/global-pull-secret-syncer-pgvd2" Apr 16 16:47:42.546152 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546056 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e078538-0666-4e1e-8b80-2f09583fbd00-host-var-lib-kubelet\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.546152 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546060 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-node-log\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.546152 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546041 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/145fe4b0-c23a-49dc-abe3-212e53cff730-lib-modules\") pod \"tuned-vh7lw\" (UID: \"145fe4b0-c23a-49dc-abe3-212e53cff730\") " pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.546152 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546098 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f6b12331-2022-4c05-ac42-753274b499dc-konnectivity-ca\") pod \"konnectivity-agent-kwz5r\" (UID: \"f6b12331-2022-4c05-ac42-753274b499dc\") " pod="kube-system/konnectivity-agent-kwz5r" Apr 16 16:47:42.546152 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546104 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-log-socket\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.546152 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546122 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/98d7d9af-1c21-4f39-b605-21a11d06f2ae-kubelet-config\") pod \"global-pull-secret-syncer-pgvd2\" (UID: \"98d7d9af-1c21-4f39-b605-21a11d06f2ae\") " pod="kube-system/global-pull-secret-syncer-pgvd2" Apr 16 16:47:42.546152 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546129 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-host-cni-netd\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.546152 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546135 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/145fe4b0-c23a-49dc-abe3-212e53cff730-lib-modules\") pod \"tuned-vh7lw\" (UID: \"145fe4b0-c23a-49dc-abe3-212e53cff730\") " pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.546152 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546148 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8e078538-0666-4e1e-8b80-2f09583fbd00-os-release\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.546653 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546174 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8e078538-0666-4e1e-8b80-2f09583fbd00-host-run-k8s-cni-cncf-io\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.546653 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546191 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/98d7d9af-1c21-4f39-b605-21a11d06f2ae-kubelet-config\") pod \"global-pull-secret-syncer-pgvd2\" (UID: \"98d7d9af-1c21-4f39-b605-21a11d06f2ae\") " pod="kube-system/global-pull-secret-syncer-pgvd2" Apr 16 16:47:42.546653 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546197 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-var-lib-openvswitch\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.546653 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546226 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/20d62408-3657-49b5-a83b-39331108c8d9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8zhpq\" (UID: \"20d62408-3657-49b5-a83b-39331108c8d9\") " pod="openshift-multus/multus-additional-cni-plugins-8zhpq" Apr 16 16:47:42.546653 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546233 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8e078538-0666-4e1e-8b80-2f09583fbd00-host-run-k8s-cni-cncf-io\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.546653 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546289 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-var-lib-openvswitch\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.546653 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546292 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/20d62408-3657-49b5-a83b-39331108c8d9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8zhpq\" (UID: \"20d62408-3657-49b5-a83b-39331108c8d9\") " pod="openshift-multus/multus-additional-cni-plugins-8zhpq" Apr 16 16:47:42.546653 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546309 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8e078538-0666-4e1e-8b80-2f09583fbd00-os-release\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.546653 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546324 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e078538-0666-4e1e-8b80-2f09583fbd00-cni-binary-copy\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.546653 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546345 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2fef7f9f-e604-47d1-877a-3dfcbbdebba0-host-slash\") pod \"iptables-alerter-9vmmq\" (UID: \"2fef7f9f-e604-47d1-877a-3dfcbbdebba0\") " pod="openshift-network-operator/iptables-alerter-9vmmq" Apr 16 16:47:42.546653 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546367 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-host-cni-bin\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.546653 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546392 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nklr8\" (UniqueName: \"kubernetes.io/projected/99da12c7-93e3-4848-afc5-36f2cbf5645d-kube-api-access-nklr8\") pod \"node-resolver-42cdh\" (UID: \"99da12c7-93e3-4848-afc5-36f2cbf5645d\") " pod="openshift-dns/node-resolver-42cdh" Apr 16 16:47:42.546653 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546398 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2fef7f9f-e604-47d1-877a-3dfcbbdebba0-host-slash\") pod \"iptables-alerter-9vmmq\" (UID: \"2fef7f9f-e604-47d1-877a-3dfcbbdebba0\") " pod="openshift-network-operator/iptables-alerter-9vmmq" Apr 16 16:47:42.546653 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546417 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/412718e2-2075-4c7b-a6a9-3b94ed57564c-metrics-certs\") pod \"network-metrics-daemon-hk5gr\" (UID: \"412718e2-2075-4c7b-a6a9-3b94ed57564c\") " pod="openshift-multus/network-metrics-daemon-hk5gr" Apr 16 16:47:42.546653 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546439 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-host-cni-bin\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.546653 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546442 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/145fe4b0-c23a-49dc-abe3-212e53cff730-etc-sysctl-d\") pod \"tuned-vh7lw\" (UID: \"145fe4b0-c23a-49dc-abe3-212e53cff730\") " pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.546653 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546466 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/145fe4b0-c23a-49dc-abe3-212e53cff730-host\") pod \"tuned-vh7lw\" (UID: \"145fe4b0-c23a-49dc-abe3-212e53cff730\") " pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.547370 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546488 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8e078538-0666-4e1e-8b80-2f09583fbd00-host-var-lib-cni-multus\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.547370 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:42.546511 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 16:47:42.547370 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546542 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e078538-0666-4e1e-8b80-2f09583fbd00-etc-kubernetes\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.547370 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546575 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/145fe4b0-c23a-49dc-abe3-212e53cff730-host\") pod \"tuned-vh7lw\" (UID: \"145fe4b0-c23a-49dc-abe3-212e53cff730\") " pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.547370 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546579 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8e078538-0666-4e1e-8b80-2f09583fbd00-host-var-lib-cni-multus\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.547370 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546513 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e078538-0666-4e1e-8b80-2f09583fbd00-etc-kubernetes\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.547370 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:42.546603 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/412718e2-2075-4c7b-a6a9-3b94ed57564c-metrics-certs podName:412718e2-2075-4c7b-a6a9-3b94ed57564c nodeName:}" failed. No retries permitted until 2026-04-16 16:47:43.046582146 +0000 UTC m=+2.107731677 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/412718e2-2075-4c7b-a6a9-3b94ed57564c-metrics-certs") pod "network-metrics-daemon-hk5gr" (UID: "412718e2-2075-4c7b-a6a9-3b94ed57564c") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 16:47:42.547370 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546640 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f6b12331-2022-4c05-ac42-753274b499dc-konnectivity-ca\") pod \"konnectivity-agent-kwz5r\" (UID: \"f6b12331-2022-4c05-ac42-753274b499dc\") " pod="kube-system/konnectivity-agent-kwz5r" Apr 16 16:47:42.547370 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546703 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/145fe4b0-c23a-49dc-abe3-212e53cff730-etc-sysctl-d\") pod \"tuned-vh7lw\" (UID: \"145fe4b0-c23a-49dc-abe3-212e53cff730\") " pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.547370 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546726 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/20d62408-3657-49b5-a83b-39331108c8d9-cnibin\") pod \"multus-additional-cni-plugins-8zhpq\" (UID: \"20d62408-3657-49b5-a83b-39331108c8d9\") " pod="openshift-multus/multus-additional-cni-plugins-8zhpq" Apr 16 16:47:42.547370 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546755 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8e078538-0666-4e1e-8b80-2f09583fbd00-multus-daemon-config\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.547370 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546780 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-host-slash\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.547370 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546805 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-run-systemd\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.547370 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546830 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c2fec5cf-333c-4342-bd77-642bb2581568-ovnkube-config\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.547370 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546853 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8e078538-0666-4e1e-8b80-2f09583fbd00-multus-socket-dir-parent\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.547370 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546850 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-host-slash\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.547370 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546863 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-run-systemd\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.548122 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546877 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/145fe4b0-c23a-49dc-abe3-212e53cff730-var-lib-kubelet\") pod \"tuned-vh7lw\" (UID: \"145fe4b0-c23a-49dc-abe3-212e53cff730\") " pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.548122 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546895 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e078538-0666-4e1e-8b80-2f09583fbd00-cni-binary-copy\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.548122 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546914 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/643eef8b-e0e5-48fb-a732-6d880b6b62e7-registration-dir\") pod \"aws-ebs-csi-driver-node-9c8bl\" (UID: \"643eef8b-e0e5-48fb-a732-6d880b6b62e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9c8bl" Apr 16 16:47:42.548122 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546935 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8e078538-0666-4e1e-8b80-2f09583fbd00-multus-socket-dir-parent\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.548122 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546919 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/145fe4b0-c23a-49dc-abe3-212e53cff730-var-lib-kubelet\") pod \"tuned-vh7lw\" (UID: \"145fe4b0-c23a-49dc-abe3-212e53cff730\") " pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.548122 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546940 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fce43675-3aee-453f-b5ad-a28fba84c810-host\") pod \"node-ca-q6fvl\" (UID: \"fce43675-3aee-453f-b5ad-a28fba84c810\") " pod="openshift-image-registry/node-ca-q6fvl" Apr 16 16:47:42.548122 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546957 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/643eef8b-e0e5-48fb-a732-6d880b6b62e7-registration-dir\") pod \"aws-ebs-csi-driver-node-9c8bl\" (UID: \"643eef8b-e0e5-48fb-a732-6d880b6b62e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9c8bl" Apr 16 16:47:42.548122 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546983 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-etc-openvswitch\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.548122 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547010 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-run-ovn\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.548122 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547011 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-etc-openvswitch\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.548122 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547035 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/643eef8b-e0e5-48fb-a732-6d880b6b62e7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9c8bl\" (UID: \"643eef8b-e0e5-48fb-a732-6d880b6b62e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9c8bl" Apr 16 16:47:42.548122 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.546973 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fce43675-3aee-453f-b5ad-a28fba84c810-host\") pod \"node-ca-q6fvl\" (UID: \"fce43675-3aee-453f-b5ad-a28fba84c810\") " pod="openshift-image-registry/node-ca-q6fvl" Apr 16 16:47:42.548122 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547059 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/643eef8b-e0e5-48fb-a732-6d880b6b62e7-device-dir\") pod \"aws-ebs-csi-driver-node-9c8bl\" (UID: \"643eef8b-e0e5-48fb-a732-6d880b6b62e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9c8bl" Apr 16 16:47:42.548122 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547066 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-run-ovn\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.548122 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547095 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/643eef8b-e0e5-48fb-a732-6d880b6b62e7-device-dir\") pod \"aws-ebs-csi-driver-node-9c8bl\" (UID: \"643eef8b-e0e5-48fb-a732-6d880b6b62e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9c8bl" Apr 16 16:47:42.548122 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547092 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qxrps\" (UniqueName: \"kubernetes.io/projected/643eef8b-e0e5-48fb-a732-6d880b6b62e7-kube-api-access-qxrps\") pod \"aws-ebs-csi-driver-node-9c8bl\" (UID: \"643eef8b-e0e5-48fb-a732-6d880b6b62e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9c8bl" Apr 16 16:47:42.548122 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547138 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/643eef8b-e0e5-48fb-a732-6d880b6b62e7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9c8bl\" (UID: \"643eef8b-e0e5-48fb-a732-6d880b6b62e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9c8bl" Apr 16 16:47:42.549077 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547143 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/145fe4b0-c23a-49dc-abe3-212e53cff730-etc-sysctl-conf\") pod \"tuned-vh7lw\" (UID: \"145fe4b0-c23a-49dc-abe3-212e53cff730\") " pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.549077 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547193 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dwmjt\" (UniqueName: \"kubernetes.io/projected/145fe4b0-c23a-49dc-abe3-212e53cff730-kube-api-access-dwmjt\") pod \"tuned-vh7lw\" (UID: \"145fe4b0-c23a-49dc-abe3-212e53cff730\") " pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.549077 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547225 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e078538-0666-4e1e-8b80-2f09583fbd00-multus-cni-dir\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.549077 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547233 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8e078538-0666-4e1e-8b80-2f09583fbd00-multus-daemon-config\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.549077 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547267 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8e078538-0666-4e1e-8b80-2f09583fbd00-hostroot\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.549077 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547274 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/145fe4b0-c23a-49dc-abe3-212e53cff730-etc-sysctl-conf\") pod \"tuned-vh7lw\" (UID: \"145fe4b0-c23a-49dc-abe3-212e53cff730\") " pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.549077 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547292 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2xg5n\" (UniqueName: \"kubernetes.io/projected/8e078538-0666-4e1e-8b80-2f09583fbd00-kube-api-access-2xg5n\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.549077 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547312 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e078538-0666-4e1e-8b80-2f09583fbd00-multus-cni-dir\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.549077 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547318 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8e078538-0666-4e1e-8b80-2f09583fbd00-hostroot\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.549077 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547319 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/98d7d9af-1c21-4f39-b605-21a11d06f2ae-original-pull-secret\") pod \"global-pull-secret-syncer-pgvd2\" (UID: \"98d7d9af-1c21-4f39-b605-21a11d06f2ae\") " pod="kube-system/global-pull-secret-syncer-pgvd2" Apr 16 16:47:42.549077 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547352 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/643eef8b-e0e5-48fb-a732-6d880b6b62e7-etc-selinux\") pod \"aws-ebs-csi-driver-node-9c8bl\" (UID: \"643eef8b-e0e5-48fb-a732-6d880b6b62e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9c8bl" Apr 16 16:47:42.549077 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547385 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/643eef8b-e0e5-48fb-a732-6d880b6b62e7-sys-fs\") pod \"aws-ebs-csi-driver-node-9c8bl\" (UID: \"643eef8b-e0e5-48fb-a732-6d880b6b62e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9c8bl" Apr 16 16:47:42.549077 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547410 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/145fe4b0-c23a-49dc-abe3-212e53cff730-etc-sysconfig\") pod \"tuned-vh7lw\" (UID: \"145fe4b0-c23a-49dc-abe3-212e53cff730\") " pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.549077 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:42.547388 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Apr 16 16:47:42.549077 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547426 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c2fec5cf-333c-4342-bd77-642bb2581568-ovnkube-config\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.549077 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547433 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-host-kubelet\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.549077 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547462 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/145fe4b0-c23a-49dc-abe3-212e53cff730-etc-sysconfig\") pod \"tuned-vh7lw\" (UID: \"145fe4b0-c23a-49dc-abe3-212e53cff730\") " pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.549077 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547462 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-host-kubelet\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.549912 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547426 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/643eef8b-e0e5-48fb-a732-6d880b6b62e7-etc-selinux\") pod \"aws-ebs-csi-driver-node-9c8bl\" (UID: \"643eef8b-e0e5-48fb-a732-6d880b6b62e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9c8bl" Apr 16 16:47:42.549912 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547478 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c2fec5cf-333c-4342-bd77-642bb2581568-ovnkube-script-lib\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.549912 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547431 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/643eef8b-e0e5-48fb-a732-6d880b6b62e7-sys-fs\") pod \"aws-ebs-csi-driver-node-9c8bl\" (UID: \"643eef8b-e0e5-48fb-a732-6d880b6b62e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9c8bl" Apr 16 16:47:42.549912 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:42.547491 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98d7d9af-1c21-4f39-b605-21a11d06f2ae-original-pull-secret podName:98d7d9af-1c21-4f39-b605-21a11d06f2ae nodeName:}" failed. No retries permitted until 2026-04-16 16:47:43.047472228 +0000 UTC m=+2.108621755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/98d7d9af-1c21-4f39-b605-21a11d06f2ae-original-pull-secret") pod "global-pull-secret-syncer-pgvd2" (UID: "98d7d9af-1c21-4f39-b605-21a11d06f2ae") : object "kube-system"/"original-pull-secret" not registered Apr 16 16:47:42.549912 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547533 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kdgpm\" (UniqueName: \"kubernetes.io/projected/c2fec5cf-333c-4342-bd77-642bb2581568-kube-api-access-kdgpm\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.549912 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547564 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20d62408-3657-49b5-a83b-39331108c8d9-system-cni-dir\") pod \"multus-additional-cni-plugins-8zhpq\" (UID: \"20d62408-3657-49b5-a83b-39331108c8d9\") " pod="openshift-multus/multus-additional-cni-plugins-8zhpq" Apr 16 16:47:42.549912 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547591 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e078538-0666-4e1e-8b80-2f09583fbd00-multus-conf-dir\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.549912 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547616 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-host-run-ovn-kubernetes\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.549912 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547631 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e078538-0666-4e1e-8b80-2f09583fbd00-multus-conf-dir\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.549912 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547644 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/20d62408-3657-49b5-a83b-39331108c8d9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8zhpq\" (UID: \"20d62408-3657-49b5-a83b-39331108c8d9\") " pod="openshift-multus/multus-additional-cni-plugins-8zhpq" Apr 16 16:47:42.549912 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547670 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtgpj\" (UniqueName: \"kubernetes.io/projected/e730d89c-85b3-4a44-883b-d65f347261ec-kube-api-access-gtgpj\") pod \"network-check-target-5kz89\" (UID: \"e730d89c-85b3-4a44-883b-d65f347261ec\") " pod="openshift-network-diagnostics/network-check-target-5kz89" Apr 16 16:47:42.549912 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547695 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/145fe4b0-c23a-49dc-abe3-212e53cff730-etc-systemd\") pod \"tuned-vh7lw\" (UID: \"145fe4b0-c23a-49dc-abe3-212e53cff730\") " pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.549912 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547762 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/145fe4b0-c23a-49dc-abe3-212e53cff730-etc-systemd\") pod \"tuned-vh7lw\" (UID: \"145fe4b0-c23a-49dc-abe3-212e53cff730\") " pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.549912 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547670 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-host-run-ovn-kubernetes\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.549912 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547843 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/145fe4b0-c23a-49dc-abe3-212e53cff730-tmp\") pod \"tuned-vh7lw\" (UID: \"145fe4b0-c23a-49dc-abe3-212e53cff730\") " pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.549912 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547871 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e078538-0666-4e1e-8b80-2f09583fbd00-system-cni-dir\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.550653 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547896 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2fef7f9f-e604-47d1-877a-3dfcbbdebba0-iptables-alerter-script\") pod \"iptables-alerter-9vmmq\" (UID: \"2fef7f9f-e604-47d1-877a-3dfcbbdebba0\") " pod="openshift-network-operator/iptables-alerter-9vmmq" Apr 16 16:47:42.550653 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547902 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c2fec5cf-333c-4342-bd77-642bb2581568-ovnkube-script-lib\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.550653 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547923 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/20d62408-3657-49b5-a83b-39331108c8d9-os-release\") pod \"multus-additional-cni-plugins-8zhpq\" (UID: \"20d62408-3657-49b5-a83b-39331108c8d9\") " pod="openshift-multus/multus-additional-cni-plugins-8zhpq" Apr 16 16:47:42.550653 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547950 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/20d62408-3657-49b5-a83b-39331108c8d9-cni-binary-copy\") pod \"multus-additional-cni-plugins-8zhpq\" (UID: \"20d62408-3657-49b5-a83b-39331108c8d9\") " pod="openshift-multus/multus-additional-cni-plugins-8zhpq" Apr 16 16:47:42.550653 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547971 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e078538-0666-4e1e-8b80-2f09583fbd00-system-cni-dir\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.550653 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.547993 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/99da12c7-93e3-4848-afc5-36f2cbf5645d-hosts-file\") pod \"node-resolver-42cdh\" (UID: \"99da12c7-93e3-4848-afc5-36f2cbf5645d\") " pod="openshift-dns/node-resolver-42cdh" Apr 16 16:47:42.550653 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.548019 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c2fec5cf-333c-4342-bd77-642bb2581568-ovn-node-metrics-cert\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.550653 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.548054 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/145fe4b0-c23a-49dc-abe3-212e53cff730-etc-kubernetes\") pod \"tuned-vh7lw\" (UID: \"145fe4b0-c23a-49dc-abe3-212e53cff730\") " pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.550653 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.548079 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/145fe4b0-c23a-49dc-abe3-212e53cff730-sys\") pod \"tuned-vh7lw\" (UID: \"145fe4b0-c23a-49dc-abe3-212e53cff730\") " pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.550653 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.548094 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/145fe4b0-c23a-49dc-abe3-212e53cff730-etc-tuned\") pod \"tuned-vh7lw\" (UID: \"145fe4b0-c23a-49dc-abe3-212e53cff730\") " pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.550653 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.548103 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/145fe4b0-c23a-49dc-abe3-212e53cff730-etc-kubernetes\") pod \"tuned-vh7lw\" (UID: \"145fe4b0-c23a-49dc-abe3-212e53cff730\") " pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.550653 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.548109 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/643eef8b-e0e5-48fb-a732-6d880b6b62e7-socket-dir\") pod \"aws-ebs-csi-driver-node-9c8bl\" (UID: \"643eef8b-e0e5-48fb-a732-6d880b6b62e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9c8bl" Apr 16 16:47:42.550653 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.548105 2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Apr 16 16:47:42.550653 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.548149 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-host-run-netns\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.550653 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.548176 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8e078538-0666-4e1e-8b80-2f09583fbd00-host-run-multus-certs\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.550653 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.548189 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/643eef8b-e0e5-48fb-a732-6d880b6b62e7-socket-dir\") pod \"aws-ebs-csi-driver-node-9c8bl\" (UID: \"643eef8b-e0e5-48fb-a732-6d880b6b62e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9c8bl" Apr 16 16:47:42.550653 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.548200 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/145fe4b0-c23a-49dc-abe3-212e53cff730-run\") pod \"tuned-vh7lw\" (UID: \"145fe4b0-c23a-49dc-abe3-212e53cff730\") " pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.550653 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.548225 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f6b12331-2022-4c05-ac42-753274b499dc-agent-certs\") pod \"konnectivity-agent-kwz5r\" (UID: \"f6b12331-2022-4c05-ac42-753274b499dc\") " pod="kube-system/konnectivity-agent-kwz5r" Apr 16 16:47:42.551477 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.548260 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/145fe4b0-c23a-49dc-abe3-212e53cff730-sys\") pod \"tuned-vh7lw\" (UID: \"145fe4b0-c23a-49dc-abe3-212e53cff730\") " pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.551477 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.548273 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rrvvc\" (UniqueName: \"kubernetes.io/projected/412718e2-2075-4c7b-a6a9-3b94ed57564c-kube-api-access-rrvvc\") pod \"network-metrics-daemon-hk5gr\" (UID: \"412718e2-2075-4c7b-a6a9-3b94ed57564c\") " pod="openshift-multus/network-metrics-daemon-hk5gr" Apr 16 16:47:42.551477 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.548299 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/145fe4b0-c23a-49dc-abe3-212e53cff730-etc-modprobe-d\") pod \"tuned-vh7lw\" (UID: \"145fe4b0-c23a-49dc-abe3-212e53cff730\") " pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.551477 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.548328 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j87pn\" (UniqueName: \"kubernetes.io/projected/20d62408-3657-49b5-a83b-39331108c8d9-kube-api-access-j87pn\") pod \"multus-additional-cni-plugins-8zhpq\" (UID: \"20d62408-3657-49b5-a83b-39331108c8d9\") " pod="openshift-multus/multus-additional-cni-plugins-8zhpq" Apr 16 16:47:42.551477 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.548355 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e078538-0666-4e1e-8b80-2f09583fbd00-cnibin\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.551477 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.548367 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/145fe4b0-c23a-49dc-abe3-212e53cff730-run\") pod \"tuned-vh7lw\" (UID: \"145fe4b0-c23a-49dc-abe3-212e53cff730\") " pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.551477 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.548379 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-systemd-units\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.551477 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.548390 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2fef7f9f-e604-47d1-877a-3dfcbbdebba0-iptables-alerter-script\") pod \"iptables-alerter-9vmmq\" (UID: \"2fef7f9f-e604-47d1-877a-3dfcbbdebba0\") " pod="openshift-network-operator/iptables-alerter-9vmmq" Apr 16 16:47:42.551477 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.548410 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-host-run-netns\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.551477 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.548403 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-run-openvswitch\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.551477 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.548443 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-run-openvswitch\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.551477 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.548447 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8e078538-0666-4e1e-8b80-2f09583fbd00-host-run-multus-certs\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.551477 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.548453 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c2fec5cf-333c-4342-bd77-642bb2581568-env-overrides\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.551477 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.548591 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/145fe4b0-c23a-49dc-abe3-212e53cff730-etc-modprobe-d\") pod \"tuned-vh7lw\" (UID: \"145fe4b0-c23a-49dc-abe3-212e53cff730\") " pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.551477 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.548647 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e078538-0666-4e1e-8b80-2f09583fbd00-cnibin\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.551477 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.548688 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-systemd-units\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.551477 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.548779 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99da12c7-93e3-4848-afc5-36f2cbf5645d-tmp-dir\") pod \"node-resolver-42cdh\" (UID: \"99da12c7-93e3-4848-afc5-36f2cbf5645d\") " pod="openshift-dns/node-resolver-42cdh" Apr 16 16:47:42.551477 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.548803 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fce43675-3aee-453f-b5ad-a28fba84c810-serviceca\") pod \"node-ca-q6fvl\" (UID: \"fce43675-3aee-453f-b5ad-a28fba84c810\") " pod="openshift-image-registry/node-ca-q6fvl" Apr 16 16:47:42.552329 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.548826 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e078538-0666-4e1e-8b80-2f09583fbd00-host-run-netns\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.552329 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.548857 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e078538-0666-4e1e-8b80-2f09583fbd00-host-var-lib-cni-bin\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.552329 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.548885 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.552329 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.548963 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2fec5cf-333c-4342-bd77-642bb2581568-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.552329 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.548975 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c2fec5cf-333c-4342-bd77-642bb2581568-env-overrides\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.552329 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.549000 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e078538-0666-4e1e-8b80-2f09583fbd00-host-var-lib-cni-bin\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.552329 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.549046 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e078538-0666-4e1e-8b80-2f09583fbd00-host-run-netns\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.552329 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.549725 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fce43675-3aee-453f-b5ad-a28fba84c810-serviceca\") pod \"node-ca-q6fvl\" (UID: \"fce43675-3aee-453f-b5ad-a28fba84c810\") " pod="openshift-image-registry/node-ca-q6fvl" Apr 16 16:47:42.552329 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.551126 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/145fe4b0-c23a-49dc-abe3-212e53cff730-etc-tuned\") pod \"tuned-vh7lw\" (UID: \"145fe4b0-c23a-49dc-abe3-212e53cff730\") " pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.552329 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.551206 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/145fe4b0-c23a-49dc-abe3-212e53cff730-tmp\") pod \"tuned-vh7lw\" (UID: \"145fe4b0-c23a-49dc-abe3-212e53cff730\") " pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.552329 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.551362 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f6b12331-2022-4c05-ac42-753274b499dc-agent-certs\") pod \"konnectivity-agent-kwz5r\" (UID: \"f6b12331-2022-4c05-ac42-753274b499dc\") " pod="kube-system/konnectivity-agent-kwz5r" Apr 16 16:47:42.552329 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.551620 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c2fec5cf-333c-4342-bd77-642bb2581568-ovn-node-metrics-cert\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.557966 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.557938 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-569qg\" (UniqueName: \"kubernetes.io/projected/2fef7f9f-e604-47d1-877a-3dfcbbdebba0-kube-api-access-569qg\") pod \"iptables-alerter-9vmmq\" (UID: \"2fef7f9f-e604-47d1-877a-3dfcbbdebba0\") " pod="openshift-network-operator/iptables-alerter-9vmmq" Apr 16 16:47:42.559326 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.559307 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rjsk\" (UniqueName: \"kubernetes.io/projected/fce43675-3aee-453f-b5ad-a28fba84c810-kube-api-access-4rjsk\") pod \"node-ca-q6fvl\" (UID: \"fce43675-3aee-453f-b5ad-a28fba84c810\") " pod="openshift-image-registry/node-ca-q6fvl" Apr 16 16:47:42.559429 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.559411 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwmjt\" (UniqueName: \"kubernetes.io/projected/145fe4b0-c23a-49dc-abe3-212e53cff730-kube-api-access-dwmjt\") pod \"tuned-vh7lw\" (UID: \"145fe4b0-c23a-49dc-abe3-212e53cff730\") " pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.559580 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.559556 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xg5n\" (UniqueName: \"kubernetes.io/projected/8e078538-0666-4e1e-8b80-2f09583fbd00-kube-api-access-2xg5n\") pod \"multus-ndl5w\" (UID: \"8e078538-0666-4e1e-8b80-2f09583fbd00\") " pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.559889 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.559873 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdgpm\" (UniqueName: \"kubernetes.io/projected/c2fec5cf-333c-4342-bd77-642bb2581568-kube-api-access-kdgpm\") pod \"ovnkube-node-p7c8c\" (UID: \"c2fec5cf-333c-4342-bd77-642bb2581568\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.560199 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.560182 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxrps\" (UniqueName: \"kubernetes.io/projected/643eef8b-e0e5-48fb-a732-6d880b6b62e7-kube-api-access-qxrps\") pod \"aws-ebs-csi-driver-node-9c8bl\" (UID: \"643eef8b-e0e5-48fb-a732-6d880b6b62e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9c8bl" Apr 16 16:47:42.560459 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.560445 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrvvc\" (UniqueName: \"kubernetes.io/projected/412718e2-2075-4c7b-a6a9-3b94ed57564c-kube-api-access-rrvvc\") pod \"network-metrics-daemon-hk5gr\" (UID: \"412718e2-2075-4c7b-a6a9-3b94ed57564c\") " pod="openshift-multus/network-metrics-daemon-hk5gr" Apr 16 16:47:42.591766 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:42.591731 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f4d89b70f22123800295882f9c30018.slice/crio-9a2a96931219326ef5603182a96e7191a0ac5baeb5851b2aec832f1fcf26a6a0 WatchSource:0}: Error finding container 9a2a96931219326ef5603182a96e7191a0ac5baeb5851b2aec832f1fcf26a6a0: Status 404 returned error can't find the container with id 9a2a96931219326ef5603182a96e7191a0ac5baeb5851b2aec832f1fcf26a6a0 Apr 16 16:47:42.597808 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.597791 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Apr 16 16:47:42.601979 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:42.601957 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b1f1c157df36eece23646e8d91601bf.slice/crio-f4c2e2149d956b9defc452c5197f8cc47d222dfb839674d6b2db5b71d6697da9 WatchSource:0}: Error finding container f4c2e2149d956b9defc452c5197f8cc47d222dfb839674d6b2db5b71d6697da9: Status 404 returned error can't find the container with id f4c2e2149d956b9defc452c5197f8cc47d222dfb839674d6b2db5b71d6697da9 Apr 16 16:47:42.649509 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.649336 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20d62408-3657-49b5-a83b-39331108c8d9-system-cni-dir\") pod \"multus-additional-cni-plugins-8zhpq\" (UID: \"20d62408-3657-49b5-a83b-39331108c8d9\") " pod="openshift-multus/multus-additional-cni-plugins-8zhpq" Apr 16 16:47:42.649637 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.649528 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/20d62408-3657-49b5-a83b-39331108c8d9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8zhpq\" (UID: \"20d62408-3657-49b5-a83b-39331108c8d9\") " pod="openshift-multus/multus-additional-cni-plugins-8zhpq" Apr 16 16:47:42.649637 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.649549 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gtgpj\" (UniqueName: \"kubernetes.io/projected/e730d89c-85b3-4a44-883b-d65f347261ec-kube-api-access-gtgpj\") pod \"network-check-target-5kz89\" (UID: \"e730d89c-85b3-4a44-883b-d65f347261ec\") " pod="openshift-network-diagnostics/network-check-target-5kz89" Apr 16 16:47:42.649637 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.649568 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/20d62408-3657-49b5-a83b-39331108c8d9-os-release\") pod \"multus-additional-cni-plugins-8zhpq\" (UID: \"20d62408-3657-49b5-a83b-39331108c8d9\") " pod="openshift-multus/multus-additional-cni-plugins-8zhpq" Apr 16 16:47:42.649637 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.649594 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/20d62408-3657-49b5-a83b-39331108c8d9-cni-binary-copy\") pod \"multus-additional-cni-plugins-8zhpq\" (UID: \"20d62408-3657-49b5-a83b-39331108c8d9\") " pod="openshift-multus/multus-additional-cni-plugins-8zhpq" Apr 16 16:47:42.649637 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.649619 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/99da12c7-93e3-4848-afc5-36f2cbf5645d-hosts-file\") pod \"node-resolver-42cdh\" (UID: \"99da12c7-93e3-4848-afc5-36f2cbf5645d\") " pod="openshift-dns/node-resolver-42cdh" Apr 16 16:47:42.649911 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.649653 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-j87pn\" (UniqueName: \"kubernetes.io/projected/20d62408-3657-49b5-a83b-39331108c8d9-kube-api-access-j87pn\") pod \"multus-additional-cni-plugins-8zhpq\" (UID: \"20d62408-3657-49b5-a83b-39331108c8d9\") " pod="openshift-multus/multus-additional-cni-plugins-8zhpq" Apr 16 16:47:42.649911 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.649663 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/20d62408-3657-49b5-a83b-39331108c8d9-os-release\") pod \"multus-additional-cni-plugins-8zhpq\" (UID: \"20d62408-3657-49b5-a83b-39331108c8d9\") " pod="openshift-multus/multus-additional-cni-plugins-8zhpq" Apr 16 16:47:42.649911 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.649680 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99da12c7-93e3-4848-afc5-36f2cbf5645d-tmp-dir\") pod \"node-resolver-42cdh\" (UID: \"99da12c7-93e3-4848-afc5-36f2cbf5645d\") " pod="openshift-dns/node-resolver-42cdh" Apr 16 16:47:42.649911 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.649437 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20d62408-3657-49b5-a83b-39331108c8d9-system-cni-dir\") pod \"multus-additional-cni-plugins-8zhpq\" (UID: \"20d62408-3657-49b5-a83b-39331108c8d9\") " pod="openshift-multus/multus-additional-cni-plugins-8zhpq" Apr 16 16:47:42.649911 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.649699 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/99da12c7-93e3-4848-afc5-36f2cbf5645d-hosts-file\") pod \"node-resolver-42cdh\" (UID: \"99da12c7-93e3-4848-afc5-36f2cbf5645d\") " pod="openshift-dns/node-resolver-42cdh" Apr 16 16:47:42.649911 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.649725 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/20d62408-3657-49b5-a83b-39331108c8d9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8zhpq\" (UID: \"20d62408-3657-49b5-a83b-39331108c8d9\") " pod="openshift-multus/multus-additional-cni-plugins-8zhpq" Apr 16 16:47:42.649911 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.649750 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/20d62408-3657-49b5-a83b-39331108c8d9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8zhpq\" (UID: \"20d62408-3657-49b5-a83b-39331108c8d9\") " pod="openshift-multus/multus-additional-cni-plugins-8zhpq" Apr 16 16:47:42.649911 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.649781 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nklr8\" (UniqueName: \"kubernetes.io/projected/99da12c7-93e3-4848-afc5-36f2cbf5645d-kube-api-access-nklr8\") pod \"node-resolver-42cdh\" (UID: \"99da12c7-93e3-4848-afc5-36f2cbf5645d\") " pod="openshift-dns/node-resolver-42cdh" Apr 16 16:47:42.649911 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.649822 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/20d62408-3657-49b5-a83b-39331108c8d9-cnibin\") pod \"multus-additional-cni-plugins-8zhpq\" (UID: \"20d62408-3657-49b5-a83b-39331108c8d9\") " pod="openshift-multus/multus-additional-cni-plugins-8zhpq" Apr 16 16:47:42.650334 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.649938 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/20d62408-3657-49b5-a83b-39331108c8d9-cnibin\") pod \"multus-additional-cni-plugins-8zhpq\" (UID: \"20d62408-3657-49b5-a83b-39331108c8d9\") " pod="openshift-multus/multus-additional-cni-plugins-8zhpq" Apr 16 16:47:42.650334 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.649985 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99da12c7-93e3-4848-afc5-36f2cbf5645d-tmp-dir\") pod \"node-resolver-42cdh\" (UID: \"99da12c7-93e3-4848-afc5-36f2cbf5645d\") " pod="openshift-dns/node-resolver-42cdh" Apr 16 16:47:42.650334 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.650159 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/20d62408-3657-49b5-a83b-39331108c8d9-cni-binary-copy\") pod \"multus-additional-cni-plugins-8zhpq\" (UID: \"20d62408-3657-49b5-a83b-39331108c8d9\") " pod="openshift-multus/multus-additional-cni-plugins-8zhpq" Apr 16 16:47:42.650334 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.650264 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/20d62408-3657-49b5-a83b-39331108c8d9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8zhpq\" (UID: \"20d62408-3657-49b5-a83b-39331108c8d9\") " pod="openshift-multus/multus-additional-cni-plugins-8zhpq" Apr 16 16:47:42.650334 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.650323 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/20d62408-3657-49b5-a83b-39331108c8d9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8zhpq\" (UID: \"20d62408-3657-49b5-a83b-39331108c8d9\") " pod="openshift-multus/multus-additional-cni-plugins-8zhpq" Apr 16 16:47:42.650498 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.650326 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/20d62408-3657-49b5-a83b-39331108c8d9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8zhpq\" (UID: \"20d62408-3657-49b5-a83b-39331108c8d9\") " pod="openshift-multus/multus-additional-cni-plugins-8zhpq" Apr 16 16:47:42.655079 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:42.655062 2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 16 16:47:42.655079 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:42.655079 2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 16 16:47:42.655213 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:42.655088 2576 projected.go:194] Error preparing data for projected volume kube-api-access-gtgpj for pod openshift-network-diagnostics/network-check-target-5kz89: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 16:47:42.655213 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:42.655139 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e730d89c-85b3-4a44-883b-d65f347261ec-kube-api-access-gtgpj podName:e730d89c-85b3-4a44-883b-d65f347261ec nodeName:}" failed. No retries permitted until 2026-04-16 16:47:43.15512645 +0000 UTC m=+2.216275959 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gtgpj" (UniqueName: "kubernetes.io/projected/e730d89c-85b3-4a44-883b-d65f347261ec-kube-api-access-gtgpj") pod "network-check-target-5kz89" (UID: "e730d89c-85b3-4a44-883b-d65f347261ec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 16:47:42.657477 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.657455 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nklr8\" (UniqueName: \"kubernetes.io/projected/99da12c7-93e3-4848-afc5-36f2cbf5645d-kube-api-access-nklr8\") pod \"node-resolver-42cdh\" (UID: \"99da12c7-93e3-4848-afc5-36f2cbf5645d\") " pod="openshift-dns/node-resolver-42cdh" Apr 16 16:47:42.657547 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.657454 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j87pn\" (UniqueName: \"kubernetes.io/projected/20d62408-3657-49b5-a83b-39331108c8d9-kube-api-access-j87pn\") pod \"multus-additional-cni-plugins-8zhpq\" (UID: \"20d62408-3657-49b5-a83b-39331108c8d9\") " pod="openshift-multus/multus-additional-cni-plugins-8zhpq" Apr 16 16:47:42.756529 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.756432 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kwz5r" Apr 16 16:47:42.763128 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:42.763100 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6b12331_2022_4c05_ac42_753274b499dc.slice/crio-d0ee0d7b075599714376ce8eb86632f8fe2d3d80da1fde683d6ee90d3ac073c1 WatchSource:0}: Error finding container d0ee0d7b075599714376ce8eb86632f8fe2d3d80da1fde683d6ee90d3ac073c1: Status 404 returned error can't find the container with id d0ee0d7b075599714376ce8eb86632f8fe2d3d80da1fde683d6ee90d3ac073c1 Apr 16 16:47:42.771898 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.771873 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q6fvl" Apr 16 16:47:42.778205 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.778107 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ndl5w" Apr 16 16:47:42.784000 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:42.783978 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e078538_0666_4e1e_8b80_2f09583fbd00.slice/crio-c1b8b3357ebc0805a1c69791ec35dc61797a112297dc5da34e794ee95ab1f240 WatchSource:0}: Error finding container c1b8b3357ebc0805a1c69791ec35dc61797a112297dc5da34e794ee95ab1f240: Status 404 returned error can't find the container with id c1b8b3357ebc0805a1c69791ec35dc61797a112297dc5da34e794ee95ab1f240 Apr 16 16:47:42.795915 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.795894 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9vmmq" Apr 16 16:47:42.801404 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:42.801383 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fef7f9f_e604_47d1_877a_3dfcbbdebba0.slice/crio-2b2d7bc2cdd212df75c7d7aeb823e41915df21bfe29ceef1e7e8807814d657fa WatchSource:0}: Error finding container 2b2d7bc2cdd212df75c7d7aeb823e41915df21bfe29ceef1e7e8807814d657fa: Status 404 returned error can't find the container with id 2b2d7bc2cdd212df75c7d7aeb823e41915df21bfe29ceef1e7e8807814d657fa Apr 16 16:47:42.817522 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.817505 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:47:42.823300 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:42.823278 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2fec5cf_333c_4342_bd77_642bb2581568.slice/crio-d611e8ed60b4d4c897e04c9872db1145ff862c7704c1a288ae10f5672241c9d9 WatchSource:0}: Error finding container d611e8ed60b4d4c897e04c9872db1145ff862c7704c1a288ae10f5672241c9d9: Status 404 returned error can't find the container with id d611e8ed60b4d4c897e04c9872db1145ff862c7704c1a288ae10f5672241c9d9 Apr 16 16:47:42.829068 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.829047 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9c8bl" Apr 16 16:47:42.835266 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:42.835225 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod643eef8b_e0e5_48fb_a732_6d880b6b62e7.slice/crio-d09c11c4c89f01e914e6d644af8406d148c1280f3d633e3514e925e040c990d6 WatchSource:0}: Error finding container d09c11c4c89f01e914e6d644af8406d148c1280f3d633e3514e925e040c990d6: Status 404 returned error can't find the container with id d09c11c4c89f01e914e6d644af8406d148c1280f3d633e3514e925e040c990d6 Apr 16 16:47:42.846488 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.846471 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" Apr 16 16:47:42.852058 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:42.852036 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod145fe4b0_c23a_49dc_abe3_212e53cff730.slice/crio-dc4d24994efa0b2ad1f4e2fd050023f8bddbb23768ceb1937c94a948a9ab6b37 WatchSource:0}: Error finding container dc4d24994efa0b2ad1f4e2fd050023f8bddbb23768ceb1937c94a948a9ab6b37: Status 404 returned error can't find the container with id dc4d24994efa0b2ad1f4e2fd050023f8bddbb23768ceb1937c94a948a9ab6b37 Apr 16 16:47:42.874897 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.874865 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-42cdh" Apr 16 16:47:42.875787 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.875773 2576 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Apr 16 16:47:42.879436 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:42.879414 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8zhpq" Apr 16 16:47:42.883038 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:42.883012 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99da12c7_93e3_4848_afc5_36f2cbf5645d.slice/crio-efc61bd4f63a2273a8e9aa8bcb06c513b48b8b4e4e275ce32fdbbc44d5074eaa WatchSource:0}: Error finding container efc61bd4f63a2273a8e9aa8bcb06c513b48b8b4e4e275ce32fdbbc44d5074eaa: Status 404 returned error can't find the container with id efc61bd4f63a2273a8e9aa8bcb06c513b48b8b4e4e275ce32fdbbc44d5074eaa Apr 16 16:47:42.887885 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:47:42.887863 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20d62408_3657_49b5_a83b_39331108c8d9.slice/crio-90dfb0d7f018616449fe970adc3908e9078e7bfc505d1ae70cdcd4fb80f9b4eb WatchSource:0}: Error finding container 90dfb0d7f018616449fe970adc3908e9078e7bfc505d1ae70cdcd4fb80f9b4eb: Status 404 returned error can't find the container with id 90dfb0d7f018616449fe970adc3908e9078e7bfc505d1ae70cdcd4fb80f9b4eb Apr 16 16:47:43.052728 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:43.052694 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/412718e2-2075-4c7b-a6a9-3b94ed57564c-metrics-certs\") pod \"network-metrics-daemon-hk5gr\" (UID: \"412718e2-2075-4c7b-a6a9-3b94ed57564c\") " pod="openshift-multus/network-metrics-daemon-hk5gr" Apr 16 16:47:43.052868 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:43.052742 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/98d7d9af-1c21-4f39-b605-21a11d06f2ae-original-pull-secret\") pod \"global-pull-secret-syncer-pgvd2\" (UID: \"98d7d9af-1c21-4f39-b605-21a11d06f2ae\") " pod="kube-system/global-pull-secret-syncer-pgvd2" Apr 16 16:47:43.052868 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:43.052841 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 16:47:43.052868 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:43.052850 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Apr 16 16:47:43.053039 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:43.052906 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/412718e2-2075-4c7b-a6a9-3b94ed57564c-metrics-certs podName:412718e2-2075-4c7b-a6a9-3b94ed57564c nodeName:}" failed. No retries permitted until 2026-04-16 16:47:44.052887646 +0000 UTC m=+3.114037169 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/412718e2-2075-4c7b-a6a9-3b94ed57564c-metrics-certs") pod "network-metrics-daemon-hk5gr" (UID: "412718e2-2075-4c7b-a6a9-3b94ed57564c") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 16:47:43.053039 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:43.052920 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98d7d9af-1c21-4f39-b605-21a11d06f2ae-original-pull-secret podName:98d7d9af-1c21-4f39-b605-21a11d06f2ae nodeName:}" failed. No retries permitted until 2026-04-16 16:47:44.052914343 +0000 UTC m=+3.114063853 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/98d7d9af-1c21-4f39-b605-21a11d06f2ae-original-pull-secret") pod "global-pull-secret-syncer-pgvd2" (UID: "98d7d9af-1c21-4f39-b605-21a11d06f2ae") : object "kube-system"/"original-pull-secret" not registered Apr 16 16:47:43.254473 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:43.254442 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gtgpj\" (UniqueName: \"kubernetes.io/projected/e730d89c-85b3-4a44-883b-d65f347261ec-kube-api-access-gtgpj\") pod \"network-check-target-5kz89\" (UID: \"e730d89c-85b3-4a44-883b-d65f347261ec\") " pod="openshift-network-diagnostics/network-check-target-5kz89" Apr 16 16:47:43.254654 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:43.254603 2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 16 16:47:43.254654 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:43.254625 2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 16 16:47:43.254654 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:43.254637 2576 projected.go:194] Error preparing data for projected volume kube-api-access-gtgpj for pod openshift-network-diagnostics/network-check-target-5kz89: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 16:47:43.254810 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:43.254684 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e730d89c-85b3-4a44-883b-d65f347261ec-kube-api-access-gtgpj podName:e730d89c-85b3-4a44-883b-d65f347261ec nodeName:}" failed. No retries permitted until 2026-04-16 16:47:44.254670912 +0000 UTC m=+3.315820424 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gtgpj" (UniqueName: "kubernetes.io/projected/e730d89c-85b3-4a44-883b-d65f347261ec-kube-api-access-gtgpj") pod "network-check-target-5kz89" (UID: "e730d89c-85b3-4a44-883b-d65f347261ec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 16:47:43.488749 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:43.488634 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-04-15 16:42:42 +0000 UTC" deadline="2027-11-25 21:48:43.79199361 +0000 UTC" Apr 16 16:47:43.488749 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:43.488672 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14117h1m0.303325706s" Apr 16 16:47:43.590891 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:43.590856 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kz89" Apr 16 16:47:43.591058 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:43.591010 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5kz89" podUID="e730d89c-85b3-4a44-883b-d65f347261ec" Apr 16 16:47:43.602521 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:43.602329 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9c8bl" event={"ID":"643eef8b-e0e5-48fb-a732-6d880b6b62e7","Type":"ContainerStarted","Data":"d09c11c4c89f01e914e6d644af8406d148c1280f3d633e3514e925e040c990d6"} Apr 16 16:47:43.613971 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:43.613934 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" event={"ID":"c2fec5cf-333c-4342-bd77-642bb2581568","Type":"ContainerStarted","Data":"d611e8ed60b4d4c897e04c9872db1145ff862c7704c1a288ae10f5672241c9d9"} Apr 16 16:47:43.626963 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:43.626910 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ndl5w" event={"ID":"8e078538-0666-4e1e-8b80-2f09583fbd00","Type":"ContainerStarted","Data":"c1b8b3357ebc0805a1c69791ec35dc61797a112297dc5da34e794ee95ab1f240"} Apr 16 16:47:43.663432 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:43.663387 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q6fvl" event={"ID":"fce43675-3aee-453f-b5ad-a28fba84c810","Type":"ContainerStarted","Data":"58feeb5eed5b26c78276fda5ecd37c300337629784310e0881a995e483ae4b91"} Apr 16 16:47:43.665266 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:43.665208 2576 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Apr 16 16:47:43.677402 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:43.677370 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-37.ec2.internal" event={"ID":"8b1f1c157df36eece23646e8d91601bf","Type":"ContainerStarted","Data":"f4c2e2149d956b9defc452c5197f8cc47d222dfb839674d6b2db5b71d6697da9"} Apr 16 16:47:43.690549 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:43.690494 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-37.ec2.internal" event={"ID":"0f4d89b70f22123800295882f9c30018","Type":"ContainerStarted","Data":"9a2a96931219326ef5603182a96e7191a0ac5baeb5851b2aec832f1fcf26a6a0"} Apr 16 16:47:43.701636 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:43.701606 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-42cdh" event={"ID":"99da12c7-93e3-4848-afc5-36f2cbf5645d","Type":"ContainerStarted","Data":"efc61bd4f63a2273a8e9aa8bcb06c513b48b8b4e4e275ce32fdbbc44d5074eaa"} Apr 16 16:47:43.727688 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:43.727653 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9vmmq" event={"ID":"2fef7f9f-e604-47d1-877a-3dfcbbdebba0","Type":"ContainerStarted","Data":"2b2d7bc2cdd212df75c7d7aeb823e41915df21bfe29ceef1e7e8807814d657fa"} Apr 16 16:47:43.741398 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:43.741326 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kwz5r" event={"ID":"f6b12331-2022-4c05-ac42-753274b499dc","Type":"ContainerStarted","Data":"d0ee0d7b075599714376ce8eb86632f8fe2d3d80da1fde683d6ee90d3ac073c1"} Apr 16 16:47:43.760239 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:43.760201 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8zhpq" event={"ID":"20d62408-3657-49b5-a83b-39331108c8d9","Type":"ContainerStarted","Data":"90dfb0d7f018616449fe970adc3908e9078e7bfc505d1ae70cdcd4fb80f9b4eb"} Apr 16 16:47:43.771037 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:43.770968 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" event={"ID":"145fe4b0-c23a-49dc-abe3-212e53cff730","Type":"ContainerStarted","Data":"dc4d24994efa0b2ad1f4e2fd050023f8bddbb23768ceb1937c94a948a9ab6b37"} Apr 16 16:47:44.059516 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:44.059479 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/412718e2-2075-4c7b-a6a9-3b94ed57564c-metrics-certs\") pod \"network-metrics-daemon-hk5gr\" (UID: \"412718e2-2075-4c7b-a6a9-3b94ed57564c\") " pod="openshift-multus/network-metrics-daemon-hk5gr" Apr 16 16:47:44.059707 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:44.059541 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/98d7d9af-1c21-4f39-b605-21a11d06f2ae-original-pull-secret\") pod \"global-pull-secret-syncer-pgvd2\" (UID: \"98d7d9af-1c21-4f39-b605-21a11d06f2ae\") " pod="kube-system/global-pull-secret-syncer-pgvd2" Apr 16 16:47:44.059707 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:44.059693 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Apr 16 16:47:44.059833 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:44.059760 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98d7d9af-1c21-4f39-b605-21a11d06f2ae-original-pull-secret podName:98d7d9af-1c21-4f39-b605-21a11d06f2ae nodeName:}" failed. No retries permitted until 2026-04-16 16:47:46.05974022 +0000 UTC m=+5.120889731 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/98d7d9af-1c21-4f39-b605-21a11d06f2ae-original-pull-secret") pod "global-pull-secret-syncer-pgvd2" (UID: "98d7d9af-1c21-4f39-b605-21a11d06f2ae") : object "kube-system"/"original-pull-secret" not registered Apr 16 16:47:44.060188 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:44.060158 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 16:47:44.060294 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:44.060214 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/412718e2-2075-4c7b-a6a9-3b94ed57564c-metrics-certs podName:412718e2-2075-4c7b-a6a9-3b94ed57564c nodeName:}" failed. No retries permitted until 2026-04-16 16:47:46.060199109 +0000 UTC m=+5.121348625 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/412718e2-2075-4c7b-a6a9-3b94ed57564c-metrics-certs") pod "network-metrics-daemon-hk5gr" (UID: "412718e2-2075-4c7b-a6a9-3b94ed57564c") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 16:47:44.261156 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:44.261108 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gtgpj\" (UniqueName: \"kubernetes.io/projected/e730d89c-85b3-4a44-883b-d65f347261ec-kube-api-access-gtgpj\") pod \"network-check-target-5kz89\" (UID: \"e730d89c-85b3-4a44-883b-d65f347261ec\") " pod="openshift-network-diagnostics/network-check-target-5kz89" Apr 16 16:47:44.261348 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:44.261296 2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 16 16:47:44.261348 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:44.261315 2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 16 16:47:44.261348 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:44.261328 2576 projected.go:194] Error preparing data for projected volume kube-api-access-gtgpj for pod openshift-network-diagnostics/network-check-target-5kz89: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 16:47:44.261521 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:44.261393 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e730d89c-85b3-4a44-883b-d65f347261ec-kube-api-access-gtgpj podName:e730d89c-85b3-4a44-883b-d65f347261ec nodeName:}" failed. No retries permitted until 2026-04-16 16:47:46.261375315 +0000 UTC m=+5.322524847 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gtgpj" (UniqueName: "kubernetes.io/projected/e730d89c-85b3-4a44-883b-d65f347261ec-kube-api-access-gtgpj") pod "network-check-target-5kz89" (UID: "e730d89c-85b3-4a44-883b-d65f347261ec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 16:47:44.489092 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:44.488985 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-04-15 16:42:42 +0000 UTC" deadline="2027-12-22 15:08:19.444796434 +0000 UTC" Apr 16 16:47:44.489092 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:44.489025 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14758h20m34.955775244s" Apr 16 16:47:44.588346 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:44.588307 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pgvd2" Apr 16 16:47:44.588533 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:44.588440 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pgvd2" podUID="98d7d9af-1c21-4f39-b605-21a11d06f2ae" Apr 16 16:47:44.588878 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:44.588860 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk5gr" Apr 16 16:47:44.588983 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:44.588964 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk5gr" podUID="412718e2-2075-4c7b-a6a9-3b94ed57564c" Apr 16 16:47:45.591080 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:45.590143 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kz89" Apr 16 16:47:45.591080 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:45.590286 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5kz89" podUID="e730d89c-85b3-4a44-883b-d65f347261ec" Apr 16 16:47:46.075266 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:46.075218 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/412718e2-2075-4c7b-a6a9-3b94ed57564c-metrics-certs\") pod \"network-metrics-daemon-hk5gr\" (UID: \"412718e2-2075-4c7b-a6a9-3b94ed57564c\") " pod="openshift-multus/network-metrics-daemon-hk5gr" Apr 16 16:47:46.075452 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:46.075295 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/98d7d9af-1c21-4f39-b605-21a11d06f2ae-original-pull-secret\") pod \"global-pull-secret-syncer-pgvd2\" (UID: \"98d7d9af-1c21-4f39-b605-21a11d06f2ae\") " pod="kube-system/global-pull-secret-syncer-pgvd2" Apr 16 16:47:46.075452 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:46.075426 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Apr 16 16:47:46.075562 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:46.075475 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98d7d9af-1c21-4f39-b605-21a11d06f2ae-original-pull-secret podName:98d7d9af-1c21-4f39-b605-21a11d06f2ae nodeName:}" failed. No retries permitted until 2026-04-16 16:47:50.075462597 +0000 UTC m=+9.136612119 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/98d7d9af-1c21-4f39-b605-21a11d06f2ae-original-pull-secret") pod "global-pull-secret-syncer-pgvd2" (UID: "98d7d9af-1c21-4f39-b605-21a11d06f2ae") : object "kube-system"/"original-pull-secret" not registered Apr 16 16:47:46.075761 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:46.075744 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 16:47:46.075834 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:46.075793 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/412718e2-2075-4c7b-a6a9-3b94ed57564c-metrics-certs podName:412718e2-2075-4c7b-a6a9-3b94ed57564c nodeName:}" failed. No retries permitted until 2026-04-16 16:47:50.075779717 +0000 UTC m=+9.136929227 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/412718e2-2075-4c7b-a6a9-3b94ed57564c-metrics-certs") pod "network-metrics-daemon-hk5gr" (UID: "412718e2-2075-4c7b-a6a9-3b94ed57564c") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 16:47:46.278207 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:46.277608 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gtgpj\" (UniqueName: \"kubernetes.io/projected/e730d89c-85b3-4a44-883b-d65f347261ec-kube-api-access-gtgpj\") pod \"network-check-target-5kz89\" (UID: \"e730d89c-85b3-4a44-883b-d65f347261ec\") " pod="openshift-network-diagnostics/network-check-target-5kz89" Apr 16 16:47:46.278207 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:46.277769 2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 16 16:47:46.278207 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:46.277787 2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 16 16:47:46.278207 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:46.277800 2576 projected.go:194] Error preparing data for projected volume kube-api-access-gtgpj for pod openshift-network-diagnostics/network-check-target-5kz89: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 16:47:46.278207 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:46.277857 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e730d89c-85b3-4a44-883b-d65f347261ec-kube-api-access-gtgpj podName:e730d89c-85b3-4a44-883b-d65f347261ec nodeName:}" failed. No retries permitted until 2026-04-16 16:47:50.277840029 +0000 UTC m=+9.338989552 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gtgpj" (UniqueName: "kubernetes.io/projected/e730d89c-85b3-4a44-883b-d65f347261ec-kube-api-access-gtgpj") pod "network-check-target-5kz89" (UID: "e730d89c-85b3-4a44-883b-d65f347261ec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 16:47:46.589051 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:46.589018 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pgvd2" Apr 16 16:47:46.589238 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:46.589149 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pgvd2" podUID="98d7d9af-1c21-4f39-b605-21a11d06f2ae" Apr 16 16:47:46.589621 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:46.589595 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk5gr" Apr 16 16:47:46.589738 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:46.589710 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk5gr" podUID="412718e2-2075-4c7b-a6a9-3b94ed57564c" Apr 16 16:47:47.590903 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:47.589050 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kz89" Apr 16 16:47:47.590903 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:47.589170 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5kz89" podUID="e730d89c-85b3-4a44-883b-d65f347261ec" Apr 16 16:47:48.588390 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:48.588356 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pgvd2" Apr 16 16:47:48.588593 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:48.588474 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pgvd2" podUID="98d7d9af-1c21-4f39-b605-21a11d06f2ae" Apr 16 16:47:48.588881 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:48.588857 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk5gr" Apr 16 16:47:48.588962 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:48.588944 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk5gr" podUID="412718e2-2075-4c7b-a6a9-3b94ed57564c" Apr 16 16:47:49.588163 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:49.588123 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kz89" Apr 16 16:47:49.588620 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:49.588263 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5kz89" podUID="e730d89c-85b3-4a44-883b-d65f347261ec" Apr 16 16:47:50.111774 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:50.111731 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/412718e2-2075-4c7b-a6a9-3b94ed57564c-metrics-certs\") pod \"network-metrics-daemon-hk5gr\" (UID: \"412718e2-2075-4c7b-a6a9-3b94ed57564c\") " pod="openshift-multus/network-metrics-daemon-hk5gr" Apr 16 16:47:50.111938 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:50.111808 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/98d7d9af-1c21-4f39-b605-21a11d06f2ae-original-pull-secret\") pod \"global-pull-secret-syncer-pgvd2\" (UID: \"98d7d9af-1c21-4f39-b605-21a11d06f2ae\") " pod="kube-system/global-pull-secret-syncer-pgvd2" Apr 16 16:47:50.111993 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:50.111968 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Apr 16 16:47:50.112041 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:50.112026 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98d7d9af-1c21-4f39-b605-21a11d06f2ae-original-pull-secret podName:98d7d9af-1c21-4f39-b605-21a11d06f2ae nodeName:}" failed. No retries permitted until 2026-04-16 16:47:58.112011843 +0000 UTC m=+17.173161368 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/98d7d9af-1c21-4f39-b605-21a11d06f2ae-original-pull-secret") pod "global-pull-secret-syncer-pgvd2" (UID: "98d7d9af-1c21-4f39-b605-21a11d06f2ae") : object "kube-system"/"original-pull-secret" not registered Apr 16 16:47:50.112447 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:50.112363 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 16:47:50.112447 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:50.112416 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/412718e2-2075-4c7b-a6a9-3b94ed57564c-metrics-certs podName:412718e2-2075-4c7b-a6a9-3b94ed57564c nodeName:}" failed. No retries permitted until 2026-04-16 16:47:58.112400445 +0000 UTC m=+17.173549958 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/412718e2-2075-4c7b-a6a9-3b94ed57564c-metrics-certs") pod "network-metrics-daemon-hk5gr" (UID: "412718e2-2075-4c7b-a6a9-3b94ed57564c") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 16:47:50.314811 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:50.314219 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gtgpj\" (UniqueName: \"kubernetes.io/projected/e730d89c-85b3-4a44-883b-d65f347261ec-kube-api-access-gtgpj\") pod \"network-check-target-5kz89\" (UID: \"e730d89c-85b3-4a44-883b-d65f347261ec\") " pod="openshift-network-diagnostics/network-check-target-5kz89" Apr 16 16:47:50.314811 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:50.314411 2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 16 16:47:50.314811 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:50.314432 2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 16 16:47:50.314811 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:50.314444 2576 projected.go:194] Error preparing data for projected volume kube-api-access-gtgpj for pod openshift-network-diagnostics/network-check-target-5kz89: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 16:47:50.314811 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:50.314502 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e730d89c-85b3-4a44-883b-d65f347261ec-kube-api-access-gtgpj podName:e730d89c-85b3-4a44-883b-d65f347261ec nodeName:}" failed. No retries permitted until 2026-04-16 16:47:58.314483959 +0000 UTC m=+17.375633484 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gtgpj" (UniqueName: "kubernetes.io/projected/e730d89c-85b3-4a44-883b-d65f347261ec-kube-api-access-gtgpj") pod "network-check-target-5kz89" (UID: "e730d89c-85b3-4a44-883b-d65f347261ec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 16:47:50.589351 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:50.588671 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pgvd2" Apr 16 16:47:50.589351 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:50.588802 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pgvd2" podUID="98d7d9af-1c21-4f39-b605-21a11d06f2ae" Apr 16 16:47:50.589351 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:50.589181 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk5gr" Apr 16 16:47:50.589351 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:50.589305 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk5gr" podUID="412718e2-2075-4c7b-a6a9-3b94ed57564c" Apr 16 16:47:51.589161 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:51.589128 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kz89" Apr 16 16:47:51.589371 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:51.589215 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5kz89" podUID="e730d89c-85b3-4a44-883b-d65f347261ec" Apr 16 16:47:52.588926 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:52.588894 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk5gr" Apr 16 16:47:52.589108 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:52.588895 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pgvd2" Apr 16 16:47:52.589108 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:52.589018 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk5gr" podUID="412718e2-2075-4c7b-a6a9-3b94ed57564c" Apr 16 16:47:52.589220 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:52.589109 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pgvd2" podUID="98d7d9af-1c21-4f39-b605-21a11d06f2ae" Apr 16 16:47:53.588488 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:53.588454 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kz89" Apr 16 16:47:53.588957 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:53.588583 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5kz89" podUID="e730d89c-85b3-4a44-883b-d65f347261ec" Apr 16 16:47:54.588543 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:54.588508 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pgvd2" Apr 16 16:47:54.589009 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:54.588508 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk5gr" Apr 16 16:47:54.589009 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:54.588657 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pgvd2" podUID="98d7d9af-1c21-4f39-b605-21a11d06f2ae" Apr 16 16:47:54.589009 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:54.588736 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk5gr" podUID="412718e2-2075-4c7b-a6a9-3b94ed57564c" Apr 16 16:47:55.588994 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:55.588952 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kz89" Apr 16 16:47:55.589423 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:55.589067 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5kz89" podUID="e730d89c-85b3-4a44-883b-d65f347261ec" Apr 16 16:47:56.588125 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:56.588091 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pgvd2" Apr 16 16:47:56.588295 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:56.588210 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pgvd2" podUID="98d7d9af-1c21-4f39-b605-21a11d06f2ae" Apr 16 16:47:56.588295 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:56.588268 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk5gr" Apr 16 16:47:56.588394 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:56.588375 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk5gr" podUID="412718e2-2075-4c7b-a6a9-3b94ed57564c" Apr 16 16:47:57.589232 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:57.589139 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kz89" Apr 16 16:47:57.589611 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:57.589279 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5kz89" podUID="e730d89c-85b3-4a44-883b-d65f347261ec" Apr 16 16:47:58.177038 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:58.177002 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/412718e2-2075-4c7b-a6a9-3b94ed57564c-metrics-certs\") pod \"network-metrics-daemon-hk5gr\" (UID: \"412718e2-2075-4c7b-a6a9-3b94ed57564c\") " pod="openshift-multus/network-metrics-daemon-hk5gr" Apr 16 16:47:58.177038 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:58.177044 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/98d7d9af-1c21-4f39-b605-21a11d06f2ae-original-pull-secret\") pod \"global-pull-secret-syncer-pgvd2\" (UID: \"98d7d9af-1c21-4f39-b605-21a11d06f2ae\") " pod="kube-system/global-pull-secret-syncer-pgvd2" Apr 16 16:47:58.177337 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:58.177159 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 16:47:58.177337 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:58.177214 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/412718e2-2075-4c7b-a6a9-3b94ed57564c-metrics-certs podName:412718e2-2075-4c7b-a6a9-3b94ed57564c nodeName:}" failed. No retries permitted until 2026-04-16 16:48:14.177199653 +0000 UTC m=+33.238349162 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/412718e2-2075-4c7b-a6a9-3b94ed57564c-metrics-certs") pod "network-metrics-daemon-hk5gr" (UID: "412718e2-2075-4c7b-a6a9-3b94ed57564c") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 16:47:58.177337 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:58.177159 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Apr 16 16:47:58.177337 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:58.177297 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98d7d9af-1c21-4f39-b605-21a11d06f2ae-original-pull-secret podName:98d7d9af-1c21-4f39-b605-21a11d06f2ae nodeName:}" failed. No retries permitted until 2026-04-16 16:48:14.177285692 +0000 UTC m=+33.238435203 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/98d7d9af-1c21-4f39-b605-21a11d06f2ae-original-pull-secret") pod "global-pull-secret-syncer-pgvd2" (UID: "98d7d9af-1c21-4f39-b605-21a11d06f2ae") : object "kube-system"/"original-pull-secret" not registered Apr 16 16:47:58.379046 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:58.379012 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gtgpj\" (UniqueName: \"kubernetes.io/projected/e730d89c-85b3-4a44-883b-d65f347261ec-kube-api-access-gtgpj\") pod \"network-check-target-5kz89\" (UID: \"e730d89c-85b3-4a44-883b-d65f347261ec\") " pod="openshift-network-diagnostics/network-check-target-5kz89" Apr 16 16:47:58.379234 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:58.379160 2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 16 16:47:58.379234 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:58.379178 2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 16 16:47:58.379234 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:58.379190 2576 projected.go:194] Error preparing data for projected volume kube-api-access-gtgpj for pod openshift-network-diagnostics/network-check-target-5kz89: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 16:47:58.379422 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:58.379267 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e730d89c-85b3-4a44-883b-d65f347261ec-kube-api-access-gtgpj podName:e730d89c-85b3-4a44-883b-d65f347261ec nodeName:}" failed. No retries permitted until 2026-04-16 16:48:14.379232808 +0000 UTC m=+33.440382319 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gtgpj" (UniqueName: "kubernetes.io/projected/e730d89c-85b3-4a44-883b-d65f347261ec-kube-api-access-gtgpj") pod "network-check-target-5kz89" (UID: "e730d89c-85b3-4a44-883b-d65f347261ec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 16:47:58.588187 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:58.588150 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pgvd2" Apr 16 16:47:58.588394 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:58.588156 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk5gr" Apr 16 16:47:58.588394 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:58.588292 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pgvd2" podUID="98d7d9af-1c21-4f39-b605-21a11d06f2ae" Apr 16 16:47:58.588494 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:58.588393 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk5gr" podUID="412718e2-2075-4c7b-a6a9-3b94ed57564c" Apr 16 16:47:59.588954 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:47:59.588920 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kz89" Apr 16 16:47:59.589370 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:47:59.589043 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5kz89" podUID="e730d89c-85b3-4a44-883b-d65f347261ec" Apr 16 16:48:00.589027 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:00.588990 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pgvd2" Apr 16 16:48:00.589499 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:00.588990 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk5gr" Apr 16 16:48:00.589499 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:00.589121 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pgvd2" podUID="98d7d9af-1c21-4f39-b605-21a11d06f2ae" Apr 16 16:48:00.589499 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:00.589264 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk5gr" podUID="412718e2-2075-4c7b-a6a9-3b94ed57564c" Apr 16 16:48:01.591695 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:01.591661 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kz89" Apr 16 16:48:01.592095 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:01.591780 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5kz89" podUID="e730d89c-85b3-4a44-883b-d65f347261ec" Apr 16 16:48:02.589533 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:02.589328 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pgvd2" Apr 16 16:48:02.589665 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:02.589328 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk5gr" Apr 16 16:48:02.589665 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:02.589634 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pgvd2" podUID="98d7d9af-1c21-4f39-b605-21a11d06f2ae" Apr 16 16:48:02.589750 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:02.589693 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk5gr" podUID="412718e2-2075-4c7b-a6a9-3b94ed57564c" Apr 16 16:48:02.813420 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:02.813385 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" event={"ID":"145fe4b0-c23a-49dc-abe3-212e53cff730","Type":"ContainerStarted","Data":"d5d0627e5a7d6dedb3c9c06c59fd4ce5895cb0dbc5f601252339fe2a761ad744"} Apr 16 16:48:02.817128 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:02.817092 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" event={"ID":"c2fec5cf-333c-4342-bd77-642bb2581568","Type":"ContainerStarted","Data":"098e7fb0ce278676d58d273f4f4bca9fa72e541db99f15ccbfe277ddcffd33be"} Apr 16 16:48:02.817311 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:02.817139 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" event={"ID":"c2fec5cf-333c-4342-bd77-642bb2581568","Type":"ContainerStarted","Data":"73edcd7482de5fad0bc1e7c4325e1d71e2624374c5b15246a2dc9edb62eaeedd"} Apr 16 16:48:02.817311 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:02.817154 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" event={"ID":"c2fec5cf-333c-4342-bd77-642bb2581568","Type":"ContainerStarted","Data":"a98f6689bc04a0779984aefaf3c3d60839f0937276f21366ae8f839ee702dab7"} Apr 16 16:48:02.817311 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:02.817169 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" event={"ID":"c2fec5cf-333c-4342-bd77-642bb2581568","Type":"ContainerStarted","Data":"250b4734454ab30ec14e398e3074d3afc1c26801c530f132ee58758290bc1008"} Apr 16 16:48:02.817311 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:02.817183 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" event={"ID":"c2fec5cf-333c-4342-bd77-642bb2581568","Type":"ContainerStarted","Data":"52fbdf4ebb49ce8f4d37d353a60ec899c18920388cfb1b6dc02c61702eb24f37"} Apr 16 16:48:02.817311 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:02.817196 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" event={"ID":"c2fec5cf-333c-4342-bd77-642bb2581568","Type":"ContainerStarted","Data":"6384debe57225d8afc049939f2c1ed92f17b07a83cde0dad73fbbb799e9fc35c"} Apr 16 16:48:02.820214 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:02.820174 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ndl5w" event={"ID":"8e078538-0666-4e1e-8b80-2f09583fbd00","Type":"ContainerStarted","Data":"39797102d368551e2d09b9d0b1ac5dede11cc35a760cd8854c8d98836fd8b567"} Apr 16 16:48:02.821850 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:02.821806 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-37.ec2.internal" event={"ID":"8b1f1c157df36eece23646e8d91601bf","Type":"ContainerStarted","Data":"4a1e2902556e7bd5e7a89fea57a3f797e6fef06dfe64c4b5915a98d1076f198d"} Apr 16 16:48:02.832653 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:02.832598 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vh7lw" podStartSLOduration=2.924834453 podStartE2EDuration="21.83258345s" podCreationTimestamp="2026-04-16 16:47:41 +0000 UTC" firstStartedPulling="2026-04-16 16:47:42.85344749 +0000 UTC m=+1.914597000" lastFinishedPulling="2026-04-16 16:48:01.761196482 +0000 UTC m=+20.822345997" observedRunningTime="2026-04-16 16:48:02.832169813 +0000 UTC m=+21.893319347" watchObservedRunningTime="2026-04-16 16:48:02.83258345 +0000 UTC m=+21.893732980" Apr 16 16:48:02.845634 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:02.845581 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-ndl5w" podStartSLOduration=2.858810105 podStartE2EDuration="21.845563839s" podCreationTimestamp="2026-04-16 16:47:41 +0000 UTC" firstStartedPulling="2026-04-16 16:47:42.785652512 +0000 UTC m=+1.846802022" lastFinishedPulling="2026-04-16 16:48:01.772406247 +0000 UTC m=+20.833555756" observedRunningTime="2026-04-16 16:48:02.845165103 +0000 UTC m=+21.906314634" watchObservedRunningTime="2026-04-16 16:48:02.845563839 +0000 UTC m=+21.906713371" Apr 16 16:48:02.857006 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:02.856955 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-37.ec2.internal" podStartSLOduration=20.856922926 podStartE2EDuration="20.856922926s" podCreationTimestamp="2026-04-16 16:47:42 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:48:02.856505368 +0000 UTC m=+21.917654899" watchObservedRunningTime="2026-04-16 16:48:02.856922926 +0000 UTC m=+21.918072457" Apr 16 16:48:03.588434 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:03.588397 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kz89" Apr 16 16:48:03.588672 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:03.588533 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5kz89" podUID="e730d89c-85b3-4a44-883b-d65f347261ec" Apr 16 16:48:03.825741 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:03.825694 2576 generic.go:358] "Generic (PLEG): container finished" podID="20d62408-3657-49b5-a83b-39331108c8d9" containerID="eed22810fb78fa061bd76aaea0a97511d70157cb6bc77da0168da9bde8733820" exitCode=0 Apr 16 16:48:03.826084 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:03.825830 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8zhpq" event={"ID":"20d62408-3657-49b5-a83b-39331108c8d9","Type":"ContainerDied","Data":"eed22810fb78fa061bd76aaea0a97511d70157cb6bc77da0168da9bde8733820"} Apr 16 16:48:03.827563 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:03.827526 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9c8bl" event={"ID":"643eef8b-e0e5-48fb-a732-6d880b6b62e7","Type":"ContainerStarted","Data":"dc4fe94670fb3145477c11dea6edb52c2d2a8e00124b7f62077851a54f44140c"} Apr 16 16:48:03.828906 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:03.828860 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q6fvl" event={"ID":"fce43675-3aee-453f-b5ad-a28fba84c810","Type":"ContainerStarted","Data":"c5b738f9ec755d62b51ca90493a928b5406b553126e667ec915188d6d34e074f"} Apr 16 16:48:03.830330 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:03.830291 2576 generic.go:358] "Generic (PLEG): container finished" podID="0f4d89b70f22123800295882f9c30018" containerID="9e3e14120a31533584a03dc721750865a3387fcef47c27c061a9ad4b3ffa17c5" exitCode=0 Apr 16 16:48:03.830427 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:03.830377 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-37.ec2.internal" event={"ID":"0f4d89b70f22123800295882f9c30018","Type":"ContainerDied","Data":"9e3e14120a31533584a03dc721750865a3387fcef47c27c061a9ad4b3ffa17c5"} Apr 16 16:48:03.831724 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:03.831703 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-42cdh" event={"ID":"99da12c7-93e3-4848-afc5-36f2cbf5645d","Type":"ContainerStarted","Data":"4783f32864a54776d1c5f47bb289b5cfaa1e65501354677079ada39dc0b5c008"} Apr 16 16:48:03.834359 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:03.833990 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9vmmq" event={"ID":"2fef7f9f-e604-47d1-877a-3dfcbbdebba0","Type":"ContainerStarted","Data":"48fcf8b50ee96a1126c72738b0c4c247215784680cfc4a3374c48f9c6484c282"} Apr 16 16:48:03.837165 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:03.837147 2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Apr 16 16:48:03.837305 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:03.837188 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kwz5r" event={"ID":"f6b12331-2022-4c05-ac42-753274b499dc","Type":"ContainerStarted","Data":"1112d77f136092f43fa2023e3521f86d84b211a021b0ede12c3047fb58a37839"} Apr 16 16:48:03.908583 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:03.908535 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-q6fvl" podStartSLOduration=3.940867162 podStartE2EDuration="22.908519125s" podCreationTimestamp="2026-04-16 16:47:41 +0000 UTC" firstStartedPulling="2026-04-16 16:47:42.779451569 +0000 UTC m=+1.840601077" lastFinishedPulling="2026-04-16 16:48:01.74710353 +0000 UTC m=+20.808253040" observedRunningTime="2026-04-16 16:48:03.908319596 +0000 UTC m=+22.969469127" watchObservedRunningTime="2026-04-16 16:48:03.908519125 +0000 UTC m=+22.969668655" Apr 16 16:48:03.929522 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:03.929481 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9vmmq" podStartSLOduration=3.985962864 podStartE2EDuration="22.929467368s" podCreationTimestamp="2026-04-16 16:47:41 +0000 UTC" firstStartedPulling="2026-04-16 16:47:42.803114271 +0000 UTC m=+1.864263781" lastFinishedPulling="2026-04-16 16:48:01.746618776 +0000 UTC m=+20.807768285" observedRunningTime="2026-04-16 16:48:03.929099773 +0000 UTC m=+22.990249307" watchObservedRunningTime="2026-04-16 16:48:03.929467368 +0000 UTC m=+22.990616897" Apr 16 16:48:03.982070 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:03.982023 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-42cdh" podStartSLOduration=4.101483932 podStartE2EDuration="22.982011301s" podCreationTimestamp="2026-04-16 16:47:41 +0000 UTC" firstStartedPulling="2026-04-16 16:47:42.884891817 +0000 UTC m=+1.946041326" lastFinishedPulling="2026-04-16 16:48:01.765419185 +0000 UTC m=+20.826568695" observedRunningTime="2026-04-16 16:48:03.981859492 +0000 UTC m=+23.043009023" watchObservedRunningTime="2026-04-16 16:48:03.982011301 +0000 UTC m=+23.043160832" Apr 16 16:48:03.982239 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:03.982176 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kwz5r" podStartSLOduration=3.999818838 podStartE2EDuration="22.982170536s" podCreationTimestamp="2026-04-16 16:47:41 +0000 UTC" firstStartedPulling="2026-04-16 16:47:42.764743031 +0000 UTC m=+1.825892539" lastFinishedPulling="2026-04-16 16:48:01.747094729 +0000 UTC m=+20.808244237" observedRunningTime="2026-04-16 16:48:03.955426486 +0000 UTC m=+23.016576016" watchObservedRunningTime="2026-04-16 16:48:03.982170536 +0000 UTC m=+23.043320067" Apr 16 16:48:04.512756 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:04.512630 2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-04-16T16:48:03.837161099Z","UUID":"b0111df8-0f08-4350-bb5a-00e87d48711b","Handler":null,"Name":"","Endpoint":""} Apr 16 16:48:04.514944 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:04.514514 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Apr 16 16:48:04.514944 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:04.514543 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Apr 16 16:48:04.588566 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:04.588500 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pgvd2" Apr 16 16:48:04.588765 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:04.588625 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pgvd2" podUID="98d7d9af-1c21-4f39-b605-21a11d06f2ae" Apr 16 16:48:04.589055 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:04.589040 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk5gr" Apr 16 16:48:04.589155 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:04.589138 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk5gr" podUID="412718e2-2075-4c7b-a6a9-3b94ed57564c" Apr 16 16:48:04.840832 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:04.840793 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-37.ec2.internal" event={"ID":"0f4d89b70f22123800295882f9c30018","Type":"ContainerStarted","Data":"9e37bb5bae39b5387df986b116ac65786a343cee434669c613a180b5dc209331"} Apr 16 16:48:04.843131 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:04.843091 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9c8bl" event={"ID":"643eef8b-e0e5-48fb-a732-6d880b6b62e7","Type":"ContainerStarted","Data":"b0c8558904983c03806b03834afeade634f38e8e62c6b6c07aadba6903e1ed0c"} Apr 16 16:48:04.847525 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:04.847495 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" event={"ID":"c2fec5cf-333c-4342-bd77-642bb2581568","Type":"ContainerStarted","Data":"fb114e126f445d40c67a0212d123b6475b1c7af836f2cebfac5e762a2d0618c6"} Apr 16 16:48:04.857625 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:04.857550 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-37.ec2.internal" podStartSLOduration=22.857533754 podStartE2EDuration="22.857533754s" podCreationTimestamp="2026-04-16 16:47:42 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:48:04.857098526 +0000 UTC m=+23.918248060" watchObservedRunningTime="2026-04-16 16:48:04.857533754 +0000 UTC m=+23.918683287" Apr 16 16:48:05.408584 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:05.408394 2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kwz5r" Apr 16 16:48:05.409089 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:05.409072 2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kwz5r" Apr 16 16:48:05.591564 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:05.591538 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kz89" Apr 16 16:48:05.591711 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:05.591630 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5kz89" podUID="e730d89c-85b3-4a44-883b-d65f347261ec" Apr 16 16:48:05.851932 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:05.851891 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9c8bl" event={"ID":"643eef8b-e0e5-48fb-a732-6d880b6b62e7","Type":"ContainerStarted","Data":"9b3792236da81aac80c005644504d3204c16075ff3a6b8b1954c213ed9abcb14"} Apr 16 16:48:05.852438 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:05.852377 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kwz5r" Apr 16 16:48:05.852937 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:05.852916 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kwz5r" Apr 16 16:48:05.869092 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:05.869030 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9c8bl" podStartSLOduration=2.9777903329999997 podStartE2EDuration="24.869011912s" podCreationTimestamp="2026-04-16 16:47:41 +0000 UTC" firstStartedPulling="2026-04-16 16:47:42.836674627 +0000 UTC m=+1.897824136" lastFinishedPulling="2026-04-16 16:48:04.727896199 +0000 UTC m=+23.789045715" observedRunningTime="2026-04-16 16:48:05.868479928 +0000 UTC m=+24.929629459" watchObservedRunningTime="2026-04-16 16:48:05.869011912 +0000 UTC m=+24.930161444" Apr 16 16:48:06.588107 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:06.588062 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk5gr" Apr 16 16:48:06.588365 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:06.588060 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pgvd2" Apr 16 16:48:06.588365 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:06.588218 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk5gr" podUID="412718e2-2075-4c7b-a6a9-3b94ed57564c" Apr 16 16:48:06.588365 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:06.588285 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pgvd2" podUID="98d7d9af-1c21-4f39-b605-21a11d06f2ae" Apr 16 16:48:07.589083 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:07.589032 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kz89" Apr 16 16:48:07.589759 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:07.589167 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5kz89" podUID="e730d89c-85b3-4a44-883b-d65f347261ec" Apr 16 16:48:08.588975 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:08.588939 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pgvd2" Apr 16 16:48:08.588975 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:08.588939 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk5gr" Apr 16 16:48:08.589567 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:08.589058 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pgvd2" podUID="98d7d9af-1c21-4f39-b605-21a11d06f2ae" Apr 16 16:48:08.589567 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:08.589181 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk5gr" podUID="412718e2-2075-4c7b-a6a9-3b94ed57564c" Apr 16 16:48:08.858852 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:08.858820 2576 generic.go:358] "Generic (PLEG): container finished" podID="20d62408-3657-49b5-a83b-39331108c8d9" containerID="f384e568f61f7894623212a57900e5d976e25a8306b594f01f141bae6a6709b8" exitCode=0 Apr 16 16:48:08.859010 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:08.858912 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8zhpq" event={"ID":"20d62408-3657-49b5-a83b-39331108c8d9","Type":"ContainerDied","Data":"f384e568f61f7894623212a57900e5d976e25a8306b594f01f141bae6a6709b8"} Apr 16 16:48:08.862212 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:08.862179 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" event={"ID":"c2fec5cf-333c-4342-bd77-642bb2581568","Type":"ContainerStarted","Data":"0db0a0aeb2b4cbd1e9eef9a04100c95da647dc6d463478342b40c27b36f06622"} Apr 16 16:48:08.862577 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:08.862554 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:48:08.877699 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:08.877677 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:48:08.900124 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:08.900052 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" podStartSLOduration=8.452777477 podStartE2EDuration="27.900040925s" podCreationTimestamp="2026-04-16 16:47:41 +0000 UTC" firstStartedPulling="2026-04-16 16:47:42.827405971 +0000 UTC m=+1.888555481" lastFinishedPulling="2026-04-16 16:48:02.274669406 +0000 UTC m=+21.335818929" observedRunningTime="2026-04-16 16:48:08.899992696 +0000 UTC m=+27.961142228" watchObservedRunningTime="2026-04-16 16:48:08.900040925 +0000 UTC m=+27.961190456" Apr 16 16:48:09.589105 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:09.589077 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kz89" Apr 16 16:48:09.589806 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:09.589186 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5kz89" podUID="e730d89c-85b3-4a44-883b-d65f347261ec" Apr 16 16:48:09.866172 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:09.866091 2576 generic.go:358] "Generic (PLEG): container finished" podID="20d62408-3657-49b5-a83b-39331108c8d9" containerID="1b873cc1a0bda97f76586f5fca11f6fca0677bc446da29bc3297022f51458c92" exitCode=0 Apr 16 16:48:09.866354 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:09.866168 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8zhpq" event={"ID":"20d62408-3657-49b5-a83b-39331108c8d9","Type":"ContainerDied","Data":"1b873cc1a0bda97f76586f5fca11f6fca0677bc446da29bc3297022f51458c92"} Apr 16 16:48:09.866882 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:09.866758 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:48:09.866882 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:09.866788 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:48:09.884501 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:09.884478 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:48:10.096630 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:10.096596 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pgvd2"] Apr 16 16:48:10.096789 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:10.096720 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pgvd2" Apr 16 16:48:10.096837 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:10.096798 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pgvd2" podUID="98d7d9af-1c21-4f39-b605-21a11d06f2ae" Apr 16 16:48:10.100230 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:10.100142 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5kz89"] Apr 16 16:48:10.100366 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:10.100325 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kz89" Apr 16 16:48:10.100441 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:10.100419 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5kz89" podUID="e730d89c-85b3-4a44-883b-d65f347261ec" Apr 16 16:48:10.100763 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:10.100747 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hk5gr"] Apr 16 16:48:10.100845 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:10.100833 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk5gr" Apr 16 16:48:10.100921 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:10.100904 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk5gr" podUID="412718e2-2075-4c7b-a6a9-3b94ed57564c" Apr 16 16:48:10.869963 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:10.869921 2576 generic.go:358] "Generic (PLEG): container finished" podID="20d62408-3657-49b5-a83b-39331108c8d9" containerID="56d49b54bf8808db66040dc6866c1b2e3ec294fa4e20afbde6f22fca293f1b00" exitCode=0 Apr 16 16:48:10.870368 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:10.870059 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8zhpq" event={"ID":"20d62408-3657-49b5-a83b-39331108c8d9","Type":"ContainerDied","Data":"56d49b54bf8808db66040dc6866c1b2e3ec294fa4e20afbde6f22fca293f1b00"} Apr 16 16:48:11.590103 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:11.590071 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pgvd2" Apr 16 16:48:11.590331 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:11.590155 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pgvd2" podUID="98d7d9af-1c21-4f39-b605-21a11d06f2ae" Apr 16 16:48:11.590331 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:11.590314 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk5gr" Apr 16 16:48:11.590432 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:11.590401 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kz89" Apr 16 16:48:11.590485 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:11.590467 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5kz89" podUID="e730d89c-85b3-4a44-883b-d65f347261ec" Apr 16 16:48:11.590536 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:11.590484 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk5gr" podUID="412718e2-2075-4c7b-a6a9-3b94ed57564c" Apr 16 16:48:13.588416 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:13.588302 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk5gr" Apr 16 16:48:13.589119 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:13.588302 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kz89" Apr 16 16:48:13.589119 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:13.588302 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pgvd2" Apr 16 16:48:13.589792 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:13.589369 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk5gr" podUID="412718e2-2075-4c7b-a6a9-3b94ed57564c" Apr 16 16:48:13.589792 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:13.589487 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5kz89" podUID="e730d89c-85b3-4a44-883b-d65f347261ec" Apr 16 16:48:13.589792 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:13.589570 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pgvd2" podUID="98d7d9af-1c21-4f39-b605-21a11d06f2ae" Apr 16 16:48:14.193582 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:14.193540 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/412718e2-2075-4c7b-a6a9-3b94ed57564c-metrics-certs\") pod \"network-metrics-daemon-hk5gr\" (UID: \"412718e2-2075-4c7b-a6a9-3b94ed57564c\") " pod="openshift-multus/network-metrics-daemon-hk5gr" Apr 16 16:48:14.193747 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:14.193592 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/98d7d9af-1c21-4f39-b605-21a11d06f2ae-original-pull-secret\") pod \"global-pull-secret-syncer-pgvd2\" (UID: \"98d7d9af-1c21-4f39-b605-21a11d06f2ae\") " pod="kube-system/global-pull-secret-syncer-pgvd2" Apr 16 16:48:14.193747 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:14.193717 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Apr 16 16:48:14.193747 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:14.193736 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 16:48:14.193862 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:14.193771 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98d7d9af-1c21-4f39-b605-21a11d06f2ae-original-pull-secret podName:98d7d9af-1c21-4f39-b605-21a11d06f2ae nodeName:}" failed. No retries permitted until 2026-04-16 16:48:46.193758383 +0000 UTC m=+65.254907895 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/98d7d9af-1c21-4f39-b605-21a11d06f2ae-original-pull-secret") pod "global-pull-secret-syncer-pgvd2" (UID: "98d7d9af-1c21-4f39-b605-21a11d06f2ae") : object "kube-system"/"original-pull-secret" not registered Apr 16 16:48:14.193862 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:14.193790 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/412718e2-2075-4c7b-a6a9-3b94ed57564c-metrics-certs podName:412718e2-2075-4c7b-a6a9-3b94ed57564c nodeName:}" failed. No retries permitted until 2026-04-16 16:48:46.193777026 +0000 UTC m=+65.254926535 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/412718e2-2075-4c7b-a6a9-3b94ed57564c-metrics-certs") pod "network-metrics-daemon-hk5gr" (UID: "412718e2-2075-4c7b-a6a9-3b94ed57564c") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 16:48:14.395259 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:14.395196 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gtgpj\" (UniqueName: \"kubernetes.io/projected/e730d89c-85b3-4a44-883b-d65f347261ec-kube-api-access-gtgpj\") pod \"network-check-target-5kz89\" (UID: \"e730d89c-85b3-4a44-883b-d65f347261ec\") " pod="openshift-network-diagnostics/network-check-target-5kz89" Apr 16 16:48:14.395437 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:14.395392 2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 16 16:48:14.395437 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:14.395421 2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 16 16:48:14.395437 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:14.395435 2576 projected.go:194] Error preparing data for projected volume kube-api-access-gtgpj for pod openshift-network-diagnostics/network-check-target-5kz89: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 16:48:14.395585 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:14.395503 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e730d89c-85b3-4a44-883b-d65f347261ec-kube-api-access-gtgpj podName:e730d89c-85b3-4a44-883b-d65f347261ec nodeName:}" failed. No retries permitted until 2026-04-16 16:48:46.395483865 +0000 UTC m=+65.456633379 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gtgpj" (UniqueName: "kubernetes.io/projected/e730d89c-85b3-4a44-883b-d65f347261ec-kube-api-access-gtgpj") pod "network-check-target-5kz89" (UID: "e730d89c-85b3-4a44-883b-d65f347261ec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 16:48:15.588749 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:15.588710 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kz89" Apr 16 16:48:15.589167 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:15.588710 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pgvd2" Apr 16 16:48:15.589167 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:15.588710 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk5gr" Apr 16 16:48:15.589604 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:15.589573 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5kz89" podUID="e730d89c-85b3-4a44-883b-d65f347261ec" Apr 16 16:48:15.589739 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:15.589642 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk5gr" podUID="412718e2-2075-4c7b-a6a9-3b94ed57564c" Apr 16 16:48:15.589739 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:15.589314 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pgvd2" podUID="98d7d9af-1c21-4f39-b605-21a11d06f2ae" Apr 16 16:48:15.757199 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:15.757167 2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-37.ec2.internal" event="NodeReady" Apr 16 16:48:15.757391 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:15.757354 2576 kubelet_node_status.go:550] "Fast updating node status as it just became ready" Apr 16 16:48:15.890450 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:15.890339 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rzx7h"] Apr 16 16:48:15.907631 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:15.907589 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xfnx6"] Apr 16 16:48:15.907801 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:15.907761 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rzx7h" Apr 16 16:48:15.910378 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:15.910353 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Apr 16 16:48:15.910497 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:15.910417 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-99nwp\"" Apr 16 16:48:15.910659 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:15.910640 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Apr 16 16:48:15.937305 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:15.937272 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rzx7h"] Apr 16 16:48:15.937305 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:15.937308 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xfnx6"] Apr 16 16:48:15.937532 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:15.937450 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xfnx6" Apr 16 16:48:15.939766 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:15.939744 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Apr 16 16:48:15.939901 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:15.939862 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-c25vw\"" Apr 16 16:48:15.940112 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:15.940093 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Apr 16 16:48:15.940404 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:15.940385 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Apr 16 16:48:16.008102 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:16.008054 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p9pn\" (UniqueName: \"kubernetes.io/projected/efee44e6-2e2b-47f0-b356-6e77267f19e0-kube-api-access-7p9pn\") pod \"ingress-canary-xfnx6\" (UID: \"efee44e6-2e2b-47f0-b356-6e77267f19e0\") " pod="openshift-ingress-canary/ingress-canary-xfnx6" Apr 16 16:48:16.008296 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:16.008117 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4e2d35c-1627-4f9c-bea6-985f08319902-tmp-dir\") pod \"dns-default-rzx7h\" (UID: \"b4e2d35c-1627-4f9c-bea6-985f08319902\") " pod="openshift-dns/dns-default-rzx7h" Apr 16 16:48:16.008296 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:16.008150 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4e2d35c-1627-4f9c-bea6-985f08319902-metrics-tls\") pod \"dns-default-rzx7h\" (UID: \"b4e2d35c-1627-4f9c-bea6-985f08319902\") " pod="openshift-dns/dns-default-rzx7h" Apr 16 16:48:16.008296 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:16.008186 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/efee44e6-2e2b-47f0-b356-6e77267f19e0-cert\") pod \"ingress-canary-xfnx6\" (UID: \"efee44e6-2e2b-47f0-b356-6e77267f19e0\") " pod="openshift-ingress-canary/ingress-canary-xfnx6" Apr 16 16:48:16.008296 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:16.008286 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b4e2d35c-1627-4f9c-bea6-985f08319902-config-volume\") pod \"dns-default-rzx7h\" (UID: \"b4e2d35c-1627-4f9c-bea6-985f08319902\") " pod="openshift-dns/dns-default-rzx7h" Apr 16 16:48:16.008488 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:16.008380 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hczn\" (UniqueName: \"kubernetes.io/projected/b4e2d35c-1627-4f9c-bea6-985f08319902-kube-api-access-2hczn\") pod \"dns-default-rzx7h\" (UID: \"b4e2d35c-1627-4f9c-bea6-985f08319902\") " pod="openshift-dns/dns-default-rzx7h" Apr 16 16:48:16.108923 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:16.108879 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4e2d35c-1627-4f9c-bea6-985f08319902-tmp-dir\") pod \"dns-default-rzx7h\" (UID: \"b4e2d35c-1627-4f9c-bea6-985f08319902\") " pod="openshift-dns/dns-default-rzx7h" Apr 16 16:48:16.108923 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:16.108928 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4e2d35c-1627-4f9c-bea6-985f08319902-metrics-tls\") pod \"dns-default-rzx7h\" (UID: \"b4e2d35c-1627-4f9c-bea6-985f08319902\") " pod="openshift-dns/dns-default-rzx7h" Apr 16 16:48:16.109143 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:16.108954 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/efee44e6-2e2b-47f0-b356-6e77267f19e0-cert\") pod \"ingress-canary-xfnx6\" (UID: \"efee44e6-2e2b-47f0-b356-6e77267f19e0\") " pod="openshift-ingress-canary/ingress-canary-xfnx6" Apr 16 16:48:16.109143 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:16.108984 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b4e2d35c-1627-4f9c-bea6-985f08319902-config-volume\") pod \"dns-default-rzx7h\" (UID: \"b4e2d35c-1627-4f9c-bea6-985f08319902\") " pod="openshift-dns/dns-default-rzx7h" Apr 16 16:48:16.109143 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:16.109030 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2hczn\" (UniqueName: \"kubernetes.io/projected/b4e2d35c-1627-4f9c-bea6-985f08319902-kube-api-access-2hczn\") pod \"dns-default-rzx7h\" (UID: \"b4e2d35c-1627-4f9c-bea6-985f08319902\") " pod="openshift-dns/dns-default-rzx7h" Apr 16 16:48:16.109143 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:16.109063 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7p9pn\" (UniqueName: \"kubernetes.io/projected/efee44e6-2e2b-47f0-b356-6e77267f19e0-kube-api-access-7p9pn\") pod \"ingress-canary-xfnx6\" (UID: \"efee44e6-2e2b-47f0-b356-6e77267f19e0\") " pod="openshift-ingress-canary/ingress-canary-xfnx6" Apr 16 16:48:16.109143 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:16.109106 2576 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 16 16:48:16.109143 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:16.109109 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 16 16:48:16.109364 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:16.109179 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efee44e6-2e2b-47f0-b356-6e77267f19e0-cert podName:efee44e6-2e2b-47f0-b356-6e77267f19e0 nodeName:}" failed. No retries permitted until 2026-04-16 16:48:16.609161822 +0000 UTC m=+35.670311342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/efee44e6-2e2b-47f0-b356-6e77267f19e0-cert") pod "ingress-canary-xfnx6" (UID: "efee44e6-2e2b-47f0-b356-6e77267f19e0") : secret "canary-serving-cert" not found Apr 16 16:48:16.109364 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:16.109198 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4e2d35c-1627-4f9c-bea6-985f08319902-metrics-tls podName:b4e2d35c-1627-4f9c-bea6-985f08319902 nodeName:}" failed. No retries permitted until 2026-04-16 16:48:16.609188919 +0000 UTC m=+35.670338427 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4e2d35c-1627-4f9c-bea6-985f08319902-metrics-tls") pod "dns-default-rzx7h" (UID: "b4e2d35c-1627-4f9c-bea6-985f08319902") : secret "dns-default-metrics-tls" not found Apr 16 16:48:16.109364 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:16.109331 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4e2d35c-1627-4f9c-bea6-985f08319902-tmp-dir\") pod \"dns-default-rzx7h\" (UID: \"b4e2d35c-1627-4f9c-bea6-985f08319902\") " pod="openshift-dns/dns-default-rzx7h" Apr 16 16:48:16.109631 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:16.109613 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b4e2d35c-1627-4f9c-bea6-985f08319902-config-volume\") pod \"dns-default-rzx7h\" (UID: \"b4e2d35c-1627-4f9c-bea6-985f08319902\") " pod="openshift-dns/dns-default-rzx7h" Apr 16 16:48:16.122199 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:16.122167 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hczn\" (UniqueName: \"kubernetes.io/projected/b4e2d35c-1627-4f9c-bea6-985f08319902-kube-api-access-2hczn\") pod \"dns-default-rzx7h\" (UID: \"b4e2d35c-1627-4f9c-bea6-985f08319902\") " pod="openshift-dns/dns-default-rzx7h" Apr 16 16:48:16.122332 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:16.122276 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p9pn\" (UniqueName: \"kubernetes.io/projected/efee44e6-2e2b-47f0-b356-6e77267f19e0-kube-api-access-7p9pn\") pod \"ingress-canary-xfnx6\" (UID: \"efee44e6-2e2b-47f0-b356-6e77267f19e0\") " pod="openshift-ingress-canary/ingress-canary-xfnx6" Apr 16 16:48:16.613964 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:16.613928 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/efee44e6-2e2b-47f0-b356-6e77267f19e0-cert\") pod \"ingress-canary-xfnx6\" (UID: \"efee44e6-2e2b-47f0-b356-6e77267f19e0\") " pod="openshift-ingress-canary/ingress-canary-xfnx6" Apr 16 16:48:16.614472 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:16.614010 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4e2d35c-1627-4f9c-bea6-985f08319902-metrics-tls\") pod \"dns-default-rzx7h\" (UID: \"b4e2d35c-1627-4f9c-bea6-985f08319902\") " pod="openshift-dns/dns-default-rzx7h" Apr 16 16:48:16.614472 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:16.614082 2576 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 16 16:48:16.614472 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:16.614089 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 16 16:48:16.614472 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:16.614145 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4e2d35c-1627-4f9c-bea6-985f08319902-metrics-tls podName:b4e2d35c-1627-4f9c-bea6-985f08319902 nodeName:}" failed. No retries permitted until 2026-04-16 16:48:17.614131541 +0000 UTC m=+36.675281050 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4e2d35c-1627-4f9c-bea6-985f08319902-metrics-tls") pod "dns-default-rzx7h" (UID: "b4e2d35c-1627-4f9c-bea6-985f08319902") : secret "dns-default-metrics-tls" not found Apr 16 16:48:16.614472 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:16.614158 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efee44e6-2e2b-47f0-b356-6e77267f19e0-cert podName:efee44e6-2e2b-47f0-b356-6e77267f19e0 nodeName:}" failed. No retries permitted until 2026-04-16 16:48:17.614152267 +0000 UTC m=+36.675301776 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/efee44e6-2e2b-47f0-b356-6e77267f19e0-cert") pod "ingress-canary-xfnx6" (UID: "efee44e6-2e2b-47f0-b356-6e77267f19e0") : secret "canary-serving-cert" not found Apr 16 16:48:17.588951 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:17.588774 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kz89" Apr 16 16:48:17.589122 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:17.588774 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk5gr" Apr 16 16:48:17.589189 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:17.588834 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pgvd2" Apr 16 16:48:17.590985 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:17.590967 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Apr 16 16:48:17.591104 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:17.591045 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Apr 16 16:48:17.591628 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:17.591608 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-877l2\"" Apr 16 16:48:17.591788 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:17.591627 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Apr 16 16:48:17.591788 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:17.591608 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wrbkk\"" Apr 16 16:48:17.591788 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:17.591642 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Apr 16 16:48:17.620470 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:17.620443 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4e2d35c-1627-4f9c-bea6-985f08319902-metrics-tls\") pod \"dns-default-rzx7h\" (UID: \"b4e2d35c-1627-4f9c-bea6-985f08319902\") " pod="openshift-dns/dns-default-rzx7h" Apr 16 16:48:17.620761 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:17.620488 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/efee44e6-2e2b-47f0-b356-6e77267f19e0-cert\") pod \"ingress-canary-xfnx6\" (UID: \"efee44e6-2e2b-47f0-b356-6e77267f19e0\") " pod="openshift-ingress-canary/ingress-canary-xfnx6" Apr 16 16:48:17.620761 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:17.620569 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 16 16:48:17.620761 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:17.620584 2576 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 16 16:48:17.620761 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:17.620622 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4e2d35c-1627-4f9c-bea6-985f08319902-metrics-tls podName:b4e2d35c-1627-4f9c-bea6-985f08319902 nodeName:}" failed. No retries permitted until 2026-04-16 16:48:19.620607485 +0000 UTC m=+38.681756998 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4e2d35c-1627-4f9c-bea6-985f08319902-metrics-tls") pod "dns-default-rzx7h" (UID: "b4e2d35c-1627-4f9c-bea6-985f08319902") : secret "dns-default-metrics-tls" not found Apr 16 16:48:17.620761 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:17.620673 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efee44e6-2e2b-47f0-b356-6e77267f19e0-cert podName:efee44e6-2e2b-47f0-b356-6e77267f19e0 nodeName:}" failed. No retries permitted until 2026-04-16 16:48:19.620667368 +0000 UTC m=+38.681816876 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/efee44e6-2e2b-47f0-b356-6e77267f19e0-cert") pod "ingress-canary-xfnx6" (UID: "efee44e6-2e2b-47f0-b356-6e77267f19e0") : secret "canary-serving-cert" not found Apr 16 16:48:17.886290 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:17.886199 2576 generic.go:358] "Generic (PLEG): container finished" podID="20d62408-3657-49b5-a83b-39331108c8d9" containerID="8d07f081cb6bbbe2867ad897d3adbe570e8dd4ffdaaf9f4845767139b8500a6b" exitCode=0 Apr 16 16:48:17.886290 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:17.886276 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8zhpq" event={"ID":"20d62408-3657-49b5-a83b-39331108c8d9","Type":"ContainerDied","Data":"8d07f081cb6bbbe2867ad897d3adbe570e8dd4ffdaaf9f4845767139b8500a6b"} Apr 16 16:48:18.890449 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:18.890419 2576 generic.go:358] "Generic (PLEG): container finished" podID="20d62408-3657-49b5-a83b-39331108c8d9" containerID="85bd65a12c40b88df9eb903a22a65c42dde4c89e1094c834755b809bd7aceab8" exitCode=0 Apr 16 16:48:18.890816 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:18.890480 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8zhpq" event={"ID":"20d62408-3657-49b5-a83b-39331108c8d9","Type":"ContainerDied","Data":"85bd65a12c40b88df9eb903a22a65c42dde4c89e1094c834755b809bd7aceab8"} Apr 16 16:48:19.637045 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:19.637008 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4e2d35c-1627-4f9c-bea6-985f08319902-metrics-tls\") pod \"dns-default-rzx7h\" (UID: \"b4e2d35c-1627-4f9c-bea6-985f08319902\") " pod="openshift-dns/dns-default-rzx7h" Apr 16 16:48:19.637230 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:19.637058 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/efee44e6-2e2b-47f0-b356-6e77267f19e0-cert\") pod \"ingress-canary-xfnx6\" (UID: \"efee44e6-2e2b-47f0-b356-6e77267f19e0\") " pod="openshift-ingress-canary/ingress-canary-xfnx6" Apr 16 16:48:19.637230 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:19.637152 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 16 16:48:19.637230 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:19.637227 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4e2d35c-1627-4f9c-bea6-985f08319902-metrics-tls podName:b4e2d35c-1627-4f9c-bea6-985f08319902 nodeName:}" failed. No retries permitted until 2026-04-16 16:48:23.637205469 +0000 UTC m=+42.698354996 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4e2d35c-1627-4f9c-bea6-985f08319902-metrics-tls") pod "dns-default-rzx7h" (UID: "b4e2d35c-1627-4f9c-bea6-985f08319902") : secret "dns-default-metrics-tls" not found Apr 16 16:48:19.637673 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:19.637237 2576 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 16 16:48:19.637673 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:19.637310 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efee44e6-2e2b-47f0-b356-6e77267f19e0-cert podName:efee44e6-2e2b-47f0-b356-6e77267f19e0 nodeName:}" failed. No retries permitted until 2026-04-16 16:48:23.637296143 +0000 UTC m=+42.698445652 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/efee44e6-2e2b-47f0-b356-6e77267f19e0-cert") pod "ingress-canary-xfnx6" (UID: "efee44e6-2e2b-47f0-b356-6e77267f19e0") : secret "canary-serving-cert" not found Apr 16 16:48:19.895194 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:19.895113 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8zhpq" event={"ID":"20d62408-3657-49b5-a83b-39331108c8d9","Type":"ContainerStarted","Data":"2cba482f15ad89d5cce68e01418b92468acdc5181a3f6464a4771a87229d225b"} Apr 16 16:48:23.667202 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:23.667159 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4e2d35c-1627-4f9c-bea6-985f08319902-metrics-tls\") pod \"dns-default-rzx7h\" (UID: \"b4e2d35c-1627-4f9c-bea6-985f08319902\") " pod="openshift-dns/dns-default-rzx7h" Apr 16 16:48:23.667720 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:23.667215 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/efee44e6-2e2b-47f0-b356-6e77267f19e0-cert\") pod \"ingress-canary-xfnx6\" (UID: \"efee44e6-2e2b-47f0-b356-6e77267f19e0\") " pod="openshift-ingress-canary/ingress-canary-xfnx6" Apr 16 16:48:23.667720 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:23.667348 2576 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 16 16:48:23.667720 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:23.667411 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efee44e6-2e2b-47f0-b356-6e77267f19e0-cert podName:efee44e6-2e2b-47f0-b356-6e77267f19e0 nodeName:}" failed. No retries permitted until 2026-04-16 16:48:31.667393073 +0000 UTC m=+50.728542597 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/efee44e6-2e2b-47f0-b356-6e77267f19e0-cert") pod "ingress-canary-xfnx6" (UID: "efee44e6-2e2b-47f0-b356-6e77267f19e0") : secret "canary-serving-cert" not found Apr 16 16:48:23.667720 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:23.667433 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 16 16:48:23.667720 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:23.667490 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4e2d35c-1627-4f9c-bea6-985f08319902-metrics-tls podName:b4e2d35c-1627-4f9c-bea6-985f08319902 nodeName:}" failed. No retries permitted until 2026-04-16 16:48:31.667475729 +0000 UTC m=+50.728625258 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4e2d35c-1627-4f9c-bea6-985f08319902-metrics-tls") pod "dns-default-rzx7h" (UID: "b4e2d35c-1627-4f9c-bea6-985f08319902") : secret "dns-default-metrics-tls" not found Apr 16 16:48:31.724131 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:31.724089 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4e2d35c-1627-4f9c-bea6-985f08319902-metrics-tls\") pod \"dns-default-rzx7h\" (UID: \"b4e2d35c-1627-4f9c-bea6-985f08319902\") " pod="openshift-dns/dns-default-rzx7h" Apr 16 16:48:31.724131 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:31.724138 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/efee44e6-2e2b-47f0-b356-6e77267f19e0-cert\") pod \"ingress-canary-xfnx6\" (UID: \"efee44e6-2e2b-47f0-b356-6e77267f19e0\") " pod="openshift-ingress-canary/ingress-canary-xfnx6" Apr 16 16:48:31.724594 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:31.724275 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 16 16:48:31.724594 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:31.724290 2576 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 16 16:48:31.724594 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:31.724340 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4e2d35c-1627-4f9c-bea6-985f08319902-metrics-tls podName:b4e2d35c-1627-4f9c-bea6-985f08319902 nodeName:}" failed. No retries permitted until 2026-04-16 16:48:47.724325734 +0000 UTC m=+66.785475243 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4e2d35c-1627-4f9c-bea6-985f08319902-metrics-tls") pod "dns-default-rzx7h" (UID: "b4e2d35c-1627-4f9c-bea6-985f08319902") : secret "dns-default-metrics-tls" not found Apr 16 16:48:31.724594 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:31.724354 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efee44e6-2e2b-47f0-b356-6e77267f19e0-cert podName:efee44e6-2e2b-47f0-b356-6e77267f19e0 nodeName:}" failed. No retries permitted until 2026-04-16 16:48:47.724348108 +0000 UTC m=+66.785497617 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/efee44e6-2e2b-47f0-b356-6e77267f19e0-cert") pod "ingress-canary-xfnx6" (UID: "efee44e6-2e2b-47f0-b356-6e77267f19e0") : secret "canary-serving-cert" not found Apr 16 16:48:41.883342 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:41.883309 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7c8c" Apr 16 16:48:41.910758 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:41.910650 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8zhpq" podStartSLOduration=26.961213798 podStartE2EDuration="1m0.910633664s" podCreationTimestamp="2026-04-16 16:47:41 +0000 UTC" firstStartedPulling="2026-04-16 16:47:42.889468889 +0000 UTC m=+1.950618398" lastFinishedPulling="2026-04-16 16:48:16.838888755 +0000 UTC m=+35.900038264" observedRunningTime="2026-04-16 16:48:19.939073038 +0000 UTC m=+39.000222570" watchObservedRunningTime="2026-04-16 16:48:41.910633664 +0000 UTC m=+60.971783196" Apr 16 16:48:46.223457 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:46.223320 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/412718e2-2075-4c7b-a6a9-3b94ed57564c-metrics-certs\") pod \"network-metrics-daemon-hk5gr\" (UID: \"412718e2-2075-4c7b-a6a9-3b94ed57564c\") " pod="openshift-multus/network-metrics-daemon-hk5gr" Apr 16 16:48:46.223457 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:46.223372 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/98d7d9af-1c21-4f39-b605-21a11d06f2ae-original-pull-secret\") pod \"global-pull-secret-syncer-pgvd2\" (UID: \"98d7d9af-1c21-4f39-b605-21a11d06f2ae\") " pod="kube-system/global-pull-secret-syncer-pgvd2" Apr 16 16:48:46.225720 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:46.225698 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Apr 16 16:48:46.225720 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:46.225714 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Apr 16 16:48:46.234216 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:46.234194 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Apr 16 16:48:46.234297 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:46.234281 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/412718e2-2075-4c7b-a6a9-3b94ed57564c-metrics-certs podName:412718e2-2075-4c7b-a6a9-3b94ed57564c nodeName:}" failed. No retries permitted until 2026-04-16 16:49:50.23424071 +0000 UTC m=+129.295390223 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/412718e2-2075-4c7b-a6a9-3b94ed57564c-metrics-certs") pod "network-metrics-daemon-hk5gr" (UID: "412718e2-2075-4c7b-a6a9-3b94ed57564c") : secret "metrics-daemon-secret" not found Apr 16 16:48:46.237533 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:46.237514 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/98d7d9af-1c21-4f39-b605-21a11d06f2ae-original-pull-secret\") pod \"global-pull-secret-syncer-pgvd2\" (UID: \"98d7d9af-1c21-4f39-b605-21a11d06f2ae\") " pod="kube-system/global-pull-secret-syncer-pgvd2" Apr 16 16:48:46.411084 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:46.411043 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pgvd2" Apr 16 16:48:46.424943 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:46.424908 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gtgpj\" (UniqueName: \"kubernetes.io/projected/e730d89c-85b3-4a44-883b-d65f347261ec-kube-api-access-gtgpj\") pod \"network-check-target-5kz89\" (UID: \"e730d89c-85b3-4a44-883b-d65f347261ec\") " pod="openshift-network-diagnostics/network-check-target-5kz89" Apr 16 16:48:46.427217 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:46.427197 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Apr 16 16:48:46.437895 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:46.437870 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Apr 16 16:48:46.448664 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:46.448636 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtgpj\" (UniqueName: \"kubernetes.io/projected/e730d89c-85b3-4a44-883b-d65f347261ec-kube-api-access-gtgpj\") pod \"network-check-target-5kz89\" (UID: \"e730d89c-85b3-4a44-883b-d65f347261ec\") " pod="openshift-network-diagnostics/network-check-target-5kz89" Apr 16 16:48:46.597983 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:46.597955 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pgvd2"] Apr 16 16:48:46.601516 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:48:46.601488 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98d7d9af_1c21_4f39_b605_21a11d06f2ae.slice/crio-71368e1031508652a2f1a1834d565e3e6b4cd6994199911e42fa554b0c78e24c WatchSource:0}: Error finding container 71368e1031508652a2f1a1834d565e3e6b4cd6994199911e42fa554b0c78e24c: Status 404 returned error can't find the container with id 71368e1031508652a2f1a1834d565e3e6b4cd6994199911e42fa554b0c78e24c Apr 16 16:48:46.703577 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:46.703547 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wrbkk\"" Apr 16 16:48:46.711544 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:46.711515 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kz89" Apr 16 16:48:46.836229 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:46.836196 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5kz89"] Apr 16 16:48:46.839107 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:48:46.839083 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode730d89c_85b3_4a44_883b_d65f347261ec.slice/crio-a6922eca6773f1ab954171bc75d1392aa9b421b68a48034388fe9dac026acc81 WatchSource:0}: Error finding container a6922eca6773f1ab954171bc75d1392aa9b421b68a48034388fe9dac026acc81: Status 404 returned error can't find the container with id a6922eca6773f1ab954171bc75d1392aa9b421b68a48034388fe9dac026acc81 Apr 16 16:48:46.948767 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:46.948727 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5kz89" event={"ID":"e730d89c-85b3-4a44-883b-d65f347261ec","Type":"ContainerStarted","Data":"a6922eca6773f1ab954171bc75d1392aa9b421b68a48034388fe9dac026acc81"} Apr 16 16:48:46.949722 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:46.949691 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pgvd2" event={"ID":"98d7d9af-1c21-4f39-b605-21a11d06f2ae","Type":"ContainerStarted","Data":"71368e1031508652a2f1a1834d565e3e6b4cd6994199911e42fa554b0c78e24c"} Apr 16 16:48:47.735151 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:47.734977 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4e2d35c-1627-4f9c-bea6-985f08319902-metrics-tls\") pod \"dns-default-rzx7h\" (UID: \"b4e2d35c-1627-4f9c-bea6-985f08319902\") " pod="openshift-dns/dns-default-rzx7h" Apr 16 16:48:47.735151 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:47.735038 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/efee44e6-2e2b-47f0-b356-6e77267f19e0-cert\") pod \"ingress-canary-xfnx6\" (UID: \"efee44e6-2e2b-47f0-b356-6e77267f19e0\") " pod="openshift-ingress-canary/ingress-canary-xfnx6" Apr 16 16:48:47.735777 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:47.735194 2576 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 16 16:48:47.735777 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:47.735273 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efee44e6-2e2b-47f0-b356-6e77267f19e0-cert podName:efee44e6-2e2b-47f0-b356-6e77267f19e0 nodeName:}" failed. No retries permitted until 2026-04-16 16:49:19.735235588 +0000 UTC m=+98.796385112 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/efee44e6-2e2b-47f0-b356-6e77267f19e0-cert") pod "ingress-canary-xfnx6" (UID: "efee44e6-2e2b-47f0-b356-6e77267f19e0") : secret "canary-serving-cert" not found Apr 16 16:48:47.735777 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:47.735689 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 16 16:48:47.735777 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:48:47.735737 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4e2d35c-1627-4f9c-bea6-985f08319902-metrics-tls podName:b4e2d35c-1627-4f9c-bea6-985f08319902 nodeName:}" failed. No retries permitted until 2026-04-16 16:49:19.735722073 +0000 UTC m=+98.796871581 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4e2d35c-1627-4f9c-bea6-985f08319902-metrics-tls") pod "dns-default-rzx7h" (UID: "b4e2d35c-1627-4f9c-bea6-985f08319902") : secret "dns-default-metrics-tls" not found Apr 16 16:48:51.962195 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:51.962163 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5kz89" event={"ID":"e730d89c-85b3-4a44-883b-d65f347261ec","Type":"ContainerStarted","Data":"3cc6b34987a6b99c8765f6b61ac92cd2c60655a20dc0f69d18931fda8c8aeb00"} Apr 16 16:48:51.962659 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:51.962273 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5kz89" Apr 16 16:48:51.963488 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:51.963466 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pgvd2" event={"ID":"98d7d9af-1c21-4f39-b605-21a11d06f2ae","Type":"ContainerStarted","Data":"1b5a7185533c46dc1cc6fbb49b203529a9a12c082ad78e2b917cebb69375bb4a"} Apr 16 16:48:51.975822 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:51.975777 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5kz89" podStartSLOduration=66.581235315 podStartE2EDuration="1m10.975762644s" podCreationTimestamp="2026-04-16 16:47:41 +0000 UTC" firstStartedPulling="2026-04-16 16:48:46.840946584 +0000 UTC m=+65.902096092" lastFinishedPulling="2026-04-16 16:48:51.235473897 +0000 UTC m=+70.296623421" observedRunningTime="2026-04-16 16:48:51.975322834 +0000 UTC m=+71.036472364" watchObservedRunningTime="2026-04-16 16:48:51.975762644 +0000 UTC m=+71.036912178" Apr 16 16:48:51.987014 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:48:51.986973 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pgvd2" podStartSLOduration=66.349556596 podStartE2EDuration="1m10.986963058s" podCreationTimestamp="2026-04-16 16:47:41 +0000 UTC" firstStartedPulling="2026-04-16 16:48:46.60326828 +0000 UTC m=+65.664417792" lastFinishedPulling="2026-04-16 16:48:51.240674742 +0000 UTC m=+70.301824254" observedRunningTime="2026-04-16 16:48:51.98662582 +0000 UTC m=+71.047775348" watchObservedRunningTime="2026-04-16 16:48:51.986963058 +0000 UTC m=+71.048112588" Apr 16 16:49:19.761818 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:19.761782 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4e2d35c-1627-4f9c-bea6-985f08319902-metrics-tls\") pod \"dns-default-rzx7h\" (UID: \"b4e2d35c-1627-4f9c-bea6-985f08319902\") " pod="openshift-dns/dns-default-rzx7h" Apr 16 16:49:19.762179 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:19.761824 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/efee44e6-2e2b-47f0-b356-6e77267f19e0-cert\") pod \"ingress-canary-xfnx6\" (UID: \"efee44e6-2e2b-47f0-b356-6e77267f19e0\") " pod="openshift-ingress-canary/ingress-canary-xfnx6" Apr 16 16:49:19.762179 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:49:19.761931 2576 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 16 16:49:19.762179 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:49:19.761930 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 16 16:49:19.762179 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:49:19.761997 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efee44e6-2e2b-47f0-b356-6e77267f19e0-cert podName:efee44e6-2e2b-47f0-b356-6e77267f19e0 nodeName:}" failed. No retries permitted until 2026-04-16 16:50:23.761982367 +0000 UTC m=+162.823131879 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/efee44e6-2e2b-47f0-b356-6e77267f19e0-cert") pod "ingress-canary-xfnx6" (UID: "efee44e6-2e2b-47f0-b356-6e77267f19e0") : secret "canary-serving-cert" not found Apr 16 16:49:19.762179 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:49:19.762012 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4e2d35c-1627-4f9c-bea6-985f08319902-metrics-tls podName:b4e2d35c-1627-4f9c-bea6-985f08319902 nodeName:}" failed. No retries permitted until 2026-04-16 16:50:23.762004839 +0000 UTC m=+162.823154348 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4e2d35c-1627-4f9c-bea6-985f08319902-metrics-tls") pod "dns-default-rzx7h" (UID: "b4e2d35c-1627-4f9c-bea6-985f08319902") : secret "dns-default-metrics-tls" not found Apr 16 16:49:22.968700 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:22.968669 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5kz89" Apr 16 16:49:45.307462 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:45.307426 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6667474d89-lqmw4"] Apr 16 16:49:45.310229 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:45.310213 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-lqmw4" Apr 16 16:49:45.312281 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:45.312261 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Apr 16 16:49:45.312402 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:45.312383 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-hjnm5\"" Apr 16 16:49:45.312914 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:45.312899 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Apr 16 16:49:45.313798 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:45.313773 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Apr 16 16:49:45.313866 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:45.313789 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Apr 16 16:49:45.320482 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:45.320461 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6667474d89-lqmw4"] Apr 16 16:49:45.417430 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:45.417396 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-85bf56ff4-nrtll"] Apr 16 16:49:45.420196 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:45.420180 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-85bf56ff4-nrtll" Apr 16 16:49:45.421998 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:45.421973 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Apr 16 16:49:45.422261 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:45.422219 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Apr 16 16:49:45.422384 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:45.422289 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Apr 16 16:49:45.422384 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:45.422314 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Apr 16 16:49:45.422523 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:45.422504 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Apr 16 16:49:45.422737 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:45.422721 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-t7hkw\"" Apr 16 16:49:45.422816 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:45.422798 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Apr 16 16:49:45.431138 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:45.431118 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-85bf56ff4-nrtll"] Apr 16 16:49:45.432316 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:45.432295 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/12d57405-05cf-462d-a747-06c91ca773cb-telemetry-config\") pod \"cluster-monitoring-operator-6667474d89-lqmw4\" (UID: \"12d57405-05cf-462d-a747-06c91ca773cb\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-lqmw4" Apr 16 16:49:45.432388 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:45.432345 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66hht\" (UniqueName: \"kubernetes.io/projected/12d57405-05cf-462d-a747-06c91ca773cb-kube-api-access-66hht\") pod \"cluster-monitoring-operator-6667474d89-lqmw4\" (UID: \"12d57405-05cf-462d-a747-06c91ca773cb\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-lqmw4" Apr 16 16:49:45.432425 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:45.432397 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/12d57405-05cf-462d-a747-06c91ca773cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6667474d89-lqmw4\" (UID: \"12d57405-05cf-462d-a747-06c91ca773cb\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-lqmw4" Apr 16 16:49:45.532732 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:45.532690 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/12d57405-05cf-462d-a747-06c91ca773cb-telemetry-config\") pod \"cluster-monitoring-operator-6667474d89-lqmw4\" (UID: \"12d57405-05cf-462d-a747-06c91ca773cb\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-lqmw4" Apr 16 16:49:45.532732 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:45.532736 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45604778-a27a-4c44-b245-0208bfb4154a-service-ca-bundle\") pod \"router-default-85bf56ff4-nrtll\" (UID: \"45604778-a27a-4c44-b245-0208bfb4154a\") " pod="openshift-ingress/router-default-85bf56ff4-nrtll" Apr 16 16:49:45.532983 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:45.532764 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/45604778-a27a-4c44-b245-0208bfb4154a-stats-auth\") pod \"router-default-85bf56ff4-nrtll\" (UID: \"45604778-a27a-4c44-b245-0208bfb4154a\") " pod="openshift-ingress/router-default-85bf56ff4-nrtll" Apr 16 16:49:45.532983 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:45.532797 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-66hht\" (UniqueName: \"kubernetes.io/projected/12d57405-05cf-462d-a747-06c91ca773cb-kube-api-access-66hht\") pod \"cluster-monitoring-operator-6667474d89-lqmw4\" (UID: \"12d57405-05cf-462d-a747-06c91ca773cb\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-lqmw4" Apr 16 16:49:45.532983 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:45.532825 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/12d57405-05cf-462d-a747-06c91ca773cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6667474d89-lqmw4\" (UID: \"12d57405-05cf-462d-a747-06c91ca773cb\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-lqmw4" Apr 16 16:49:45.532983 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:45.532868 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggqnj\" (UniqueName: \"kubernetes.io/projected/45604778-a27a-4c44-b245-0208bfb4154a-kube-api-access-ggqnj\") pod \"router-default-85bf56ff4-nrtll\" (UID: \"45604778-a27a-4c44-b245-0208bfb4154a\") " pod="openshift-ingress/router-default-85bf56ff4-nrtll" Apr 16 16:49:45.532983 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:45.532926 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45604778-a27a-4c44-b245-0208bfb4154a-metrics-certs\") pod \"router-default-85bf56ff4-nrtll\" (UID: \"45604778-a27a-4c44-b245-0208bfb4154a\") " pod="openshift-ingress/router-default-85bf56ff4-nrtll" Apr 16 16:49:45.533167 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:45.532990 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/45604778-a27a-4c44-b245-0208bfb4154a-default-certificate\") pod \"router-default-85bf56ff4-nrtll\" (UID: \"45604778-a27a-4c44-b245-0208bfb4154a\") " pod="openshift-ingress/router-default-85bf56ff4-nrtll" Apr 16 16:49:45.533167 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:49:45.533023 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Apr 16 16:49:45.533167 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:49:45.533084 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12d57405-05cf-462d-a747-06c91ca773cb-cluster-monitoring-operator-tls podName:12d57405-05cf-462d-a747-06c91ca773cb nodeName:}" failed. No retries permitted until 2026-04-16 16:49:46.033069309 +0000 UTC m=+125.094218818 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/12d57405-05cf-462d-a747-06c91ca773cb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6667474d89-lqmw4" (UID: "12d57405-05cf-462d-a747-06c91ca773cb") : secret "cluster-monitoring-operator-tls" not found Apr 16 16:49:45.533448 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:45.533430 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/12d57405-05cf-462d-a747-06c91ca773cb-telemetry-config\") pod \"cluster-monitoring-operator-6667474d89-lqmw4\" (UID: \"12d57405-05cf-462d-a747-06c91ca773cb\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-lqmw4" Apr 16 16:49:45.545031 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:45.545007 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66hht\" (UniqueName: \"kubernetes.io/projected/12d57405-05cf-462d-a747-06c91ca773cb-kube-api-access-66hht\") pod \"cluster-monitoring-operator-6667474d89-lqmw4\" (UID: \"12d57405-05cf-462d-a747-06c91ca773cb\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-lqmw4" Apr 16 16:49:45.634327 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:45.634263 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/45604778-a27a-4c44-b245-0208bfb4154a-stats-auth\") pod \"router-default-85bf56ff4-nrtll\" (UID: \"45604778-a27a-4c44-b245-0208bfb4154a\") " pod="openshift-ingress/router-default-85bf56ff4-nrtll" Apr 16 16:49:45.634327 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:45.634315 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ggqnj\" (UniqueName: \"kubernetes.io/projected/45604778-a27a-4c44-b245-0208bfb4154a-kube-api-access-ggqnj\") pod \"router-default-85bf56ff4-nrtll\" (UID: \"45604778-a27a-4c44-b245-0208bfb4154a\") " pod="openshift-ingress/router-default-85bf56ff4-nrtll" Apr 16 16:49:45.634478 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:45.634353 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45604778-a27a-4c44-b245-0208bfb4154a-metrics-certs\") pod \"router-default-85bf56ff4-nrtll\" (UID: \"45604778-a27a-4c44-b245-0208bfb4154a\") " pod="openshift-ingress/router-default-85bf56ff4-nrtll" Apr 16 16:49:45.634478 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:45.634372 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/45604778-a27a-4c44-b245-0208bfb4154a-default-certificate\") pod \"router-default-85bf56ff4-nrtll\" (UID: \"45604778-a27a-4c44-b245-0208bfb4154a\") " pod="openshift-ingress/router-default-85bf56ff4-nrtll" Apr 16 16:49:45.634478 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:45.634396 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45604778-a27a-4c44-b245-0208bfb4154a-service-ca-bundle\") pod \"router-default-85bf56ff4-nrtll\" (UID: \"45604778-a27a-4c44-b245-0208bfb4154a\") " pod="openshift-ingress/router-default-85bf56ff4-nrtll" Apr 16 16:49:45.634626 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:49:45.634470 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Apr 16 16:49:45.634626 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:49:45.634508 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/45604778-a27a-4c44-b245-0208bfb4154a-service-ca-bundle podName:45604778-a27a-4c44-b245-0208bfb4154a nodeName:}" failed. No retries permitted until 2026-04-16 16:49:46.134494587 +0000 UTC m=+125.195644097 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/45604778-a27a-4c44-b245-0208bfb4154a-service-ca-bundle") pod "router-default-85bf56ff4-nrtll" (UID: "45604778-a27a-4c44-b245-0208bfb4154a") : configmap references non-existent config key: service-ca.crt Apr 16 16:49:45.634626 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:49:45.634553 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45604778-a27a-4c44-b245-0208bfb4154a-metrics-certs podName:45604778-a27a-4c44-b245-0208bfb4154a nodeName:}" failed. No retries permitted until 2026-04-16 16:49:46.134535981 +0000 UTC m=+125.195685505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/45604778-a27a-4c44-b245-0208bfb4154a-metrics-certs") pod "router-default-85bf56ff4-nrtll" (UID: "45604778-a27a-4c44-b245-0208bfb4154a") : secret "router-metrics-certs-default" not found Apr 16 16:49:45.636758 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:45.636736 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/45604778-a27a-4c44-b245-0208bfb4154a-default-certificate\") pod \"router-default-85bf56ff4-nrtll\" (UID: \"45604778-a27a-4c44-b245-0208bfb4154a\") " pod="openshift-ingress/router-default-85bf56ff4-nrtll" Apr 16 16:49:45.636841 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:45.636820 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/45604778-a27a-4c44-b245-0208bfb4154a-stats-auth\") pod \"router-default-85bf56ff4-nrtll\" (UID: \"45604778-a27a-4c44-b245-0208bfb4154a\") " pod="openshift-ingress/router-default-85bf56ff4-nrtll" Apr 16 16:49:45.644755 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:45.644734 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggqnj\" (UniqueName: \"kubernetes.io/projected/45604778-a27a-4c44-b245-0208bfb4154a-kube-api-access-ggqnj\") pod \"router-default-85bf56ff4-nrtll\" (UID: \"45604778-a27a-4c44-b245-0208bfb4154a\") " pod="openshift-ingress/router-default-85bf56ff4-nrtll" Apr 16 16:49:46.037280 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:46.037162 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/12d57405-05cf-462d-a747-06c91ca773cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6667474d89-lqmw4\" (UID: \"12d57405-05cf-462d-a747-06c91ca773cb\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-lqmw4" Apr 16 16:49:46.037442 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:49:46.037335 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Apr 16 16:49:46.037442 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:49:46.037405 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12d57405-05cf-462d-a747-06c91ca773cb-cluster-monitoring-operator-tls podName:12d57405-05cf-462d-a747-06c91ca773cb nodeName:}" failed. No retries permitted until 2026-04-16 16:49:47.037389735 +0000 UTC m=+126.098539243 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/12d57405-05cf-462d-a747-06c91ca773cb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6667474d89-lqmw4" (UID: "12d57405-05cf-462d-a747-06c91ca773cb") : secret "cluster-monitoring-operator-tls" not found Apr 16 16:49:46.138448 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:46.138405 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45604778-a27a-4c44-b245-0208bfb4154a-metrics-certs\") pod \"router-default-85bf56ff4-nrtll\" (UID: \"45604778-a27a-4c44-b245-0208bfb4154a\") " pod="openshift-ingress/router-default-85bf56ff4-nrtll" Apr 16 16:49:46.138448 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:46.138457 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45604778-a27a-4c44-b245-0208bfb4154a-service-ca-bundle\") pod \"router-default-85bf56ff4-nrtll\" (UID: \"45604778-a27a-4c44-b245-0208bfb4154a\") " pod="openshift-ingress/router-default-85bf56ff4-nrtll" Apr 16 16:49:46.138636 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:49:46.138550 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/45604778-a27a-4c44-b245-0208bfb4154a-service-ca-bundle podName:45604778-a27a-4c44-b245-0208bfb4154a nodeName:}" failed. No retries permitted until 2026-04-16 16:49:47.138537354 +0000 UTC m=+126.199686863 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/45604778-a27a-4c44-b245-0208bfb4154a-service-ca-bundle") pod "router-default-85bf56ff4-nrtll" (UID: "45604778-a27a-4c44-b245-0208bfb4154a") : configmap references non-existent config key: service-ca.crt Apr 16 16:49:46.138636 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:49:46.138553 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Apr 16 16:49:46.138636 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:49:46.138611 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45604778-a27a-4c44-b245-0208bfb4154a-metrics-certs podName:45604778-a27a-4c44-b245-0208bfb4154a nodeName:}" failed. No retries permitted until 2026-04-16 16:49:47.138596795 +0000 UTC m=+126.199746331 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/45604778-a27a-4c44-b245-0208bfb4154a-metrics-certs") pod "router-default-85bf56ff4-nrtll" (UID: "45604778-a27a-4c44-b245-0208bfb4154a") : secret "router-metrics-certs-default" not found Apr 16 16:49:47.045260 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:47.045206 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/12d57405-05cf-462d-a747-06c91ca773cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6667474d89-lqmw4\" (UID: \"12d57405-05cf-462d-a747-06c91ca773cb\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-lqmw4" Apr 16 16:49:47.045653 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:49:47.045360 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Apr 16 16:49:47.045653 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:49:47.045422 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12d57405-05cf-462d-a747-06c91ca773cb-cluster-monitoring-operator-tls podName:12d57405-05cf-462d-a747-06c91ca773cb nodeName:}" failed. No retries permitted until 2026-04-16 16:49:49.045405998 +0000 UTC m=+128.106555523 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/12d57405-05cf-462d-a747-06c91ca773cb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6667474d89-lqmw4" (UID: "12d57405-05cf-462d-a747-06c91ca773cb") : secret "cluster-monitoring-operator-tls" not found Apr 16 16:49:47.146506 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:47.146474 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45604778-a27a-4c44-b245-0208bfb4154a-service-ca-bundle\") pod \"router-default-85bf56ff4-nrtll\" (UID: \"45604778-a27a-4c44-b245-0208bfb4154a\") " pod="openshift-ingress/router-default-85bf56ff4-nrtll" Apr 16 16:49:47.146665 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:47.146561 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45604778-a27a-4c44-b245-0208bfb4154a-metrics-certs\") pod \"router-default-85bf56ff4-nrtll\" (UID: \"45604778-a27a-4c44-b245-0208bfb4154a\") " pod="openshift-ingress/router-default-85bf56ff4-nrtll" Apr 16 16:49:47.146665 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:49:47.146637 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/45604778-a27a-4c44-b245-0208bfb4154a-service-ca-bundle podName:45604778-a27a-4c44-b245-0208bfb4154a nodeName:}" failed. No retries permitted until 2026-04-16 16:49:49.146617539 +0000 UTC m=+128.207767051 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/45604778-a27a-4c44-b245-0208bfb4154a-service-ca-bundle") pod "router-default-85bf56ff4-nrtll" (UID: "45604778-a27a-4c44-b245-0208bfb4154a") : configmap references non-existent config key: service-ca.crt Apr 16 16:49:47.146665 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:49:47.146641 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Apr 16 16:49:47.146766 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:49:47.146683 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45604778-a27a-4c44-b245-0208bfb4154a-metrics-certs podName:45604778-a27a-4c44-b245-0208bfb4154a nodeName:}" failed. No retries permitted until 2026-04-16 16:49:49.146671828 +0000 UTC m=+128.207821337 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/45604778-a27a-4c44-b245-0208bfb4154a-metrics-certs") pod "router-default-85bf56ff4-nrtll" (UID: "45604778-a27a-4c44-b245-0208bfb4154a") : secret "router-metrics-certs-default" not found Apr 16 16:49:49.061619 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:49.061574 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/12d57405-05cf-462d-a747-06c91ca773cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6667474d89-lqmw4\" (UID: \"12d57405-05cf-462d-a747-06c91ca773cb\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-lqmw4" Apr 16 16:49:49.062042 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:49:49.061695 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Apr 16 16:49:49.062042 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:49:49.061755 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12d57405-05cf-462d-a747-06c91ca773cb-cluster-monitoring-operator-tls podName:12d57405-05cf-462d-a747-06c91ca773cb nodeName:}" failed. No retries permitted until 2026-04-16 16:49:53.061741717 +0000 UTC m=+132.122891225 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/12d57405-05cf-462d-a747-06c91ca773cb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6667474d89-lqmw4" (UID: "12d57405-05cf-462d-a747-06c91ca773cb") : secret "cluster-monitoring-operator-tls" not found Apr 16 16:49:49.162822 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:49.162780 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45604778-a27a-4c44-b245-0208bfb4154a-metrics-certs\") pod \"router-default-85bf56ff4-nrtll\" (UID: \"45604778-a27a-4c44-b245-0208bfb4154a\") " pod="openshift-ingress/router-default-85bf56ff4-nrtll" Apr 16 16:49:49.162991 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:49.162832 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45604778-a27a-4c44-b245-0208bfb4154a-service-ca-bundle\") pod \"router-default-85bf56ff4-nrtll\" (UID: \"45604778-a27a-4c44-b245-0208bfb4154a\") " pod="openshift-ingress/router-default-85bf56ff4-nrtll" Apr 16 16:49:49.162991 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:49:49.162928 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Apr 16 16:49:49.162991 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:49:49.162944 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/45604778-a27a-4c44-b245-0208bfb4154a-service-ca-bundle podName:45604778-a27a-4c44-b245-0208bfb4154a nodeName:}" failed. No retries permitted until 2026-04-16 16:49:53.162929017 +0000 UTC m=+132.224078525 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/45604778-a27a-4c44-b245-0208bfb4154a-service-ca-bundle") pod "router-default-85bf56ff4-nrtll" (UID: "45604778-a27a-4c44-b245-0208bfb4154a") : configmap references non-existent config key: service-ca.crt Apr 16 16:49:49.162991 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:49:49.162982 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45604778-a27a-4c44-b245-0208bfb4154a-metrics-certs podName:45604778-a27a-4c44-b245-0208bfb4154a nodeName:}" failed. No retries permitted until 2026-04-16 16:49:53.162968381 +0000 UTC m=+132.224117890 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/45604778-a27a-4c44-b245-0208bfb4154a-metrics-certs") pod "router-default-85bf56ff4-nrtll" (UID: "45604778-a27a-4c44-b245-0208bfb4154a") : secret "router-metrics-certs-default" not found Apr 16 16:49:50.271273 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:50.271228 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/412718e2-2075-4c7b-a6a9-3b94ed57564c-metrics-certs\") pod \"network-metrics-daemon-hk5gr\" (UID: \"412718e2-2075-4c7b-a6a9-3b94ed57564c\") " pod="openshift-multus/network-metrics-daemon-hk5gr" Apr 16 16:49:50.271639 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:49:50.271340 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Apr 16 16:49:50.271639 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:49:50.271389 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/412718e2-2075-4c7b-a6a9-3b94ed57564c-metrics-certs podName:412718e2-2075-4c7b-a6a9-3b94ed57564c nodeName:}" failed. No retries permitted until 2026-04-16 16:51:52.271375317 +0000 UTC m=+251.332524825 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/412718e2-2075-4c7b-a6a9-3b94ed57564c-metrics-certs") pod "network-metrics-daemon-hk5gr" (UID: "412718e2-2075-4c7b-a6a9-3b94ed57564c") : secret "metrics-daemon-secret" not found Apr 16 16:49:50.913851 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:50.913823 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-42cdh_99da12c7-93e3-4848-afc5-36f2cbf5645d/dns-node-resolver/0.log" Apr 16 16:49:52.114178 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:52.114148 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q6fvl_fce43675-3aee-453f-b5ad-a28fba84c810/node-ca/0.log" Apr 16 16:49:53.094945 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:53.094904 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/12d57405-05cf-462d-a747-06c91ca773cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6667474d89-lqmw4\" (UID: \"12d57405-05cf-462d-a747-06c91ca773cb\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-lqmw4" Apr 16 16:49:53.095104 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:49:53.095023 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Apr 16 16:49:53.095104 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:49:53.095075 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12d57405-05cf-462d-a747-06c91ca773cb-cluster-monitoring-operator-tls podName:12d57405-05cf-462d-a747-06c91ca773cb nodeName:}" failed. No retries permitted until 2026-04-16 16:50:01.095062309 +0000 UTC m=+140.156211818 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/12d57405-05cf-462d-a747-06c91ca773cb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6667474d89-lqmw4" (UID: "12d57405-05cf-462d-a747-06c91ca773cb") : secret "cluster-monitoring-operator-tls" not found Apr 16 16:49:53.195486 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:53.195448 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45604778-a27a-4c44-b245-0208bfb4154a-metrics-certs\") pod \"router-default-85bf56ff4-nrtll\" (UID: \"45604778-a27a-4c44-b245-0208bfb4154a\") " pod="openshift-ingress/router-default-85bf56ff4-nrtll" Apr 16 16:49:53.195888 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:53.195509 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45604778-a27a-4c44-b245-0208bfb4154a-service-ca-bundle\") pod \"router-default-85bf56ff4-nrtll\" (UID: \"45604778-a27a-4c44-b245-0208bfb4154a\") " pod="openshift-ingress/router-default-85bf56ff4-nrtll" Apr 16 16:49:53.195888 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:49:53.195604 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Apr 16 16:49:53.195888 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:49:53.195676 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45604778-a27a-4c44-b245-0208bfb4154a-metrics-certs podName:45604778-a27a-4c44-b245-0208bfb4154a nodeName:}" failed. No retries permitted until 2026-04-16 16:50:01.195661055 +0000 UTC m=+140.256810568 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/45604778-a27a-4c44-b245-0208bfb4154a-metrics-certs") pod "router-default-85bf56ff4-nrtll" (UID: "45604778-a27a-4c44-b245-0208bfb4154a") : secret "router-metrics-certs-default" not found Apr 16 16:49:53.195888 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:49:53.195690 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/45604778-a27a-4c44-b245-0208bfb4154a-service-ca-bundle podName:45604778-a27a-4c44-b245-0208bfb4154a nodeName:}" failed. No retries permitted until 2026-04-16 16:50:01.195684075 +0000 UTC m=+140.256833584 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/45604778-a27a-4c44-b245-0208bfb4154a-service-ca-bundle") pod "router-default-85bf56ff4-nrtll" (UID: "45604778-a27a-4c44-b245-0208bfb4154a") : configmap references non-existent config key: service-ca.crt Apr 16 16:49:55.402886 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:55.402851 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-756bb7d76f-dzm7p"] Apr 16 16:49:55.406002 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:55.405985 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-756bb7d76f-dzm7p" Apr 16 16:49:55.408106 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:55.408082 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Apr 16 16:49:55.408228 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:55.408091 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Apr 16 16:49:55.408408 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:55.408392 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Apr 16 16:49:55.408454 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:55.408392 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Apr 16 16:49:55.408682 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:55.408668 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-jbwgh\"" Apr 16 16:49:55.413106 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:55.413087 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-756bb7d76f-dzm7p"] Apr 16 16:49:55.513426 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:55.513388 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4bb5aacd-20aa-4fb6-ae32-eee3df133067-serving-cert\") pod \"kube-storage-version-migrator-operator-756bb7d76f-dzm7p\" (UID: \"4bb5aacd-20aa-4fb6-ae32-eee3df133067\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-756bb7d76f-dzm7p" Apr 16 16:49:55.513595 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:55.513446 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4bb5aacd-20aa-4fb6-ae32-eee3df133067-config\") pod \"kube-storage-version-migrator-operator-756bb7d76f-dzm7p\" (UID: \"4bb5aacd-20aa-4fb6-ae32-eee3df133067\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-756bb7d76f-dzm7p" Apr 16 16:49:55.513595 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:55.513546 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsljq\" (UniqueName: \"kubernetes.io/projected/4bb5aacd-20aa-4fb6-ae32-eee3df133067-kube-api-access-xsljq\") pod \"kube-storage-version-migrator-operator-756bb7d76f-dzm7p\" (UID: \"4bb5aacd-20aa-4fb6-ae32-eee3df133067\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-756bb7d76f-dzm7p" Apr 16 16:49:55.614233 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:55.614198 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xsljq\" (UniqueName: \"kubernetes.io/projected/4bb5aacd-20aa-4fb6-ae32-eee3df133067-kube-api-access-xsljq\") pod \"kube-storage-version-migrator-operator-756bb7d76f-dzm7p\" (UID: \"4bb5aacd-20aa-4fb6-ae32-eee3df133067\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-756bb7d76f-dzm7p" Apr 16 16:49:55.614441 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:55.614272 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4bb5aacd-20aa-4fb6-ae32-eee3df133067-serving-cert\") pod \"kube-storage-version-migrator-operator-756bb7d76f-dzm7p\" (UID: \"4bb5aacd-20aa-4fb6-ae32-eee3df133067\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-756bb7d76f-dzm7p" Apr 16 16:49:55.614441 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:55.614304 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4bb5aacd-20aa-4fb6-ae32-eee3df133067-config\") pod \"kube-storage-version-migrator-operator-756bb7d76f-dzm7p\" (UID: \"4bb5aacd-20aa-4fb6-ae32-eee3df133067\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-756bb7d76f-dzm7p" Apr 16 16:49:55.614948 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:55.614928 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4bb5aacd-20aa-4fb6-ae32-eee3df133067-config\") pod \"kube-storage-version-migrator-operator-756bb7d76f-dzm7p\" (UID: \"4bb5aacd-20aa-4fb6-ae32-eee3df133067\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-756bb7d76f-dzm7p" Apr 16 16:49:55.616767 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:55.616649 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4bb5aacd-20aa-4fb6-ae32-eee3df133067-serving-cert\") pod \"kube-storage-version-migrator-operator-756bb7d76f-dzm7p\" (UID: \"4bb5aacd-20aa-4fb6-ae32-eee3df133067\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-756bb7d76f-dzm7p" Apr 16 16:49:55.622789 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:55.622767 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsljq\" (UniqueName: \"kubernetes.io/projected/4bb5aacd-20aa-4fb6-ae32-eee3df133067-kube-api-access-xsljq\") pod \"kube-storage-version-migrator-operator-756bb7d76f-dzm7p\" (UID: \"4bb5aacd-20aa-4fb6-ae32-eee3df133067\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-756bb7d76f-dzm7p" Apr 16 16:49:55.715146 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:55.715035 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-756bb7d76f-dzm7p" Apr 16 16:49:55.827822 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:55.827794 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-756bb7d76f-dzm7p"] Apr 16 16:49:55.831287 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:49:55.831237 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bb5aacd_20aa_4fb6_ae32_eee3df133067.slice/crio-2bc069e03f22200e81c547af3eafb6b6ad1a227f8a716fa736ff095728f7ce7e WatchSource:0}: Error finding container 2bc069e03f22200e81c547af3eafb6b6ad1a227f8a716fa736ff095728f7ce7e: Status 404 returned error can't find the container with id 2bc069e03f22200e81c547af3eafb6b6ad1a227f8a716fa736ff095728f7ce7e Apr 16 16:49:56.081376 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:56.081333 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-756bb7d76f-dzm7p" event={"ID":"4bb5aacd-20aa-4fb6-ae32-eee3df133067","Type":"ContainerStarted","Data":"2bc069e03f22200e81c547af3eafb6b6ad1a227f8a716fa736ff095728f7ce7e"} Apr 16 16:49:57.626209 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:57.626174 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7b678d77c7-pk46g"] Apr 16 16:49:57.629063 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:57.629046 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7b678d77c7-pk46g" Apr 16 16:49:57.631079 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:57.631062 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-khq8k\"" Apr 16 16:49:57.641659 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:57.641634 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7b678d77c7-pk46g"] Apr 16 16:49:57.731875 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:57.731836 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfmr4\" (UniqueName: \"kubernetes.io/projected/120a4def-c7cf-426f-8c72-f746c56c4674-kube-api-access-pfmr4\") pod \"network-check-source-7b678d77c7-pk46g\" (UID: \"120a4def-c7cf-426f-8c72-f746c56c4674\") " pod="openshift-network-diagnostics/network-check-source-7b678d77c7-pk46g" Apr 16 16:49:57.833135 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:57.833088 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pfmr4\" (UniqueName: \"kubernetes.io/projected/120a4def-c7cf-426f-8c72-f746c56c4674-kube-api-access-pfmr4\") pod \"network-check-source-7b678d77c7-pk46g\" (UID: \"120a4def-c7cf-426f-8c72-f746c56c4674\") " pod="openshift-network-diagnostics/network-check-source-7b678d77c7-pk46g" Apr 16 16:49:57.844631 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:57.844598 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfmr4\" (UniqueName: \"kubernetes.io/projected/120a4def-c7cf-426f-8c72-f746c56c4674-kube-api-access-pfmr4\") pod \"network-check-source-7b678d77c7-pk46g\" (UID: \"120a4def-c7cf-426f-8c72-f746c56c4674\") " pod="openshift-network-diagnostics/network-check-source-7b678d77c7-pk46g" Apr 16 16:49:57.937411 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:57.937321 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7b678d77c7-pk46g" Apr 16 16:49:58.050176 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:58.050145 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7b678d77c7-pk46g"] Apr 16 16:49:58.054465 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:49:58.054138 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod120a4def_c7cf_426f_8c72_f746c56c4674.slice/crio-0838234bb37ea2c71ff8fdd7eda125a583c6bd16112483c30ed443dba11a0aa9 WatchSource:0}: Error finding container 0838234bb37ea2c71ff8fdd7eda125a583c6bd16112483c30ed443dba11a0aa9: Status 404 returned error can't find the container with id 0838234bb37ea2c71ff8fdd7eda125a583c6bd16112483c30ed443dba11a0aa9 Apr 16 16:49:58.086637 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:58.086609 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-756bb7d76f-dzm7p" event={"ID":"4bb5aacd-20aa-4fb6-ae32-eee3df133067","Type":"ContainerStarted","Data":"67c4a59a42cf2f73ee8aaef1c197522159a2f20d4cab8c65c7810fa1c8449fb0"} Apr 16 16:49:58.087641 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:58.087615 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7b678d77c7-pk46g" event={"ID":"120a4def-c7cf-426f-8c72-f746c56c4674","Type":"ContainerStarted","Data":"0838234bb37ea2c71ff8fdd7eda125a583c6bd16112483c30ed443dba11a0aa9"} Apr 16 16:49:58.104819 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:58.104771 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-756bb7d76f-dzm7p" podStartSLOduration=1.552386651 podStartE2EDuration="3.10475333s" podCreationTimestamp="2026-04-16 16:49:55 +0000 UTC" firstStartedPulling="2026-04-16 16:49:55.832897586 +0000 UTC m=+134.894047095" lastFinishedPulling="2026-04-16 16:49:57.385264262 +0000 UTC m=+136.446413774" observedRunningTime="2026-04-16 16:49:58.104352858 +0000 UTC m=+137.165502389" watchObservedRunningTime="2026-04-16 16:49:58.10475333 +0000 UTC m=+137.165902865" Apr 16 16:49:59.090522 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:59.090484 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7b678d77c7-pk46g" event={"ID":"120a4def-c7cf-426f-8c72-f746c56c4674","Type":"ContainerStarted","Data":"1f3188b6dbec57b543c6ec2e257a86a869cfd52dde8f4190900d1bbcc3bfa52c"} Apr 16 16:49:59.108320 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:49:59.108275 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7b678d77c7-pk46g" podStartSLOduration=2.108237198 podStartE2EDuration="2.108237198s" podCreationTimestamp="2026-04-16 16:49:57 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:49:59.107956503 +0000 UTC m=+138.169106025" watchObservedRunningTime="2026-04-16 16:49:59.108237198 +0000 UTC m=+138.169386730" Apr 16 16:50:01.159305 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:01.159260 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/12d57405-05cf-462d-a747-06c91ca773cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6667474d89-lqmw4\" (UID: \"12d57405-05cf-462d-a747-06c91ca773cb\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-lqmw4" Apr 16 16:50:01.159681 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:50:01.159408 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Apr 16 16:50:01.159681 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:50:01.159476 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12d57405-05cf-462d-a747-06c91ca773cb-cluster-monitoring-operator-tls podName:12d57405-05cf-462d-a747-06c91ca773cb nodeName:}" failed. No retries permitted until 2026-04-16 16:50:17.159460035 +0000 UTC m=+156.220609543 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/12d57405-05cf-462d-a747-06c91ca773cb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6667474d89-lqmw4" (UID: "12d57405-05cf-462d-a747-06c91ca773cb") : secret "cluster-monitoring-operator-tls" not found Apr 16 16:50:01.259928 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:01.259879 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45604778-a27a-4c44-b245-0208bfb4154a-service-ca-bundle\") pod \"router-default-85bf56ff4-nrtll\" (UID: \"45604778-a27a-4c44-b245-0208bfb4154a\") " pod="openshift-ingress/router-default-85bf56ff4-nrtll" Apr 16 16:50:01.260050 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:01.259976 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45604778-a27a-4c44-b245-0208bfb4154a-metrics-certs\") pod \"router-default-85bf56ff4-nrtll\" (UID: \"45604778-a27a-4c44-b245-0208bfb4154a\") " pod="openshift-ingress/router-default-85bf56ff4-nrtll" Apr 16 16:50:01.260085 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:50:01.260056 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Apr 16 16:50:01.260121 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:50:01.260056 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/45604778-a27a-4c44-b245-0208bfb4154a-service-ca-bundle podName:45604778-a27a-4c44-b245-0208bfb4154a nodeName:}" failed. No retries permitted until 2026-04-16 16:50:17.2600372 +0000 UTC m=+156.321186730 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/45604778-a27a-4c44-b245-0208bfb4154a-service-ca-bundle") pod "router-default-85bf56ff4-nrtll" (UID: "45604778-a27a-4c44-b245-0208bfb4154a") : configmap references non-existent config key: service-ca.crt Apr 16 16:50:01.260121 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:50:01.260110 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45604778-a27a-4c44-b245-0208bfb4154a-metrics-certs podName:45604778-a27a-4c44-b245-0208bfb4154a nodeName:}" failed. No retries permitted until 2026-04-16 16:50:17.26009814 +0000 UTC m=+156.321247650 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/45604778-a27a-4c44-b245-0208bfb4154a-metrics-certs") pod "router-default-85bf56ff4-nrtll" (UID: "45604778-a27a-4c44-b245-0208bfb4154a") : secret "router-metrics-certs-default" not found Apr 16 16:50:17.183596 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:17.183518 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/12d57405-05cf-462d-a747-06c91ca773cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6667474d89-lqmw4\" (UID: \"12d57405-05cf-462d-a747-06c91ca773cb\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-lqmw4" Apr 16 16:50:17.186059 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:17.186040 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/12d57405-05cf-462d-a747-06c91ca773cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6667474d89-lqmw4\" (UID: \"12d57405-05cf-462d-a747-06c91ca773cb\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-lqmw4" Apr 16 16:50:17.284469 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:17.284418 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45604778-a27a-4c44-b245-0208bfb4154a-metrics-certs\") pod \"router-default-85bf56ff4-nrtll\" (UID: \"45604778-a27a-4c44-b245-0208bfb4154a\") " pod="openshift-ingress/router-default-85bf56ff4-nrtll" Apr 16 16:50:17.284469 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:17.284482 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45604778-a27a-4c44-b245-0208bfb4154a-service-ca-bundle\") pod \"router-default-85bf56ff4-nrtll\" (UID: \"45604778-a27a-4c44-b245-0208bfb4154a\") " pod="openshift-ingress/router-default-85bf56ff4-nrtll" Apr 16 16:50:17.285061 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:17.285035 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45604778-a27a-4c44-b245-0208bfb4154a-service-ca-bundle\") pod \"router-default-85bf56ff4-nrtll\" (UID: \"45604778-a27a-4c44-b245-0208bfb4154a\") " pod="openshift-ingress/router-default-85bf56ff4-nrtll" Apr 16 16:50:17.286914 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:17.286887 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45604778-a27a-4c44-b245-0208bfb4154a-metrics-certs\") pod \"router-default-85bf56ff4-nrtll\" (UID: \"45604778-a27a-4c44-b245-0208bfb4154a\") " pod="openshift-ingress/router-default-85bf56ff4-nrtll" Apr 16 16:50:17.419196 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:17.419151 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-lqmw4" Apr 16 16:50:17.532455 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:17.532429 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6667474d89-lqmw4"] Apr 16 16:50:17.532557 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:17.532538 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-85bf56ff4-nrtll" Apr 16 16:50:17.533919 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:50:17.533895 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12d57405_05cf_462d_a747_06c91ca773cb.slice/crio-7a5890e109645e9bad1c0d9a0908fea7d1660a6c7925583ad3e7455ce1262dab WatchSource:0}: Error finding container 7a5890e109645e9bad1c0d9a0908fea7d1660a6c7925583ad3e7455ce1262dab: Status 404 returned error can't find the container with id 7a5890e109645e9bad1c0d9a0908fea7d1660a6c7925583ad3e7455ce1262dab Apr 16 16:50:17.646519 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:17.646489 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-85bf56ff4-nrtll"] Apr 16 16:50:17.649660 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:50:17.649637 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45604778_a27a_4c44_b245_0208bfb4154a.slice/crio-4dbf321012886b3fbe3771852376a42a3dd2366563badfd971b60065ec9b7d93 WatchSource:0}: Error finding container 4dbf321012886b3fbe3771852376a42a3dd2366563badfd971b60065ec9b7d93: Status 404 returned error can't find the container with id 4dbf321012886b3fbe3771852376a42a3dd2366563badfd971b60065ec9b7d93 Apr 16 16:50:18.132528 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:18.132481 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-lqmw4" event={"ID":"12d57405-05cf-462d-a747-06c91ca773cb","Type":"ContainerStarted","Data":"7a5890e109645e9bad1c0d9a0908fea7d1660a6c7925583ad3e7455ce1262dab"} Apr 16 16:50:18.133592 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:18.133567 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-85bf56ff4-nrtll" event={"ID":"45604778-a27a-4c44-b245-0208bfb4154a","Type":"ContainerStarted","Data":"a1c7078b48e5252ea89c919c95e6e1683e07a8671c0d934d398c6588cc02a6df"} Apr 16 16:50:18.133694 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:18.133598 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-85bf56ff4-nrtll" event={"ID":"45604778-a27a-4c44-b245-0208bfb4154a","Type":"ContainerStarted","Data":"4dbf321012886b3fbe3771852376a42a3dd2366563badfd971b60065ec9b7d93"} Apr 16 16:50:18.150528 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:18.150490 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/router-default-85bf56ff4-nrtll" podStartSLOduration=33.150477893 podStartE2EDuration="33.150477893s" podCreationTimestamp="2026-04-16 16:49:45 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:50:18.150258412 +0000 UTC m=+157.211407934" watchObservedRunningTime="2026-04-16 16:50:18.150477893 +0000 UTC m=+157.211627424" Apr 16 16:50:18.533083 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:18.533000 2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-85bf56ff4-nrtll" Apr 16 16:50:18.535746 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:18.535719 2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-85bf56ff4-nrtll" Apr 16 16:50:18.919201 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:50:18.919150 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-rzx7h" podUID="b4e2d35c-1627-4f9c-bea6-985f08319902" Apr 16 16:50:18.948428 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:50:18.948376 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-xfnx6" podUID="efee44e6-2e2b-47f0-b356-6e77267f19e0" Apr 16 16:50:19.135616 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:19.135587 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rzx7h" Apr 16 16:50:19.135894 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:19.135871 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-85bf56ff4-nrtll" Apr 16 16:50:19.137294 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:19.137240 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-85bf56ff4-nrtll" Apr 16 16:50:20.139655 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.139619 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-lqmw4" event={"ID":"12d57405-05cf-462d-a747-06c91ca773cb","Type":"ContainerStarted","Data":"0aeac1a57ad877bd836bc0d7fbee6be61910ba14dc4b6dbf7c78eaf75336a5e1"} Apr 16 16:50:20.155754 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.155704 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-lqmw4" podStartSLOduration=33.436213483 podStartE2EDuration="35.155684459s" podCreationTimestamp="2026-04-16 16:49:45 +0000 UTC" firstStartedPulling="2026-04-16 16:50:17.536361525 +0000 UTC m=+156.597511042" lastFinishedPulling="2026-04-16 16:50:19.25583251 +0000 UTC m=+158.316982018" observedRunningTime="2026-04-16 16:50:20.155113694 +0000 UTC m=+159.216263225" watchObservedRunningTime="2026-04-16 16:50:20.155684459 +0000 UTC m=+159.216833990" Apr 16 16:50:20.201483 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.201450 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-747d5c778b-gxtt7"] Apr 16 16:50:20.204428 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.204406 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-747d5c778b-gxtt7" Apr 16 16:50:20.206480 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.206459 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-djqlx\"" Apr 16 16:50:20.206942 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.206922 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-installation-pull-secrets\") pod \"image-registry-747d5c778b-gxtt7\" (UID: \"0c8dc33b-f115-4614-a6f4-66d9f9fcfd58\") " pod="openshift-image-registry/image-registry-747d5c778b-gxtt7" Apr 16 16:50:20.207055 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.206963 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Apr 16 16:50:20.207055 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.207014 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-registry-tls\") pod \"image-registry-747d5c778b-gxtt7\" (UID: \"0c8dc33b-f115-4614-a6f4-66d9f9fcfd58\") " pod="openshift-image-registry/image-registry-747d5c778b-gxtt7" Apr 16 16:50:20.207167 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.207068 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-ca-trust-extracted\") pod \"image-registry-747d5c778b-gxtt7\" (UID: \"0c8dc33b-f115-4614-a6f4-66d9f9fcfd58\") " pod="openshift-image-registry/image-registry-747d5c778b-gxtt7" Apr 16 16:50:20.207167 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.207097 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-registry-certificates\") pod \"image-registry-747d5c778b-gxtt7\" (UID: \"0c8dc33b-f115-4614-a6f4-66d9f9fcfd58\") " pod="openshift-image-registry/image-registry-747d5c778b-gxtt7" Apr 16 16:50:20.207167 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.207125 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-trusted-ca\") pod \"image-registry-747d5c778b-gxtt7\" (UID: \"0c8dc33b-f115-4614-a6f4-66d9f9fcfd58\") " pod="openshift-image-registry/image-registry-747d5c778b-gxtt7" Apr 16 16:50:20.207167 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.207151 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-bound-sa-token\") pod \"image-registry-747d5c778b-gxtt7\" (UID: \"0c8dc33b-f115-4614-a6f4-66d9f9fcfd58\") " pod="openshift-image-registry/image-registry-747d5c778b-gxtt7" Apr 16 16:50:20.207362 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.207284 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d4tc\" (UniqueName: \"kubernetes.io/projected/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-kube-api-access-5d4tc\") pod \"image-registry-747d5c778b-gxtt7\" (UID: \"0c8dc33b-f115-4614-a6f4-66d9f9fcfd58\") " pod="openshift-image-registry/image-registry-747d5c778b-gxtt7" Apr 16 16:50:20.207413 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.207385 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-image-registry-private-configuration\") pod \"image-registry-747d5c778b-gxtt7\" (UID: \"0c8dc33b-f115-4614-a6f4-66d9f9fcfd58\") " pod="openshift-image-registry/image-registry-747d5c778b-gxtt7" Apr 16 16:50:20.208371 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.208350 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Apr 16 16:50:20.208655 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.208636 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Apr 16 16:50:20.213763 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.213740 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Apr 16 16:50:20.216894 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.216867 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-747d5c778b-gxtt7"] Apr 16 16:50:20.281606 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.281572 2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-747d5c778b-gxtt7"] Apr 16 16:50:20.281828 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:50:20.281807 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-5d4tc registry-certificates registry-tls trusted-ca], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-747d5c778b-gxtt7" podUID="0c8dc33b-f115-4614-a6f4-66d9f9fcfd58" Apr 16 16:50:20.307728 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.307688 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-ca-trust-extracted\") pod \"image-registry-747d5c778b-gxtt7\" (UID: \"0c8dc33b-f115-4614-a6f4-66d9f9fcfd58\") " pod="openshift-image-registry/image-registry-747d5c778b-gxtt7" Apr 16 16:50:20.308292 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.308264 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-registry-certificates\") pod \"image-registry-747d5c778b-gxtt7\" (UID: \"0c8dc33b-f115-4614-a6f4-66d9f9fcfd58\") " pod="openshift-image-registry/image-registry-747d5c778b-gxtt7" Apr 16 16:50:20.308292 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.308302 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-trusted-ca\") pod \"image-registry-747d5c778b-gxtt7\" (UID: \"0c8dc33b-f115-4614-a6f4-66d9f9fcfd58\") " pod="openshift-image-registry/image-registry-747d5c778b-gxtt7" Apr 16 16:50:20.308487 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.308333 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-bound-sa-token\") pod \"image-registry-747d5c778b-gxtt7\" (UID: \"0c8dc33b-f115-4614-a6f4-66d9f9fcfd58\") " pod="openshift-image-registry/image-registry-747d5c778b-gxtt7" Apr 16 16:50:20.308487 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.308348 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-ca-trust-extracted\") pod \"image-registry-747d5c778b-gxtt7\" (UID: \"0c8dc33b-f115-4614-a6f4-66d9f9fcfd58\") " pod="openshift-image-registry/image-registry-747d5c778b-gxtt7" Apr 16 16:50:20.308487 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.308358 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5d4tc\" (UniqueName: \"kubernetes.io/projected/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-kube-api-access-5d4tc\") pod \"image-registry-747d5c778b-gxtt7\" (UID: \"0c8dc33b-f115-4614-a6f4-66d9f9fcfd58\") " pod="openshift-image-registry/image-registry-747d5c778b-gxtt7" Apr 16 16:50:20.308487 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.308417 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-image-registry-private-configuration\") pod \"image-registry-747d5c778b-gxtt7\" (UID: \"0c8dc33b-f115-4614-a6f4-66d9f9fcfd58\") " pod="openshift-image-registry/image-registry-747d5c778b-gxtt7" Apr 16 16:50:20.308685 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.308513 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-installation-pull-secrets\") pod \"image-registry-747d5c778b-gxtt7\" (UID: \"0c8dc33b-f115-4614-a6f4-66d9f9fcfd58\") " pod="openshift-image-registry/image-registry-747d5c778b-gxtt7" Apr 16 16:50:20.308685 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.308566 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-registry-tls\") pod \"image-registry-747d5c778b-gxtt7\" (UID: \"0c8dc33b-f115-4614-a6f4-66d9f9fcfd58\") " pod="openshift-image-registry/image-registry-747d5c778b-gxtt7" Apr 16 16:50:20.309997 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.309672 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-rwt5j"] Apr 16 16:50:20.309997 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.309771 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-registry-certificates\") pod \"image-registry-747d5c778b-gxtt7\" (UID: \"0c8dc33b-f115-4614-a6f4-66d9f9fcfd58\") " pod="openshift-image-registry/image-registry-747d5c778b-gxtt7" Apr 16 16:50:20.310362 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.310292 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-trusted-ca\") pod \"image-registry-747d5c778b-gxtt7\" (UID: \"0c8dc33b-f115-4614-a6f4-66d9f9fcfd58\") " pod="openshift-image-registry/image-registry-747d5c778b-gxtt7" Apr 16 16:50:20.312553 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.312521 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-installation-pull-secrets\") pod \"image-registry-747d5c778b-gxtt7\" (UID: \"0c8dc33b-f115-4614-a6f4-66d9f9fcfd58\") " pod="openshift-image-registry/image-registry-747d5c778b-gxtt7" Apr 16 16:50:20.312643 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.312562 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-image-registry-private-configuration\") pod \"image-registry-747d5c778b-gxtt7\" (UID: \"0c8dc33b-f115-4614-a6f4-66d9f9fcfd58\") " pod="openshift-image-registry/image-registry-747d5c778b-gxtt7" Apr 16 16:50:20.312686 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.312645 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-registry-tls\") pod \"image-registry-747d5c778b-gxtt7\" (UID: \"0c8dc33b-f115-4614-a6f4-66d9f9fcfd58\") " pod="openshift-image-registry/image-registry-747d5c778b-gxtt7" Apr 16 16:50:20.313292 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.313271 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5cb6cf4cb4-rvpkl"] Apr 16 16:50:20.313400 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.313375 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-rwt5j" Apr 16 16:50:20.315415 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.315395 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-ddjpc\"" Apr 16 16:50:20.315974 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.315960 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5cb6cf4cb4-rvpkl" Apr 16 16:50:20.317535 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.317520 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Apr 16 16:50:20.317608 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.317542 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Apr 16 16:50:20.317928 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.317894 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Apr 16 16:50:20.318019 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.317988 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-9mmbk\"" Apr 16 16:50:20.322878 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.322861 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5cb6cf4cb4-rvpkl"] Apr 16 16:50:20.326827 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.326806 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bgr9k"] Apr 16 16:50:20.330409 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.330391 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bgr9k" Apr 16 16:50:20.337223 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.337186 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-rwt5j"] Apr 16 16:50:20.338128 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.337686 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Apr 16 16:50:20.338128 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.337747 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Apr 16 16:50:20.338128 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.337776 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Apr 16 16:50:20.338128 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.337687 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nbldh\"" Apr 16 16:50:20.338128 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.338003 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Apr 16 16:50:20.341015 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.340992 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-bound-sa-token\") pod \"image-registry-747d5c778b-gxtt7\" (UID: \"0c8dc33b-f115-4614-a6f4-66d9f9fcfd58\") " pod="openshift-image-registry/image-registry-747d5c778b-gxtt7" Apr 16 16:50:20.344058 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.343990 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bgr9k"] Apr 16 16:50:20.347911 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.347859 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d4tc\" (UniqueName: \"kubernetes.io/projected/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-kube-api-access-5d4tc\") pod \"image-registry-747d5c778b-gxtt7\" (UID: \"0c8dc33b-f115-4614-a6f4-66d9f9fcfd58\") " pod="openshift-image-registry/image-registry-747d5c778b-gxtt7" Apr 16 16:50:20.409360 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.409267 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/faaff24f-341b-422f-833f-902c51df7815-nginx-conf\") pod \"networking-console-plugin-5cb6cf4cb4-rvpkl\" (UID: \"faaff24f-341b-422f-833f-902c51df7815\") " pod="openshift-network-console/networking-console-plugin-5cb6cf4cb4-rvpkl" Apr 16 16:50:20.409360 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.409313 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b7a95004-7ff6-4be3-bb13-bb10aa6d5840-crio-socket\") pod \"insights-runtime-extractor-bgr9k\" (UID: \"b7a95004-7ff6-4be3-bb13-bb10aa6d5840\") " pod="openshift-insights/insights-runtime-extractor-bgr9k" Apr 16 16:50:20.409360 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.409365 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7a95004-7ff6-4be3-bb13-bb10aa6d5840-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bgr9k\" (UID: \"b7a95004-7ff6-4be3-bb13-bb10aa6d5840\") " pod="openshift-insights/insights-runtime-extractor-bgr9k" Apr 16 16:50:20.409630 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.409391 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5c8c501e-0685-46ea-a440-9417fdc93588-tls-certificates\") pod \"prometheus-operator-admission-webhook-9cb97cd87-rwt5j\" (UID: \"5c8c501e-0685-46ea-a440-9417fdc93588\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-rwt5j" Apr 16 16:50:20.409630 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.409459 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b7a95004-7ff6-4be3-bb13-bb10aa6d5840-data-volume\") pod \"insights-runtime-extractor-bgr9k\" (UID: \"b7a95004-7ff6-4be3-bb13-bb10aa6d5840\") " pod="openshift-insights/insights-runtime-extractor-bgr9k" Apr 16 16:50:20.409630 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.409502 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h74sd\" (UniqueName: \"kubernetes.io/projected/b7a95004-7ff6-4be3-bb13-bb10aa6d5840-kube-api-access-h74sd\") pod \"insights-runtime-extractor-bgr9k\" (UID: \"b7a95004-7ff6-4be3-bb13-bb10aa6d5840\") " pod="openshift-insights/insights-runtime-extractor-bgr9k" Apr 16 16:50:20.409630 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.409530 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/faaff24f-341b-422f-833f-902c51df7815-networking-console-plugin-cert\") pod \"networking-console-plugin-5cb6cf4cb4-rvpkl\" (UID: \"faaff24f-341b-422f-833f-902c51df7815\") " pod="openshift-network-console/networking-console-plugin-5cb6cf4cb4-rvpkl" Apr 16 16:50:20.409630 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.409619 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7a95004-7ff6-4be3-bb13-bb10aa6d5840-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bgr9k\" (UID: \"b7a95004-7ff6-4be3-bb13-bb10aa6d5840\") " pod="openshift-insights/insights-runtime-extractor-bgr9k" Apr 16 16:50:20.510519 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.510486 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b7a95004-7ff6-4be3-bb13-bb10aa6d5840-data-volume\") pod \"insights-runtime-extractor-bgr9k\" (UID: \"b7a95004-7ff6-4be3-bb13-bb10aa6d5840\") " pod="openshift-insights/insights-runtime-extractor-bgr9k" Apr 16 16:50:20.510519 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.510521 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h74sd\" (UniqueName: \"kubernetes.io/projected/b7a95004-7ff6-4be3-bb13-bb10aa6d5840-kube-api-access-h74sd\") pod \"insights-runtime-extractor-bgr9k\" (UID: \"b7a95004-7ff6-4be3-bb13-bb10aa6d5840\") " pod="openshift-insights/insights-runtime-extractor-bgr9k" Apr 16 16:50:20.510776 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.510542 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/faaff24f-341b-422f-833f-902c51df7815-networking-console-plugin-cert\") pod \"networking-console-plugin-5cb6cf4cb4-rvpkl\" (UID: \"faaff24f-341b-422f-833f-902c51df7815\") " pod="openshift-network-console/networking-console-plugin-5cb6cf4cb4-rvpkl" Apr 16 16:50:20.510776 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.510603 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7a95004-7ff6-4be3-bb13-bb10aa6d5840-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bgr9k\" (UID: \"b7a95004-7ff6-4be3-bb13-bb10aa6d5840\") " pod="openshift-insights/insights-runtime-extractor-bgr9k" Apr 16 16:50:20.510776 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.510621 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/faaff24f-341b-422f-833f-902c51df7815-nginx-conf\") pod \"networking-console-plugin-5cb6cf4cb4-rvpkl\" (UID: \"faaff24f-341b-422f-833f-902c51df7815\") " pod="openshift-network-console/networking-console-plugin-5cb6cf4cb4-rvpkl" Apr 16 16:50:20.510776 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.510642 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b7a95004-7ff6-4be3-bb13-bb10aa6d5840-crio-socket\") pod \"insights-runtime-extractor-bgr9k\" (UID: \"b7a95004-7ff6-4be3-bb13-bb10aa6d5840\") " pod="openshift-insights/insights-runtime-extractor-bgr9k" Apr 16 16:50:20.510776 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.510710 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b7a95004-7ff6-4be3-bb13-bb10aa6d5840-crio-socket\") pod \"insights-runtime-extractor-bgr9k\" (UID: \"b7a95004-7ff6-4be3-bb13-bb10aa6d5840\") " pod="openshift-insights/insights-runtime-extractor-bgr9k" Apr 16 16:50:20.510994 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.510786 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7a95004-7ff6-4be3-bb13-bb10aa6d5840-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bgr9k\" (UID: \"b7a95004-7ff6-4be3-bb13-bb10aa6d5840\") " pod="openshift-insights/insights-runtime-extractor-bgr9k" Apr 16 16:50:20.510994 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.510841 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5c8c501e-0685-46ea-a440-9417fdc93588-tls-certificates\") pod \"prometheus-operator-admission-webhook-9cb97cd87-rwt5j\" (UID: \"5c8c501e-0685-46ea-a440-9417fdc93588\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-rwt5j" Apr 16 16:50:20.510994 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.510918 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b7a95004-7ff6-4be3-bb13-bb10aa6d5840-data-volume\") pod \"insights-runtime-extractor-bgr9k\" (UID: \"b7a95004-7ff6-4be3-bb13-bb10aa6d5840\") " pod="openshift-insights/insights-runtime-extractor-bgr9k" Apr 16 16:50:20.511382 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.511357 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7a95004-7ff6-4be3-bb13-bb10aa6d5840-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bgr9k\" (UID: \"b7a95004-7ff6-4be3-bb13-bb10aa6d5840\") " pod="openshift-insights/insights-runtime-extractor-bgr9k" Apr 16 16:50:20.511691 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.511534 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/faaff24f-341b-422f-833f-902c51df7815-nginx-conf\") pod \"networking-console-plugin-5cb6cf4cb4-rvpkl\" (UID: \"faaff24f-341b-422f-833f-902c51df7815\") " pod="openshift-network-console/networking-console-plugin-5cb6cf4cb4-rvpkl" Apr 16 16:50:20.513494 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.513471 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7a95004-7ff6-4be3-bb13-bb10aa6d5840-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bgr9k\" (UID: \"b7a95004-7ff6-4be3-bb13-bb10aa6d5840\") " pod="openshift-insights/insights-runtime-extractor-bgr9k" Apr 16 16:50:20.513586 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.513473 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5c8c501e-0685-46ea-a440-9417fdc93588-tls-certificates\") pod \"prometheus-operator-admission-webhook-9cb97cd87-rwt5j\" (UID: \"5c8c501e-0685-46ea-a440-9417fdc93588\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-rwt5j" Apr 16 16:50:20.513586 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.513473 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/faaff24f-341b-422f-833f-902c51df7815-networking-console-plugin-cert\") pod \"networking-console-plugin-5cb6cf4cb4-rvpkl\" (UID: \"faaff24f-341b-422f-833f-902c51df7815\") " pod="openshift-network-console/networking-console-plugin-5cb6cf4cb4-rvpkl" Apr 16 16:50:20.524014 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.523989 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h74sd\" (UniqueName: \"kubernetes.io/projected/b7a95004-7ff6-4be3-bb13-bb10aa6d5840-kube-api-access-h74sd\") pod \"insights-runtime-extractor-bgr9k\" (UID: \"b7a95004-7ff6-4be3-bb13-bb10aa6d5840\") " pod="openshift-insights/insights-runtime-extractor-bgr9k" Apr 16 16:50:20.606502 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:50:20.606459 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-hk5gr" podUID="412718e2-2075-4c7b-a6a9-3b94ed57564c" Apr 16 16:50:20.635958 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.635923 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-rwt5j" Apr 16 16:50:20.659280 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.659234 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bgr9k" Apr 16 16:50:20.659855 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.659788 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5cb6cf4cb4-rvpkl" Apr 16 16:50:20.767217 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.767187 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-rwt5j"] Apr 16 16:50:20.774774 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:50:20.774727 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c8c501e_0685_46ea_a440_9417fdc93588.slice/crio-4bd01e907713bb9b4f0a7eada92a3bc502b781bb6a87b47139b407c6a06f95aa WatchSource:0}: Error finding container 4bd01e907713bb9b4f0a7eada92a3bc502b781bb6a87b47139b407c6a06f95aa: Status 404 returned error can't find the container with id 4bd01e907713bb9b4f0a7eada92a3bc502b781bb6a87b47139b407c6a06f95aa Apr 16 16:50:20.802516 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.802490 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bgr9k"] Apr 16 16:50:20.805489 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:50:20.805462 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7a95004_7ff6_4be3_bb13_bb10aa6d5840.slice/crio-5188910749c076bde51713f26bebbeb8aac1b2b29733d4742c26a9cf410e2f10 WatchSource:0}: Error finding container 5188910749c076bde51713f26bebbeb8aac1b2b29733d4742c26a9cf410e2f10: Status 404 returned error can't find the container with id 5188910749c076bde51713f26bebbeb8aac1b2b29733d4742c26a9cf410e2f10 Apr 16 16:50:20.816273 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:20.816123 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5cb6cf4cb4-rvpkl"] Apr 16 16:50:20.821473 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:50:20.821450 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaaff24f_341b_422f_833f_902c51df7815.slice/crio-e9f993c356bfd9a850a8437bd3c02323b20c4acf157862f65ce7a7a67eaad370 WatchSource:0}: Error finding container e9f993c356bfd9a850a8437bd3c02323b20c4acf157862f65ce7a7a67eaad370: Status 404 returned error can't find the container with id e9f993c356bfd9a850a8437bd3c02323b20c4acf157862f65ce7a7a67eaad370 Apr 16 16:50:21.143372 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:21.143334 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bgr9k" event={"ID":"b7a95004-7ff6-4be3-bb13-bb10aa6d5840","Type":"ContainerStarted","Data":"125c2646a552e6d2a582bd47fcb51f89b6b67632425d3e23c3ee4dee18172933"} Apr 16 16:50:21.143372 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:21.143369 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bgr9k" event={"ID":"b7a95004-7ff6-4be3-bb13-bb10aa6d5840","Type":"ContainerStarted","Data":"5188910749c076bde51713f26bebbeb8aac1b2b29733d4742c26a9cf410e2f10"} Apr 16 16:50:21.144318 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:21.144295 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5cb6cf4cb4-rvpkl" event={"ID":"faaff24f-341b-422f-833f-902c51df7815","Type":"ContainerStarted","Data":"e9f993c356bfd9a850a8437bd3c02323b20c4acf157862f65ce7a7a67eaad370"} Apr 16 16:50:21.145194 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:21.145172 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-rwt5j" event={"ID":"5c8c501e-0685-46ea-a440-9417fdc93588","Type":"ContainerStarted","Data":"4bd01e907713bb9b4f0a7eada92a3bc502b781bb6a87b47139b407c6a06f95aa"} Apr 16 16:50:21.145296 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:21.145268 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-747d5c778b-gxtt7" Apr 16 16:50:21.149223 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:21.149205 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-747d5c778b-gxtt7" Apr 16 16:50:21.216612 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:21.216574 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-registry-certificates\") pod \"0c8dc33b-f115-4614-a6f4-66d9f9fcfd58\" (UID: \"0c8dc33b-f115-4614-a6f4-66d9f9fcfd58\") " Apr 16 16:50:21.216790 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:21.216623 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-installation-pull-secrets\") pod \"0c8dc33b-f115-4614-a6f4-66d9f9fcfd58\" (UID: \"0c8dc33b-f115-4614-a6f4-66d9f9fcfd58\") " Apr 16 16:50:21.216790 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:21.216652 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-ca-trust-extracted\") pod \"0c8dc33b-f115-4614-a6f4-66d9f9fcfd58\" (UID: \"0c8dc33b-f115-4614-a6f4-66d9f9fcfd58\") " Apr 16 16:50:21.216790 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:21.216676 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5d4tc\" (UniqueName: \"kubernetes.io/projected/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-kube-api-access-5d4tc\") pod \"0c8dc33b-f115-4614-a6f4-66d9f9fcfd58\" (UID: \"0c8dc33b-f115-4614-a6f4-66d9f9fcfd58\") " Apr 16 16:50:21.216790 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:21.216699 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-image-registry-private-configuration\") pod \"0c8dc33b-f115-4614-a6f4-66d9f9fcfd58\" (UID: \"0c8dc33b-f115-4614-a6f4-66d9f9fcfd58\") " Apr 16 16:50:21.216790 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:21.216731 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-registry-tls\") pod \"0c8dc33b-f115-4614-a6f4-66d9f9fcfd58\" (UID: \"0c8dc33b-f115-4614-a6f4-66d9f9fcfd58\") " Apr 16 16:50:21.216790 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:21.216756 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-bound-sa-token\") pod \"0c8dc33b-f115-4614-a6f4-66d9f9fcfd58\" (UID: \"0c8dc33b-f115-4614-a6f4-66d9f9fcfd58\") " Apr 16 16:50:21.216790 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:21.216781 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-trusted-ca\") pod \"0c8dc33b-f115-4614-a6f4-66d9f9fcfd58\" (UID: \"0c8dc33b-f115-4614-a6f4-66d9f9fcfd58\") " Apr 16 16:50:21.217115 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:21.217043 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "0c8dc33b-f115-4614-a6f4-66d9f9fcfd58" (UID: "0c8dc33b-f115-4614-a6f4-66d9f9fcfd58"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 16 16:50:21.217115 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:21.217070 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "0c8dc33b-f115-4614-a6f4-66d9f9fcfd58" (UID: "0c8dc33b-f115-4614-a6f4-66d9f9fcfd58"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 16:50:21.217222 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:21.217123 2576 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-ca-trust-extracted\") on node \"ip-10-0-142-37.ec2.internal\" DevicePath \"\"" Apr 16 16:50:21.217884 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:21.217839 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "0c8dc33b-f115-4614-a6f4-66d9f9fcfd58" (UID: "0c8dc33b-f115-4614-a6f4-66d9f9fcfd58"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 16:50:21.219307 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:21.219283 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "0c8dc33b-f115-4614-a6f4-66d9f9fcfd58" (UID: "0c8dc33b-f115-4614-a6f4-66d9f9fcfd58"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 16:50:21.219401 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:21.219374 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-kube-api-access-5d4tc" (OuterVolumeSpecName: "kube-api-access-5d4tc") pod "0c8dc33b-f115-4614-a6f4-66d9f9fcfd58" (UID: "0c8dc33b-f115-4614-a6f4-66d9f9fcfd58"). InnerVolumeSpecName "kube-api-access-5d4tc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 16:50:21.219570 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:21.219551 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "0c8dc33b-f115-4614-a6f4-66d9f9fcfd58" (UID: "0c8dc33b-f115-4614-a6f4-66d9f9fcfd58"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 16:50:21.219792 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:21.219773 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "0c8dc33b-f115-4614-a6f4-66d9f9fcfd58" (UID: "0c8dc33b-f115-4614-a6f4-66d9f9fcfd58"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 16:50:21.219792 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:21.219788 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "0c8dc33b-f115-4614-a6f4-66d9f9fcfd58" (UID: "0c8dc33b-f115-4614-a6f4-66d9f9fcfd58"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 16:50:21.317600 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:21.317562 2576 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-installation-pull-secrets\") on node \"ip-10-0-142-37.ec2.internal\" DevicePath \"\"" Apr 16 16:50:21.317600 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:21.317604 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d4tc\" (UniqueName: \"kubernetes.io/projected/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-kube-api-access-5d4tc\") on node \"ip-10-0-142-37.ec2.internal\" DevicePath \"\"" Apr 16 16:50:21.317834 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:21.317618 2576 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-image-registry-private-configuration\") on node \"ip-10-0-142-37.ec2.internal\" DevicePath \"\"" Apr 16 16:50:21.317834 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:21.317633 2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-registry-tls\") on node \"ip-10-0-142-37.ec2.internal\" DevicePath \"\"" Apr 16 16:50:21.317834 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:21.317646 2576 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-bound-sa-token\") on node \"ip-10-0-142-37.ec2.internal\" DevicePath \"\"" Apr 16 16:50:21.317834 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:21.317658 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-trusted-ca\") on node \"ip-10-0-142-37.ec2.internal\" DevicePath \"\"" Apr 16 16:50:21.317834 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:21.317669 2576 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58-registry-certificates\") on node \"ip-10-0-142-37.ec2.internal\" DevicePath \"\"" Apr 16 16:50:22.149741 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:22.149693 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5cb6cf4cb4-rvpkl" event={"ID":"faaff24f-341b-422f-833f-902c51df7815","Type":"ContainerStarted","Data":"412b9f7b50f0871b91c8594915c417ccc90be1c579bd967028d82121bcedd6ba"} Apr 16 16:50:22.151321 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:22.151274 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-rwt5j" event={"ID":"5c8c501e-0685-46ea-a440-9417fdc93588","Type":"ContainerStarted","Data":"e3fa6fe1ec3d6ceb31430afad0990605f882bdcc597e656852ef8bd0615e44b6"} Apr 16 16:50:22.151505 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:22.151469 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-rwt5j" Apr 16 16:50:22.152911 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:22.152883 2576 patch_prober.go:28] interesting pod/prometheus-operator-admission-webhook-9cb97cd87-rwt5j container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.132.0.13:8443/healthz\": dial tcp 10.132.0.13:8443: connect: connection refused" start-of-body= Apr 16 16:50:22.153019 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:22.152933 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-rwt5j" podUID="5c8c501e-0685-46ea-a440-9417fdc93588" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.13:8443/healthz\": dial tcp 10.132.0.13:8443: connect: connection refused" Apr 16 16:50:22.154045 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:22.154017 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bgr9k" event={"ID":"b7a95004-7ff6-4be3-bb13-bb10aa6d5840","Type":"ContainerStarted","Data":"661056d51991ea9a928c08ac001d7842eac20889bea213913791045bc9b49453"} Apr 16 16:50:22.154125 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:22.154063 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-747d5c778b-gxtt7" Apr 16 16:50:22.162712 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:22.162662 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5cb6cf4cb4-rvpkl" podStartSLOduration=0.918568301 podStartE2EDuration="2.162647567s" podCreationTimestamp="2026-04-16 16:50:20 +0000 UTC" firstStartedPulling="2026-04-16 16:50:20.823235221 +0000 UTC m=+159.884384730" lastFinishedPulling="2026-04-16 16:50:22.067314487 +0000 UTC m=+161.128463996" observedRunningTime="2026-04-16 16:50:22.162263585 +0000 UTC m=+161.223413110" watchObservedRunningTime="2026-04-16 16:50:22.162647567 +0000 UTC m=+161.223797099" Apr 16 16:50:22.188469 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:22.188413 2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-747d5c778b-gxtt7"] Apr 16 16:50:22.192422 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:22.192364 2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-747d5c778b-gxtt7"] Apr 16 16:50:22.206125 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:22.204897 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-rwt5j" podStartSLOduration=0.916059128 podStartE2EDuration="2.204881312s" podCreationTimestamp="2026-04-16 16:50:20 +0000 UTC" firstStartedPulling="2026-04-16 16:50:20.776919959 +0000 UTC m=+159.838069467" lastFinishedPulling="2026-04-16 16:50:22.065742135 +0000 UTC m=+161.126891651" observedRunningTime="2026-04-16 16:50:22.204560249 +0000 UTC m=+161.265709781" watchObservedRunningTime="2026-04-16 16:50:22.204881312 +0000 UTC m=+161.266030848" Apr 16 16:50:23.158260 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:23.158202 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bgr9k" event={"ID":"b7a95004-7ff6-4be3-bb13-bb10aa6d5840","Type":"ContainerStarted","Data":"aa4a2a161f94a025f57eaf38e9e19bd829d824b0e8d49f3beb1019f3eb218e5a"} Apr 16 16:50:23.162240 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:23.162219 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-rwt5j" Apr 16 16:50:23.175017 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:23.174978 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bgr9k" podStartSLOduration=1.014110657 podStartE2EDuration="3.174967298s" podCreationTimestamp="2026-04-16 16:50:20 +0000 UTC" firstStartedPulling="2026-04-16 16:50:20.858434798 +0000 UTC m=+159.919584307" lastFinishedPulling="2026-04-16 16:50:23.01929144 +0000 UTC m=+162.080440948" observedRunningTime="2026-04-16 16:50:23.174415819 +0000 UTC m=+162.235565362" watchObservedRunningTime="2026-04-16 16:50:23.174967298 +0000 UTC m=+162.236116829" Apr 16 16:50:23.592155 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:23.592120 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c8dc33b-f115-4614-a6f4-66d9f9fcfd58" path="/var/lib/kubelet/pods/0c8dc33b-f115-4614-a6f4-66d9f9fcfd58/volumes" Apr 16 16:50:23.825905 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:23.825873 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-78f957474d-8qhj4"] Apr 16 16:50:23.829073 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:23.829059 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-78f957474d-8qhj4" Apr 16 16:50:23.831079 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:23.831058 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-fd89t\"" Apr 16 16:50:23.831599 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:23.831576 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Apr 16 16:50:23.831699 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:23.831609 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Apr 16 16:50:23.831699 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:23.831580 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Apr 16 16:50:23.837274 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:23.837236 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-78f957474d-8qhj4"] Apr 16 16:50:23.839959 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:23.839936 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a627f9f0-08a8-49a0-a0f6-34229f6c3330-prometheus-operator-tls\") pod \"prometheus-operator-78f957474d-8qhj4\" (UID: \"a627f9f0-08a8-49a0-a0f6-34229f6c3330\") " pod="openshift-monitoring/prometheus-operator-78f957474d-8qhj4" Apr 16 16:50:23.840056 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:23.839971 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a627f9f0-08a8-49a0-a0f6-34229f6c3330-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-78f957474d-8qhj4\" (UID: \"a627f9f0-08a8-49a0-a0f6-34229f6c3330\") " pod="openshift-monitoring/prometheus-operator-78f957474d-8qhj4" Apr 16 16:50:23.840056 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:23.839995 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm2cf\" (UniqueName: \"kubernetes.io/projected/a627f9f0-08a8-49a0-a0f6-34229f6c3330-kube-api-access-cm2cf\") pod \"prometheus-operator-78f957474d-8qhj4\" (UID: \"a627f9f0-08a8-49a0-a0f6-34229f6c3330\") " pod="openshift-monitoring/prometheus-operator-78f957474d-8qhj4" Apr 16 16:50:23.840056 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:23.840028 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/efee44e6-2e2b-47f0-b356-6e77267f19e0-cert\") pod \"ingress-canary-xfnx6\" (UID: \"efee44e6-2e2b-47f0-b356-6e77267f19e0\") " pod="openshift-ingress-canary/ingress-canary-xfnx6" Apr 16 16:50:23.840193 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:23.840063 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a627f9f0-08a8-49a0-a0f6-34229f6c3330-metrics-client-ca\") pod \"prometheus-operator-78f957474d-8qhj4\" (UID: \"a627f9f0-08a8-49a0-a0f6-34229f6c3330\") " pod="openshift-monitoring/prometheus-operator-78f957474d-8qhj4" Apr 16 16:50:23.840193 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:23.840102 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4e2d35c-1627-4f9c-bea6-985f08319902-metrics-tls\") pod \"dns-default-rzx7h\" (UID: \"b4e2d35c-1627-4f9c-bea6-985f08319902\") " pod="openshift-dns/dns-default-rzx7h" Apr 16 16:50:23.842623 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:23.842573 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/efee44e6-2e2b-47f0-b356-6e77267f19e0-cert\") pod \"ingress-canary-xfnx6\" (UID: \"efee44e6-2e2b-47f0-b356-6e77267f19e0\") " pod="openshift-ingress-canary/ingress-canary-xfnx6" Apr 16 16:50:23.842767 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:23.842752 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4e2d35c-1627-4f9c-bea6-985f08319902-metrics-tls\") pod \"dns-default-rzx7h\" (UID: \"b4e2d35c-1627-4f9c-bea6-985f08319902\") " pod="openshift-dns/dns-default-rzx7h" Apr 16 16:50:23.938715 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:23.938685 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-99nwp\"" Apr 16 16:50:23.940710 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:23.940688 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a627f9f0-08a8-49a0-a0f6-34229f6c3330-metrics-client-ca\") pod \"prometheus-operator-78f957474d-8qhj4\" (UID: \"a627f9f0-08a8-49a0-a0f6-34229f6c3330\") " pod="openshift-monitoring/prometheus-operator-78f957474d-8qhj4" Apr 16 16:50:23.940804 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:23.940736 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a627f9f0-08a8-49a0-a0f6-34229f6c3330-prometheus-operator-tls\") pod \"prometheus-operator-78f957474d-8qhj4\" (UID: \"a627f9f0-08a8-49a0-a0f6-34229f6c3330\") " pod="openshift-monitoring/prometheus-operator-78f957474d-8qhj4" Apr 16 16:50:23.940804 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:23.940757 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a627f9f0-08a8-49a0-a0f6-34229f6c3330-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-78f957474d-8qhj4\" (UID: \"a627f9f0-08a8-49a0-a0f6-34229f6c3330\") " pod="openshift-monitoring/prometheus-operator-78f957474d-8qhj4" Apr 16 16:50:23.940804 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:23.940794 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cm2cf\" (UniqueName: \"kubernetes.io/projected/a627f9f0-08a8-49a0-a0f6-34229f6c3330-kube-api-access-cm2cf\") pod \"prometheus-operator-78f957474d-8qhj4\" (UID: \"a627f9f0-08a8-49a0-a0f6-34229f6c3330\") " pod="openshift-monitoring/prometheus-operator-78f957474d-8qhj4" Apr 16 16:50:23.940952 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:50:23.940845 2576 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Apr 16 16:50:23.940952 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:50:23.940910 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a627f9f0-08a8-49a0-a0f6-34229f6c3330-prometheus-operator-tls podName:a627f9f0-08a8-49a0-a0f6-34229f6c3330 nodeName:}" failed. No retries permitted until 2026-04-16 16:50:24.440891786 +0000 UTC m=+163.502041321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/a627f9f0-08a8-49a0-a0f6-34229f6c3330-prometheus-operator-tls") pod "prometheus-operator-78f957474d-8qhj4" (UID: "a627f9f0-08a8-49a0-a0f6-34229f6c3330") : secret "prometheus-operator-tls" not found Apr 16 16:50:23.941480 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:23.941463 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a627f9f0-08a8-49a0-a0f6-34229f6c3330-metrics-client-ca\") pod \"prometheus-operator-78f957474d-8qhj4\" (UID: \"a627f9f0-08a8-49a0-a0f6-34229f6c3330\") " pod="openshift-monitoring/prometheus-operator-78f957474d-8qhj4" Apr 16 16:50:23.943177 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:23.943162 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a627f9f0-08a8-49a0-a0f6-34229f6c3330-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-78f957474d-8qhj4\" (UID: \"a627f9f0-08a8-49a0-a0f6-34229f6c3330\") " pod="openshift-monitoring/prometheus-operator-78f957474d-8qhj4" Apr 16 16:50:23.947088 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:23.947070 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rzx7h" Apr 16 16:50:23.951353 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:23.951331 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm2cf\" (UniqueName: \"kubernetes.io/projected/a627f9f0-08a8-49a0-a0f6-34229f6c3330-kube-api-access-cm2cf\") pod \"prometheus-operator-78f957474d-8qhj4\" (UID: \"a627f9f0-08a8-49a0-a0f6-34229f6c3330\") " pod="openshift-monitoring/prometheus-operator-78f957474d-8qhj4" Apr 16 16:50:24.061064 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:24.061036 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rzx7h"] Apr 16 16:50:24.063684 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:50:24.063656 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e2d35c_1627_4f9c_bea6_985f08319902.slice/crio-b430d14563c1780a5ac252971750af9504f0972dca303be66c9f97b06fb64ab1 WatchSource:0}: Error finding container b430d14563c1780a5ac252971750af9504f0972dca303be66c9f97b06fb64ab1: Status 404 returned error can't find the container with id b430d14563c1780a5ac252971750af9504f0972dca303be66c9f97b06fb64ab1 Apr 16 16:50:24.166152 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:24.166058 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rzx7h" event={"ID":"b4e2d35c-1627-4f9c-bea6-985f08319902","Type":"ContainerStarted","Data":"b430d14563c1780a5ac252971750af9504f0972dca303be66c9f97b06fb64ab1"} Apr 16 16:50:24.444814 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:24.444718 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a627f9f0-08a8-49a0-a0f6-34229f6c3330-prometheus-operator-tls\") pod \"prometheus-operator-78f957474d-8qhj4\" (UID: \"a627f9f0-08a8-49a0-a0f6-34229f6c3330\") " pod="openshift-monitoring/prometheus-operator-78f957474d-8qhj4" Apr 16 16:50:24.447646 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:24.447617 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a627f9f0-08a8-49a0-a0f6-34229f6c3330-prometheus-operator-tls\") pod \"prometheus-operator-78f957474d-8qhj4\" (UID: \"a627f9f0-08a8-49a0-a0f6-34229f6c3330\") " pod="openshift-monitoring/prometheus-operator-78f957474d-8qhj4" Apr 16 16:50:24.738235 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:24.738151 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-78f957474d-8qhj4" Apr 16 16:50:24.868500 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:24.868468 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-78f957474d-8qhj4"] Apr 16 16:50:25.157987 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:50:25.157952 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda627f9f0_08a8_49a0_a0f6_34229f6c3330.slice/crio-fe782e54ebf422fc75f18878cbf67f26311e4cbd3758619355e08566c09e19f8 WatchSource:0}: Error finding container fe782e54ebf422fc75f18878cbf67f26311e4cbd3758619355e08566c09e19f8: Status 404 returned error can't find the container with id fe782e54ebf422fc75f18878cbf67f26311e4cbd3758619355e08566c09e19f8 Apr 16 16:50:25.169600 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:25.169568 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-78f957474d-8qhj4" event={"ID":"a627f9f0-08a8-49a0-a0f6-34229f6c3330","Type":"ContainerStarted","Data":"fe782e54ebf422fc75f18878cbf67f26311e4cbd3758619355e08566c09e19f8"} Apr 16 16:50:26.174886 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:26.174846 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rzx7h" event={"ID":"b4e2d35c-1627-4f9c-bea6-985f08319902","Type":"ContainerStarted","Data":"cf10bbcf6411276d8efb5f9aeaf43fe2b599c3070fe92963c02978b87e3a9bbc"} Apr 16 16:50:26.174886 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:26.174893 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rzx7h" event={"ID":"b4e2d35c-1627-4f9c-bea6-985f08319902","Type":"ContainerStarted","Data":"ecd0b1f968cf645e6e56a550d0a4eff7d165c3dcb3645c3deef84b551ef76b70"} Apr 16 16:50:26.175405 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:26.175005 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rzx7h" Apr 16 16:50:26.190296 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:26.190239 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-rzx7h" podStartSLOduration=130.05300155 podStartE2EDuration="2m11.190224339s" podCreationTimestamp="2026-04-16 16:48:15 +0000 UTC" firstStartedPulling="2026-04-16 16:50:24.065365641 +0000 UTC m=+163.126515150" lastFinishedPulling="2026-04-16 16:50:25.20258843 +0000 UTC m=+164.263737939" observedRunningTime="2026-04-16 16:50:26.190169912 +0000 UTC m=+165.251319446" watchObservedRunningTime="2026-04-16 16:50:26.190224339 +0000 UTC m=+165.251373873" Apr 16 16:50:27.179160 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:27.179123 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-78f957474d-8qhj4" event={"ID":"a627f9f0-08a8-49a0-a0f6-34229f6c3330","Type":"ContainerStarted","Data":"e4e05488356520929022bcf37b9289dbdc5b381125244f8d99ceb4851f374f5b"} Apr 16 16:50:27.179160 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:27.179166 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-78f957474d-8qhj4" event={"ID":"a627f9f0-08a8-49a0-a0f6-34229f6c3330","Type":"ContainerStarted","Data":"355e646c892326148ad5ab64cdc3ae997252eee1c31de7700bf5a978257aa841"} Apr 16 16:50:27.195102 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:27.195052 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-78f957474d-8qhj4" podStartSLOduration=2.959138651 podStartE2EDuration="4.195031764s" podCreationTimestamp="2026-04-16 16:50:23 +0000 UTC" firstStartedPulling="2026-04-16 16:50:25.159837094 +0000 UTC m=+164.220986618" lastFinishedPulling="2026-04-16 16:50:26.395730213 +0000 UTC m=+165.456879731" observedRunningTime="2026-04-16 16:50:27.193859271 +0000 UTC m=+166.255008814" watchObservedRunningTime="2026-04-16 16:50:27.195031764 +0000 UTC m=+166.256181314" Apr 16 16:50:28.567935 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:28.567903 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-6fb554db88-dcz4j"] Apr 16 16:50:28.571208 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:28.571192 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fb554db88-dcz4j" Apr 16 16:50:28.573434 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:28.573404 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Apr 16 16:50:28.573552 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:28.573436 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Apr 16 16:50:28.573596 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:28.573585 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Apr 16 16:50:28.574138 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:28.574123 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Apr 16 16:50:28.574316 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:28.574154 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Apr 16 16:50:28.574316 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:28.574216 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Apr 16 16:50:28.574316 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:28.574265 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-q9pts\"" Apr 16 16:50:28.574316 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:28.574272 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Apr 16 16:50:28.581718 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:28.581688 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fb554db88-dcz4j"] Apr 16 16:50:28.582672 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:28.582649 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Apr 16 16:50:28.678459 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:28.678420 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d9783fe-808a-4862-81c1-31dcf22df1a4-console-oauth-config\") pod \"console-6fb554db88-dcz4j\" (UID: \"1d9783fe-808a-4862-81c1-31dcf22df1a4\") " pod="openshift-console/console-6fb554db88-dcz4j" Apr 16 16:50:28.678459 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:28.678457 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d9783fe-808a-4862-81c1-31dcf22df1a4-service-ca\") pod \"console-6fb554db88-dcz4j\" (UID: \"1d9783fe-808a-4862-81c1-31dcf22df1a4\") " pod="openshift-console/console-6fb554db88-dcz4j" Apr 16 16:50:28.678656 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:28.678477 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d9783fe-808a-4862-81c1-31dcf22df1a4-console-serving-cert\") pod \"console-6fb554db88-dcz4j\" (UID: \"1d9783fe-808a-4862-81c1-31dcf22df1a4\") " pod="openshift-console/console-6fb554db88-dcz4j" Apr 16 16:50:28.678656 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:28.678493 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d9783fe-808a-4862-81c1-31dcf22df1a4-oauth-serving-cert\") pod \"console-6fb554db88-dcz4j\" (UID: \"1d9783fe-808a-4862-81c1-31dcf22df1a4\") " pod="openshift-console/console-6fb554db88-dcz4j" Apr 16 16:50:28.678656 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:28.678561 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d9783fe-808a-4862-81c1-31dcf22df1a4-console-config\") pod \"console-6fb554db88-dcz4j\" (UID: \"1d9783fe-808a-4862-81c1-31dcf22df1a4\") " pod="openshift-console/console-6fb554db88-dcz4j" Apr 16 16:50:28.678656 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:28.678584 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c69qs\" (UniqueName: \"kubernetes.io/projected/1d9783fe-808a-4862-81c1-31dcf22df1a4-kube-api-access-c69qs\") pod \"console-6fb554db88-dcz4j\" (UID: \"1d9783fe-808a-4862-81c1-31dcf22df1a4\") " pod="openshift-console/console-6fb554db88-dcz4j" Apr 16 16:50:28.678656 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:28.678606 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d9783fe-808a-4862-81c1-31dcf22df1a4-trusted-ca-bundle\") pod \"console-6fb554db88-dcz4j\" (UID: \"1d9783fe-808a-4862-81c1-31dcf22df1a4\") " pod="openshift-console/console-6fb554db88-dcz4j" Apr 16 16:50:28.779188 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:28.779157 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d9783fe-808a-4862-81c1-31dcf22df1a4-console-config\") pod \"console-6fb554db88-dcz4j\" (UID: \"1d9783fe-808a-4862-81c1-31dcf22df1a4\") " pod="openshift-console/console-6fb554db88-dcz4j" Apr 16 16:50:28.779275 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:28.779200 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c69qs\" (UniqueName: \"kubernetes.io/projected/1d9783fe-808a-4862-81c1-31dcf22df1a4-kube-api-access-c69qs\") pod \"console-6fb554db88-dcz4j\" (UID: \"1d9783fe-808a-4862-81c1-31dcf22df1a4\") " pod="openshift-console/console-6fb554db88-dcz4j" Apr 16 16:50:28.779275 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:28.779221 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d9783fe-808a-4862-81c1-31dcf22df1a4-trusted-ca-bundle\") pod \"console-6fb554db88-dcz4j\" (UID: \"1d9783fe-808a-4862-81c1-31dcf22df1a4\") " pod="openshift-console/console-6fb554db88-dcz4j" Apr 16 16:50:28.779346 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:28.779284 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d9783fe-808a-4862-81c1-31dcf22df1a4-console-oauth-config\") pod \"console-6fb554db88-dcz4j\" (UID: \"1d9783fe-808a-4862-81c1-31dcf22df1a4\") " pod="openshift-console/console-6fb554db88-dcz4j" Apr 16 16:50:28.779346 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:28.779300 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d9783fe-808a-4862-81c1-31dcf22df1a4-service-ca\") pod \"console-6fb554db88-dcz4j\" (UID: \"1d9783fe-808a-4862-81c1-31dcf22df1a4\") " pod="openshift-console/console-6fb554db88-dcz4j" Apr 16 16:50:28.779346 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:28.779316 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d9783fe-808a-4862-81c1-31dcf22df1a4-console-serving-cert\") pod \"console-6fb554db88-dcz4j\" (UID: \"1d9783fe-808a-4862-81c1-31dcf22df1a4\") " pod="openshift-console/console-6fb554db88-dcz4j" Apr 16 16:50:28.779346 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:28.779338 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d9783fe-808a-4862-81c1-31dcf22df1a4-oauth-serving-cert\") pod \"console-6fb554db88-dcz4j\" (UID: \"1d9783fe-808a-4862-81c1-31dcf22df1a4\") " pod="openshift-console/console-6fb554db88-dcz4j" Apr 16 16:50:28.780032 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:28.779998 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d9783fe-808a-4862-81c1-31dcf22df1a4-console-config\") pod \"console-6fb554db88-dcz4j\" (UID: \"1d9783fe-808a-4862-81c1-31dcf22df1a4\") " pod="openshift-console/console-6fb554db88-dcz4j" Apr 16 16:50:28.780160 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:28.780048 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d9783fe-808a-4862-81c1-31dcf22df1a4-oauth-serving-cert\") pod \"console-6fb554db88-dcz4j\" (UID: \"1d9783fe-808a-4862-81c1-31dcf22df1a4\") " pod="openshift-console/console-6fb554db88-dcz4j" Apr 16 16:50:28.780160 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:28.780085 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d9783fe-808a-4862-81c1-31dcf22df1a4-service-ca\") pod \"console-6fb554db88-dcz4j\" (UID: \"1d9783fe-808a-4862-81c1-31dcf22df1a4\") " pod="openshift-console/console-6fb554db88-dcz4j" Apr 16 16:50:28.780285 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:28.780205 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d9783fe-808a-4862-81c1-31dcf22df1a4-trusted-ca-bundle\") pod \"console-6fb554db88-dcz4j\" (UID: \"1d9783fe-808a-4862-81c1-31dcf22df1a4\") " pod="openshift-console/console-6fb554db88-dcz4j" Apr 16 16:50:28.782027 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:28.781994 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d9783fe-808a-4862-81c1-31dcf22df1a4-console-oauth-config\") pod \"console-6fb554db88-dcz4j\" (UID: \"1d9783fe-808a-4862-81c1-31dcf22df1a4\") " pod="openshift-console/console-6fb554db88-dcz4j" Apr 16 16:50:28.782123 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:28.782047 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d9783fe-808a-4862-81c1-31dcf22df1a4-console-serving-cert\") pod \"console-6fb554db88-dcz4j\" (UID: \"1d9783fe-808a-4862-81c1-31dcf22df1a4\") " pod="openshift-console/console-6fb554db88-dcz4j" Apr 16 16:50:28.787092 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:28.787072 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c69qs\" (UniqueName: \"kubernetes.io/projected/1d9783fe-808a-4862-81c1-31dcf22df1a4-kube-api-access-c69qs\") pod \"console-6fb554db88-dcz4j\" (UID: \"1d9783fe-808a-4862-81c1-31dcf22df1a4\") " pod="openshift-console/console-6fb554db88-dcz4j" Apr 16 16:50:28.885125 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:28.885046 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fb554db88-dcz4j" Apr 16 16:50:29.022384 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.022350 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fb554db88-dcz4j"] Apr 16 16:50:29.025421 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:50:29.025346 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d9783fe_808a_4862_81c1_31dcf22df1a4.slice/crio-a533715623674db382a8ac1871dc17464a08b689d5e875af35a22f9697561b6d WatchSource:0}: Error finding container a533715623674db382a8ac1871dc17464a08b689d5e875af35a22f9697561b6d: Status 404 returned error can't find the container with id a533715623674db382a8ac1871dc17464a08b689d5e875af35a22f9697561b6d Apr 16 16:50:29.185237 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.185159 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fb554db88-dcz4j" event={"ID":"1d9783fe-808a-4862-81c1-31dcf22df1a4","Type":"ContainerStarted","Data":"a533715623674db382a8ac1871dc17464a08b689d5e875af35a22f9697561b6d"} Apr 16 16:50:29.189145 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.189121 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7479c89684-8kzt5"] Apr 16 16:50:29.193762 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.193742 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7479c89684-8kzt5" Apr 16 16:50:29.195946 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.195922 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Apr 16 16:50:29.196047 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.196030 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-sk482\"" Apr 16 16:50:29.196170 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.196156 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Apr 16 16:50:29.196405 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.196382 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Apr 16 16:50:29.203416 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.203392 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-59tvd"] Apr 16 16:50:29.206839 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.206819 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7479c89684-8kzt5"] Apr 16 16:50:29.206965 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.206952 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-59tvd" Apr 16 16:50:29.208613 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.208595 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-66nmg\"" Apr 16 16:50:29.208749 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.208701 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Apr 16 16:50:29.208749 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.208704 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Apr 16 16:50:29.209030 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.209013 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Apr 16 16:50:29.284068 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.284037 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/26f9c3a6-a68f-4294-ad49-f50574e36b2d-node-exporter-textfile\") pod \"node-exporter-59tvd\" (UID: \"26f9c3a6-a68f-4294-ad49-f50574e36b2d\") " pod="openshift-monitoring/node-exporter-59tvd" Apr 16 16:50:29.284068 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.284072 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/26f9c3a6-a68f-4294-ad49-f50574e36b2d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-59tvd\" (UID: \"26f9c3a6-a68f-4294-ad49-f50574e36b2d\") " pod="openshift-monitoring/node-exporter-59tvd" Apr 16 16:50:29.284363 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.284101 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/26f9c3a6-a68f-4294-ad49-f50574e36b2d-node-exporter-wtmp\") pod \"node-exporter-59tvd\" (UID: \"26f9c3a6-a68f-4294-ad49-f50574e36b2d\") " pod="openshift-monitoring/node-exporter-59tvd" Apr 16 16:50:29.284363 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.284157 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1a20b040-7ded-4aba-a389-84ea64de0827-volume-directive-shadow\") pod \"kube-state-metrics-7479c89684-8kzt5\" (UID: \"1a20b040-7ded-4aba-a389-84ea64de0827\") " pod="openshift-monitoring/kube-state-metrics-7479c89684-8kzt5" Apr 16 16:50:29.284363 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.284216 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1a20b040-7ded-4aba-a389-84ea64de0827-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7479c89684-8kzt5\" (UID: \"1a20b040-7ded-4aba-a389-84ea64de0827\") " pod="openshift-monitoring/kube-state-metrics-7479c89684-8kzt5" Apr 16 16:50:29.284363 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.284240 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26f9c3a6-a68f-4294-ad49-f50574e36b2d-metrics-client-ca\") pod \"node-exporter-59tvd\" (UID: \"26f9c3a6-a68f-4294-ad49-f50574e36b2d\") " pod="openshift-monitoring/node-exporter-59tvd" Apr 16 16:50:29.284363 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.284294 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pnrv\" (UniqueName: \"kubernetes.io/projected/26f9c3a6-a68f-4294-ad49-f50574e36b2d-kube-api-access-8pnrv\") pod \"node-exporter-59tvd\" (UID: \"26f9c3a6-a68f-4294-ad49-f50574e36b2d\") " pod="openshift-monitoring/node-exporter-59tvd" Apr 16 16:50:29.284363 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.284324 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26f9c3a6-a68f-4294-ad49-f50574e36b2d-sys\") pod \"node-exporter-59tvd\" (UID: \"26f9c3a6-a68f-4294-ad49-f50574e36b2d\") " pod="openshift-monitoring/node-exporter-59tvd" Apr 16 16:50:29.284363 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.284354 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a20b040-7ded-4aba-a389-84ea64de0827-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7479c89684-8kzt5\" (UID: \"1a20b040-7ded-4aba-a389-84ea64de0827\") " pod="openshift-monitoring/kube-state-metrics-7479c89684-8kzt5" Apr 16 16:50:29.284663 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.284382 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb85n\" (UniqueName: \"kubernetes.io/projected/1a20b040-7ded-4aba-a389-84ea64de0827-kube-api-access-tb85n\") pod \"kube-state-metrics-7479c89684-8kzt5\" (UID: \"1a20b040-7ded-4aba-a389-84ea64de0827\") " pod="openshift-monitoring/kube-state-metrics-7479c89684-8kzt5" Apr 16 16:50:29.284663 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.284409 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/26f9c3a6-a68f-4294-ad49-f50574e36b2d-node-exporter-tls\") pod \"node-exporter-59tvd\" (UID: \"26f9c3a6-a68f-4294-ad49-f50574e36b2d\") " pod="openshift-monitoring/node-exporter-59tvd" Apr 16 16:50:29.284663 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.284435 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/26f9c3a6-a68f-4294-ad49-f50574e36b2d-node-exporter-accelerators-collector-config\") pod \"node-exporter-59tvd\" (UID: \"26f9c3a6-a68f-4294-ad49-f50574e36b2d\") " pod="openshift-monitoring/node-exporter-59tvd" Apr 16 16:50:29.284663 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.284479 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a20b040-7ded-4aba-a389-84ea64de0827-metrics-client-ca\") pod \"kube-state-metrics-7479c89684-8kzt5\" (UID: \"1a20b040-7ded-4aba-a389-84ea64de0827\") " pod="openshift-monitoring/kube-state-metrics-7479c89684-8kzt5" Apr 16 16:50:29.284663 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.284522 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/26f9c3a6-a68f-4294-ad49-f50574e36b2d-root\") pod \"node-exporter-59tvd\" (UID: \"26f9c3a6-a68f-4294-ad49-f50574e36b2d\") " pod="openshift-monitoring/node-exporter-59tvd" Apr 16 16:50:29.284663 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.284552 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a20b040-7ded-4aba-a389-84ea64de0827-kube-state-metrics-tls\") pod \"kube-state-metrics-7479c89684-8kzt5\" (UID: \"1a20b040-7ded-4aba-a389-84ea64de0827\") " pod="openshift-monitoring/kube-state-metrics-7479c89684-8kzt5" Apr 16 16:50:29.384960 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.384924 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tb85n\" (UniqueName: \"kubernetes.io/projected/1a20b040-7ded-4aba-a389-84ea64de0827-kube-api-access-tb85n\") pod \"kube-state-metrics-7479c89684-8kzt5\" (UID: \"1a20b040-7ded-4aba-a389-84ea64de0827\") " pod="openshift-monitoring/kube-state-metrics-7479c89684-8kzt5" Apr 16 16:50:29.384960 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.384968 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/26f9c3a6-a68f-4294-ad49-f50574e36b2d-node-exporter-tls\") pod \"node-exporter-59tvd\" (UID: \"26f9c3a6-a68f-4294-ad49-f50574e36b2d\") " pod="openshift-monitoring/node-exporter-59tvd" Apr 16 16:50:29.385179 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.384994 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/26f9c3a6-a68f-4294-ad49-f50574e36b2d-node-exporter-accelerators-collector-config\") pod \"node-exporter-59tvd\" (UID: \"26f9c3a6-a68f-4294-ad49-f50574e36b2d\") " pod="openshift-monitoring/node-exporter-59tvd" Apr 16 16:50:29.385179 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.385037 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a20b040-7ded-4aba-a389-84ea64de0827-metrics-client-ca\") pod \"kube-state-metrics-7479c89684-8kzt5\" (UID: \"1a20b040-7ded-4aba-a389-84ea64de0827\") " pod="openshift-monitoring/kube-state-metrics-7479c89684-8kzt5" Apr 16 16:50:29.385179 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.385076 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/26f9c3a6-a68f-4294-ad49-f50574e36b2d-root\") pod \"node-exporter-59tvd\" (UID: \"26f9c3a6-a68f-4294-ad49-f50574e36b2d\") " pod="openshift-monitoring/node-exporter-59tvd" Apr 16 16:50:29.385179 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:50:29.385097 2576 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Apr 16 16:50:29.385179 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:50:29.385173 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26f9c3a6-a68f-4294-ad49-f50574e36b2d-node-exporter-tls podName:26f9c3a6-a68f-4294-ad49-f50574e36b2d nodeName:}" failed. No retries permitted until 2026-04-16 16:50:29.885151309 +0000 UTC m=+168.946300818 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/26f9c3a6-a68f-4294-ad49-f50574e36b2d-node-exporter-tls") pod "node-exporter-59tvd" (UID: "26f9c3a6-a68f-4294-ad49-f50574e36b2d") : secret "node-exporter-tls" not found Apr 16 16:50:29.385179 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.385173 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/26f9c3a6-a68f-4294-ad49-f50574e36b2d-root\") pod \"node-exporter-59tvd\" (UID: \"26f9c3a6-a68f-4294-ad49-f50574e36b2d\") " pod="openshift-monitoring/node-exporter-59tvd" Apr 16 16:50:29.385496 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.385103 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a20b040-7ded-4aba-a389-84ea64de0827-kube-state-metrics-tls\") pod \"kube-state-metrics-7479c89684-8kzt5\" (UID: \"1a20b040-7ded-4aba-a389-84ea64de0827\") " pod="openshift-monitoring/kube-state-metrics-7479c89684-8kzt5" Apr 16 16:50:29.385496 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.385307 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/26f9c3a6-a68f-4294-ad49-f50574e36b2d-node-exporter-textfile\") pod \"node-exporter-59tvd\" (UID: \"26f9c3a6-a68f-4294-ad49-f50574e36b2d\") " pod="openshift-monitoring/node-exporter-59tvd" Apr 16 16:50:29.385496 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.385350 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/26f9c3a6-a68f-4294-ad49-f50574e36b2d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-59tvd\" (UID: \"26f9c3a6-a68f-4294-ad49-f50574e36b2d\") " pod="openshift-monitoring/node-exporter-59tvd" Apr 16 16:50:29.385496 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.385404 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/26f9c3a6-a68f-4294-ad49-f50574e36b2d-node-exporter-wtmp\") pod \"node-exporter-59tvd\" (UID: \"26f9c3a6-a68f-4294-ad49-f50574e36b2d\") " pod="openshift-monitoring/node-exporter-59tvd" Apr 16 16:50:29.385496 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.385436 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1a20b040-7ded-4aba-a389-84ea64de0827-volume-directive-shadow\") pod \"kube-state-metrics-7479c89684-8kzt5\" (UID: \"1a20b040-7ded-4aba-a389-84ea64de0827\") " pod="openshift-monitoring/kube-state-metrics-7479c89684-8kzt5" Apr 16 16:50:29.385496 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.385477 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1a20b040-7ded-4aba-a389-84ea64de0827-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7479c89684-8kzt5\" (UID: \"1a20b040-7ded-4aba-a389-84ea64de0827\") " pod="openshift-monitoring/kube-state-metrics-7479c89684-8kzt5" Apr 16 16:50:29.385761 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.385503 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26f9c3a6-a68f-4294-ad49-f50574e36b2d-metrics-client-ca\") pod \"node-exporter-59tvd\" (UID: \"26f9c3a6-a68f-4294-ad49-f50574e36b2d\") " pod="openshift-monitoring/node-exporter-59tvd" Apr 16 16:50:29.385761 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.385531 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8pnrv\" (UniqueName: \"kubernetes.io/projected/26f9c3a6-a68f-4294-ad49-f50574e36b2d-kube-api-access-8pnrv\") pod \"node-exporter-59tvd\" (UID: \"26f9c3a6-a68f-4294-ad49-f50574e36b2d\") " pod="openshift-monitoring/node-exporter-59tvd" Apr 16 16:50:29.385761 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.385562 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26f9c3a6-a68f-4294-ad49-f50574e36b2d-sys\") pod \"node-exporter-59tvd\" (UID: \"26f9c3a6-a68f-4294-ad49-f50574e36b2d\") " pod="openshift-monitoring/node-exporter-59tvd" Apr 16 16:50:29.385761 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.385619 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a20b040-7ded-4aba-a389-84ea64de0827-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7479c89684-8kzt5\" (UID: \"1a20b040-7ded-4aba-a389-84ea64de0827\") " pod="openshift-monitoring/kube-state-metrics-7479c89684-8kzt5" Apr 16 16:50:29.385761 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.385667 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/26f9c3a6-a68f-4294-ad49-f50574e36b2d-node-exporter-textfile\") pod \"node-exporter-59tvd\" (UID: \"26f9c3a6-a68f-4294-ad49-f50574e36b2d\") " pod="openshift-monitoring/node-exporter-59tvd" Apr 16 16:50:29.385761 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.385749 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/26f9c3a6-a68f-4294-ad49-f50574e36b2d-node-exporter-accelerators-collector-config\") pod \"node-exporter-59tvd\" (UID: \"26f9c3a6-a68f-4294-ad49-f50574e36b2d\") " pod="openshift-monitoring/node-exporter-59tvd" Apr 16 16:50:29.386032 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.385819 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1a20b040-7ded-4aba-a389-84ea64de0827-volume-directive-shadow\") pod \"kube-state-metrics-7479c89684-8kzt5\" (UID: \"1a20b040-7ded-4aba-a389-84ea64de0827\") " pod="openshift-monitoring/kube-state-metrics-7479c89684-8kzt5" Apr 16 16:50:29.386032 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.385898 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a20b040-7ded-4aba-a389-84ea64de0827-metrics-client-ca\") pod \"kube-state-metrics-7479c89684-8kzt5\" (UID: \"1a20b040-7ded-4aba-a389-84ea64de0827\") " pod="openshift-monitoring/kube-state-metrics-7479c89684-8kzt5" Apr 16 16:50:29.386032 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.385962 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/26f9c3a6-a68f-4294-ad49-f50574e36b2d-node-exporter-wtmp\") pod \"node-exporter-59tvd\" (UID: \"26f9c3a6-a68f-4294-ad49-f50574e36b2d\") " pod="openshift-monitoring/node-exporter-59tvd" Apr 16 16:50:29.386032 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.385962 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26f9c3a6-a68f-4294-ad49-f50574e36b2d-sys\") pod \"node-exporter-59tvd\" (UID: \"26f9c3a6-a68f-4294-ad49-f50574e36b2d\") " pod="openshift-monitoring/node-exporter-59tvd" Apr 16 16:50:29.386208 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.386188 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26f9c3a6-a68f-4294-ad49-f50574e36b2d-metrics-client-ca\") pod \"node-exporter-59tvd\" (UID: \"26f9c3a6-a68f-4294-ad49-f50574e36b2d\") " pod="openshift-monitoring/node-exporter-59tvd" Apr 16 16:50:29.386616 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.386595 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1a20b040-7ded-4aba-a389-84ea64de0827-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7479c89684-8kzt5\" (UID: \"1a20b040-7ded-4aba-a389-84ea64de0827\") " pod="openshift-monitoring/kube-state-metrics-7479c89684-8kzt5" Apr 16 16:50:29.388437 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.388418 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/26f9c3a6-a68f-4294-ad49-f50574e36b2d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-59tvd\" (UID: \"26f9c3a6-a68f-4294-ad49-f50574e36b2d\") " pod="openshift-monitoring/node-exporter-59tvd" Apr 16 16:50:29.388627 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.388610 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a20b040-7ded-4aba-a389-84ea64de0827-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7479c89684-8kzt5\" (UID: \"1a20b040-7ded-4aba-a389-84ea64de0827\") " pod="openshift-monitoring/kube-state-metrics-7479c89684-8kzt5" Apr 16 16:50:29.389065 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.389049 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a20b040-7ded-4aba-a389-84ea64de0827-kube-state-metrics-tls\") pod \"kube-state-metrics-7479c89684-8kzt5\" (UID: \"1a20b040-7ded-4aba-a389-84ea64de0827\") " pod="openshift-monitoring/kube-state-metrics-7479c89684-8kzt5" Apr 16 16:50:29.397385 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.397357 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb85n\" (UniqueName: \"kubernetes.io/projected/1a20b040-7ded-4aba-a389-84ea64de0827-kube-api-access-tb85n\") pod \"kube-state-metrics-7479c89684-8kzt5\" (UID: \"1a20b040-7ded-4aba-a389-84ea64de0827\") " pod="openshift-monitoring/kube-state-metrics-7479c89684-8kzt5" Apr 16 16:50:29.397531 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.397415 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pnrv\" (UniqueName: \"kubernetes.io/projected/26f9c3a6-a68f-4294-ad49-f50574e36b2d-kube-api-access-8pnrv\") pod \"node-exporter-59tvd\" (UID: \"26f9c3a6-a68f-4294-ad49-f50574e36b2d\") " pod="openshift-monitoring/node-exporter-59tvd" Apr 16 16:50:29.503540 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.503460 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7479c89684-8kzt5" Apr 16 16:50:29.641695 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.641665 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7479c89684-8kzt5"] Apr 16 16:50:29.644343 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:50:29.644314 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a20b040_7ded_4aba_a389_84ea64de0827.slice/crio-d4e0437c79cc66f2fcbcf3ab90365a62a0723a2771d60f7e9ada0831fe309947 WatchSource:0}: Error finding container d4e0437c79cc66f2fcbcf3ab90365a62a0723a2771d60f7e9ada0831fe309947: Status 404 returned error can't find the container with id d4e0437c79cc66f2fcbcf3ab90365a62a0723a2771d60f7e9ada0831fe309947 Apr 16 16:50:29.890758 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.890728 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/26f9c3a6-a68f-4294-ad49-f50574e36b2d-node-exporter-tls\") pod \"node-exporter-59tvd\" (UID: \"26f9c3a6-a68f-4294-ad49-f50574e36b2d\") " pod="openshift-monitoring/node-exporter-59tvd" Apr 16 16:50:29.893460 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:29.893432 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/26f9c3a6-a68f-4294-ad49-f50574e36b2d-node-exporter-tls\") pod \"node-exporter-59tvd\" (UID: \"26f9c3a6-a68f-4294-ad49-f50574e36b2d\") " pod="openshift-monitoring/node-exporter-59tvd" Apr 16 16:50:30.115044 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.115004 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-59tvd" Apr 16 16:50:30.128543 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:50:30.128508 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26f9c3a6_a68f_4294_ad49_f50574e36b2d.slice/crio-7040e7a62b686334d34335fc10e3b6e1caf6c625043f37660b24b354d581c87c WatchSource:0}: Error finding container 7040e7a62b686334d34335fc10e3b6e1caf6c625043f37660b24b354d581c87c: Status 404 returned error can't find the container with id 7040e7a62b686334d34335fc10e3b6e1caf6c625043f37660b24b354d581c87c Apr 16 16:50:30.195935 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.195785 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-59tvd" event={"ID":"26f9c3a6-a68f-4294-ad49-f50574e36b2d","Type":"ContainerStarted","Data":"7040e7a62b686334d34335fc10e3b6e1caf6c625043f37660b24b354d581c87c"} Apr 16 16:50:30.197861 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.197829 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7479c89684-8kzt5" event={"ID":"1a20b040-7ded-4aba-a389-84ea64de0827","Type":"ContainerStarted","Data":"d4e0437c79cc66f2fcbcf3ab90365a62a0723a2771d60f7e9ada0831fe309947"} Apr 16 16:50:30.249733 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.249528 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Apr 16 16:50:30.255097 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.255062 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:50:30.257388 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.257339 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Apr 16 16:50:30.257525 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.257433 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Apr 16 16:50:30.257525 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.257491 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Apr 16 16:50:30.258735 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.257865 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Apr 16 16:50:30.258735 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.257936 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Apr 16 16:50:30.258735 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.257955 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Apr 16 16:50:30.258735 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.257866 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Apr 16 16:50:30.258735 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.258124 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Apr 16 16:50:30.258735 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.258575 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Apr 16 16:50:30.258735 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.258641 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-nr6qj\"" Apr 16 16:50:30.280709 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.280012 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Apr 16 16:50:30.294703 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.294665 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1c13e2d-7bc4-4764-b62a-01c231a9b003-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e1c13e2d-7bc4-4764-b62a-01c231a9b003\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:50:30.294858 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.294723 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e1c13e2d-7bc4-4764-b62a-01c231a9b003-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e1c13e2d-7bc4-4764-b62a-01c231a9b003\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:50:30.294858 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.294753 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e1c13e2d-7bc4-4764-b62a-01c231a9b003-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e1c13e2d-7bc4-4764-b62a-01c231a9b003\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:50:30.294858 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.294781 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e1c13e2d-7bc4-4764-b62a-01c231a9b003-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e1c13e2d-7bc4-4764-b62a-01c231a9b003\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:50:30.295017 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.294885 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e1c13e2d-7bc4-4764-b62a-01c231a9b003-config-volume\") pod \"alertmanager-main-0\" (UID: \"e1c13e2d-7bc4-4764-b62a-01c231a9b003\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:50:30.295017 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.294925 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e1c13e2d-7bc4-4764-b62a-01c231a9b003-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e1c13e2d-7bc4-4764-b62a-01c231a9b003\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:50:30.295017 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.294961 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e1c13e2d-7bc4-4764-b62a-01c231a9b003-config-out\") pod \"alertmanager-main-0\" (UID: \"e1c13e2d-7bc4-4764-b62a-01c231a9b003\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:50:30.295017 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.294989 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e1c13e2d-7bc4-4764-b62a-01c231a9b003-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e1c13e2d-7bc4-4764-b62a-01c231a9b003\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:50:30.295211 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.295018 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e1c13e2d-7bc4-4764-b62a-01c231a9b003-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e1c13e2d-7bc4-4764-b62a-01c231a9b003\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:50:30.295211 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.295048 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e1c13e2d-7bc4-4764-b62a-01c231a9b003-web-config\") pod \"alertmanager-main-0\" (UID: \"e1c13e2d-7bc4-4764-b62a-01c231a9b003\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:50:30.295211 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.295077 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1c13e2d-7bc4-4764-b62a-01c231a9b003-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e1c13e2d-7bc4-4764-b62a-01c231a9b003\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:50:30.295211 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.295116 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e1c13e2d-7bc4-4764-b62a-01c231a9b003-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e1c13e2d-7bc4-4764-b62a-01c231a9b003\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:50:30.295211 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.295162 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxstg\" (UniqueName: \"kubernetes.io/projected/e1c13e2d-7bc4-4764-b62a-01c231a9b003-kube-api-access-bxstg\") pod \"alertmanager-main-0\" (UID: \"e1c13e2d-7bc4-4764-b62a-01c231a9b003\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:50:30.396151 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.396066 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bxstg\" (UniqueName: \"kubernetes.io/projected/e1c13e2d-7bc4-4764-b62a-01c231a9b003-kube-api-access-bxstg\") pod \"alertmanager-main-0\" (UID: \"e1c13e2d-7bc4-4764-b62a-01c231a9b003\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:50:30.396151 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.396127 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1c13e2d-7bc4-4764-b62a-01c231a9b003-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e1c13e2d-7bc4-4764-b62a-01c231a9b003\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:50:30.396419 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.396164 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e1c13e2d-7bc4-4764-b62a-01c231a9b003-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e1c13e2d-7bc4-4764-b62a-01c231a9b003\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:50:30.396419 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.396199 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e1c13e2d-7bc4-4764-b62a-01c231a9b003-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e1c13e2d-7bc4-4764-b62a-01c231a9b003\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:50:30.396419 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.396231 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e1c13e2d-7bc4-4764-b62a-01c231a9b003-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e1c13e2d-7bc4-4764-b62a-01c231a9b003\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:50:30.396419 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.396310 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e1c13e2d-7bc4-4764-b62a-01c231a9b003-config-volume\") pod \"alertmanager-main-0\" (UID: \"e1c13e2d-7bc4-4764-b62a-01c231a9b003\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:50:30.396419 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.396340 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e1c13e2d-7bc4-4764-b62a-01c231a9b003-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e1c13e2d-7bc4-4764-b62a-01c231a9b003\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:50:30.396661 ip-10-0-142-37 kubenswrapper[2576]: E0416 16:50:30.396456 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e1c13e2d-7bc4-4764-b62a-01c231a9b003-alertmanager-trusted-ca-bundle podName:e1c13e2d-7bc4-4764-b62a-01c231a9b003 nodeName:}" failed. No retries permitted until 2026-04-16 16:50:30.896335299 +0000 UTC m=+169.957484821 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/e1c13e2d-7bc4-4764-b62a-01c231a9b003-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "e1c13e2d-7bc4-4764-b62a-01c231a9b003") : configmap references non-existent config key: ca-bundle.crt Apr 16 16:50:30.396661 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.396613 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e1c13e2d-7bc4-4764-b62a-01c231a9b003-config-out\") pod \"alertmanager-main-0\" (UID: \"e1c13e2d-7bc4-4764-b62a-01c231a9b003\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:50:30.396771 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.396652 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e1c13e2d-7bc4-4764-b62a-01c231a9b003-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e1c13e2d-7bc4-4764-b62a-01c231a9b003\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:50:30.396820 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.396784 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e1c13e2d-7bc4-4764-b62a-01c231a9b003-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e1c13e2d-7bc4-4764-b62a-01c231a9b003\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:50:30.397412 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.396933 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e1c13e2d-7bc4-4764-b62a-01c231a9b003-web-config\") pod \"alertmanager-main-0\" (UID: \"e1c13e2d-7bc4-4764-b62a-01c231a9b003\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:50:30.397412 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.396970 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1c13e2d-7bc4-4764-b62a-01c231a9b003-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e1c13e2d-7bc4-4764-b62a-01c231a9b003\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:50:30.397412 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.397009 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e1c13e2d-7bc4-4764-b62a-01c231a9b003-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e1c13e2d-7bc4-4764-b62a-01c231a9b003\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:50:30.397946 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.397902 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e1c13e2d-7bc4-4764-b62a-01c231a9b003-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e1c13e2d-7bc4-4764-b62a-01c231a9b003\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:50:30.398600 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.398553 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1c13e2d-7bc4-4764-b62a-01c231a9b003-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e1c13e2d-7bc4-4764-b62a-01c231a9b003\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:50:30.401531 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.401466 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e1c13e2d-7bc4-4764-b62a-01c231a9b003-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e1c13e2d-7bc4-4764-b62a-01c231a9b003\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:50:30.406911 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.403457 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e1c13e2d-7bc4-4764-b62a-01c231a9b003-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e1c13e2d-7bc4-4764-b62a-01c231a9b003\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:50:30.406911 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.406741 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e1c13e2d-7bc4-4764-b62a-01c231a9b003-web-config\") pod \"alertmanager-main-0\" (UID: \"e1c13e2d-7bc4-4764-b62a-01c231a9b003\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:50:30.406911 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.406785 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e1c13e2d-7bc4-4764-b62a-01c231a9b003-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e1c13e2d-7bc4-4764-b62a-01c231a9b003\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:50:30.407108 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.406934 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e1c13e2d-7bc4-4764-b62a-01c231a9b003-config-out\") pod \"alertmanager-main-0\" (UID: \"e1c13e2d-7bc4-4764-b62a-01c231a9b003\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:50:30.407435 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.407400 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e1c13e2d-7bc4-4764-b62a-01c231a9b003-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e1c13e2d-7bc4-4764-b62a-01c231a9b003\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:50:30.408009 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.407958 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e1c13e2d-7bc4-4764-b62a-01c231a9b003-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e1c13e2d-7bc4-4764-b62a-01c231a9b003\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:50:30.408114 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.408053 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e1c13e2d-7bc4-4764-b62a-01c231a9b003-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e1c13e2d-7bc4-4764-b62a-01c231a9b003\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:50:30.409496 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.409472 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e1c13e2d-7bc4-4764-b62a-01c231a9b003-config-volume\") pod \"alertmanager-main-0\" (UID: \"e1c13e2d-7bc4-4764-b62a-01c231a9b003\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:50:30.409692 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.409670 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxstg\" (UniqueName: \"kubernetes.io/projected/e1c13e2d-7bc4-4764-b62a-01c231a9b003-kube-api-access-bxstg\") pod \"alertmanager-main-0\" (UID: \"e1c13e2d-7bc4-4764-b62a-01c231a9b003\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:50:30.588844 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.588810 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xfnx6" Apr 16 16:50:30.591553 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.591523 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-c25vw\"" Apr 16 16:50:30.600565 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.599397 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xfnx6" Apr 16 16:50:30.903263 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.903155 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1c13e2d-7bc4-4764-b62a-01c231a9b003-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e1c13e2d-7bc4-4764-b62a-01c231a9b003\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:50:30.904108 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:30.904078 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1c13e2d-7bc4-4764-b62a-01c231a9b003-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e1c13e2d-7bc4-4764-b62a-01c231a9b003\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:50:31.182974 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:31.182895 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:50:32.248872 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:32.248834 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-78df5754bd-ngrn7"] Apr 16 16:50:32.252978 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:32.252955 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-78df5754bd-ngrn7" Apr 16 16:50:32.254908 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:32.254871 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Apr 16 16:50:32.255014 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:32.254997 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Apr 16 16:50:32.255082 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:32.255048 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-v7z7d\"" Apr 16 16:50:32.255152 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:32.255135 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-2qviq3j6seq26\"" Apr 16 16:50:32.255208 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:32.255166 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Apr 16 16:50:32.255431 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:32.255407 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Apr 16 16:50:32.255634 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:32.255615 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Apr 16 16:50:32.264555 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:32.264531 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-78df5754bd-ngrn7"] Apr 16 16:50:32.320111 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:32.317453 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/05869c45-9aab-4f0a-93bd-06c4bba46382-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-78df5754bd-ngrn7\" (UID: \"05869c45-9aab-4f0a-93bd-06c4bba46382\") " pod="openshift-monitoring/thanos-querier-78df5754bd-ngrn7" Apr 16 16:50:32.320111 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:32.317517 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tffx\" (UniqueName: \"kubernetes.io/projected/05869c45-9aab-4f0a-93bd-06c4bba46382-kube-api-access-2tffx\") pod \"thanos-querier-78df5754bd-ngrn7\" (UID: \"05869c45-9aab-4f0a-93bd-06c4bba46382\") " pod="openshift-monitoring/thanos-querier-78df5754bd-ngrn7" Apr 16 16:50:32.320111 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:32.317559 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/05869c45-9aab-4f0a-93bd-06c4bba46382-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-78df5754bd-ngrn7\" (UID: \"05869c45-9aab-4f0a-93bd-06c4bba46382\") " pod="openshift-monitoring/thanos-querier-78df5754bd-ngrn7" Apr 16 16:50:32.320111 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:32.317586 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05869c45-9aab-4f0a-93bd-06c4bba46382-metrics-client-ca\") pod \"thanos-querier-78df5754bd-ngrn7\" (UID: \"05869c45-9aab-4f0a-93bd-06c4bba46382\") " pod="openshift-monitoring/thanos-querier-78df5754bd-ngrn7" Apr 16 16:50:32.320111 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:32.317624 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/05869c45-9aab-4f0a-93bd-06c4bba46382-secret-grpc-tls\") pod \"thanos-querier-78df5754bd-ngrn7\" (UID: \"05869c45-9aab-4f0a-93bd-06c4bba46382\") " pod="openshift-monitoring/thanos-querier-78df5754bd-ngrn7" Apr 16 16:50:32.320111 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:32.317710 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/05869c45-9aab-4f0a-93bd-06c4bba46382-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-78df5754bd-ngrn7\" (UID: \"05869c45-9aab-4f0a-93bd-06c4bba46382\") " pod="openshift-monitoring/thanos-querier-78df5754bd-ngrn7" Apr 16 16:50:32.320111 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:32.317749 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/05869c45-9aab-4f0a-93bd-06c4bba46382-secret-thanos-querier-tls\") pod \"thanos-querier-78df5754bd-ngrn7\" (UID: \"05869c45-9aab-4f0a-93bd-06c4bba46382\") " pod="openshift-monitoring/thanos-querier-78df5754bd-ngrn7" Apr 16 16:50:32.320111 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:32.317775 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/05869c45-9aab-4f0a-93bd-06c4bba46382-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-78df5754bd-ngrn7\" (UID: \"05869c45-9aab-4f0a-93bd-06c4bba46382\") " pod="openshift-monitoring/thanos-querier-78df5754bd-ngrn7" Apr 16 16:50:32.419642 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:32.419602 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/05869c45-9aab-4f0a-93bd-06c4bba46382-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-78df5754bd-ngrn7\" (UID: \"05869c45-9aab-4f0a-93bd-06c4bba46382\") " pod="openshift-monitoring/thanos-querier-78df5754bd-ngrn7" Apr 16 16:50:32.419817 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:32.419654 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05869c45-9aab-4f0a-93bd-06c4bba46382-metrics-client-ca\") pod \"thanos-querier-78df5754bd-ngrn7\" (UID: \"05869c45-9aab-4f0a-93bd-06c4bba46382\") " pod="openshift-monitoring/thanos-querier-78df5754bd-ngrn7" Apr 16 16:50:32.419817 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:32.419700 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/05869c45-9aab-4f0a-93bd-06c4bba46382-secret-grpc-tls\") pod \"thanos-querier-78df5754bd-ngrn7\" (UID: \"05869c45-9aab-4f0a-93bd-06c4bba46382\") " pod="openshift-monitoring/thanos-querier-78df5754bd-ngrn7" Apr 16 16:50:32.419817 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:32.419781 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/05869c45-9aab-4f0a-93bd-06c4bba46382-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-78df5754bd-ngrn7\" (UID: \"05869c45-9aab-4f0a-93bd-06c4bba46382\") " pod="openshift-monitoring/thanos-querier-78df5754bd-ngrn7" Apr 16 16:50:32.420354 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:32.420308 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/05869c45-9aab-4f0a-93bd-06c4bba46382-secret-thanos-querier-tls\") pod \"thanos-querier-78df5754bd-ngrn7\" (UID: \"05869c45-9aab-4f0a-93bd-06c4bba46382\") " pod="openshift-monitoring/thanos-querier-78df5754bd-ngrn7" Apr 16 16:50:32.420354 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:32.420349 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/05869c45-9aab-4f0a-93bd-06c4bba46382-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-78df5754bd-ngrn7\" (UID: \"05869c45-9aab-4f0a-93bd-06c4bba46382\") " pod="openshift-monitoring/thanos-querier-78df5754bd-ngrn7" Apr 16 16:50:32.420511 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:32.420409 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/05869c45-9aab-4f0a-93bd-06c4bba46382-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-78df5754bd-ngrn7\" (UID: \"05869c45-9aab-4f0a-93bd-06c4bba46382\") " pod="openshift-monitoring/thanos-querier-78df5754bd-ngrn7" Apr 16 16:50:32.420511 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:32.420452 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2tffx\" (UniqueName: \"kubernetes.io/projected/05869c45-9aab-4f0a-93bd-06c4bba46382-kube-api-access-2tffx\") pod \"thanos-querier-78df5754bd-ngrn7\" (UID: \"05869c45-9aab-4f0a-93bd-06c4bba46382\") " pod="openshift-monitoring/thanos-querier-78df5754bd-ngrn7" Apr 16 16:50:32.420622 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:32.420571 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Apr 16 16:50:32.424345 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:32.424264 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05869c45-9aab-4f0a-93bd-06c4bba46382-metrics-client-ca\") pod \"thanos-querier-78df5754bd-ngrn7\" (UID: \"05869c45-9aab-4f0a-93bd-06c4bba46382\") " pod="openshift-monitoring/thanos-querier-78df5754bd-ngrn7" Apr 16 16:50:32.426346 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:32.424869 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/05869c45-9aab-4f0a-93bd-06c4bba46382-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-78df5754bd-ngrn7\" (UID: \"05869c45-9aab-4f0a-93bd-06c4bba46382\") " pod="openshift-monitoring/thanos-querier-78df5754bd-ngrn7" Apr 16 16:50:32.426346 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:32.426075 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/05869c45-9aab-4f0a-93bd-06c4bba46382-secret-thanos-querier-tls\") pod \"thanos-querier-78df5754bd-ngrn7\" (UID: \"05869c45-9aab-4f0a-93bd-06c4bba46382\") " pod="openshift-monitoring/thanos-querier-78df5754bd-ngrn7" Apr 16 16:50:32.426996 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:32.426543 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/05869c45-9aab-4f0a-93bd-06c4bba46382-secret-grpc-tls\") pod \"thanos-querier-78df5754bd-ngrn7\" (UID: \"05869c45-9aab-4f0a-93bd-06c4bba46382\") " pod="openshift-monitoring/thanos-querier-78df5754bd-ngrn7" Apr 16 16:50:32.426996 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:32.426722 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/05869c45-9aab-4f0a-93bd-06c4bba46382-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-78df5754bd-ngrn7\" (UID: \"05869c45-9aab-4f0a-93bd-06c4bba46382\") " pod="openshift-monitoring/thanos-querier-78df5754bd-ngrn7" Apr 16 16:50:32.426996 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:32.426954 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/05869c45-9aab-4f0a-93bd-06c4bba46382-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-78df5754bd-ngrn7\" (UID: \"05869c45-9aab-4f0a-93bd-06c4bba46382\") " pod="openshift-monitoring/thanos-querier-78df5754bd-ngrn7" Apr 16 16:50:32.427770 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:32.427499 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/05869c45-9aab-4f0a-93bd-06c4bba46382-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-78df5754bd-ngrn7\" (UID: \"05869c45-9aab-4f0a-93bd-06c4bba46382\") " pod="openshift-monitoring/thanos-querier-78df5754bd-ngrn7" Apr 16 16:50:32.429790 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:32.429188 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tffx\" (UniqueName: \"kubernetes.io/projected/05869c45-9aab-4f0a-93bd-06c4bba46382-kube-api-access-2tffx\") pod \"thanos-querier-78df5754bd-ngrn7\" (UID: \"05869c45-9aab-4f0a-93bd-06c4bba46382\") " pod="openshift-monitoring/thanos-querier-78df5754bd-ngrn7" Apr 16 16:50:32.431484 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:32.431439 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xfnx6"] Apr 16 16:50:32.431738 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:50:32.431695 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1c13e2d_7bc4_4764_b62a_01c231a9b003.slice/crio-ce7f5d0e3fb0bfb4cb67e10f0c4a10eb16359c5065e5b4cc74f169b030d09905 WatchSource:0}: Error finding container ce7f5d0e3fb0bfb4cb67e10f0c4a10eb16359c5065e5b4cc74f169b030d09905: Status 404 returned error can't find the container with id ce7f5d0e3fb0bfb4cb67e10f0c4a10eb16359c5065e5b4cc74f169b030d09905 Apr 16 16:50:32.436585 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:50:32.436528 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefee44e6_2e2b_47f0_b356_6e77267f19e0.slice/crio-08ab8332dcb300025e2c4899a6cd31c12f6d337846bbe859b8cb2d8066110914 WatchSource:0}: Error finding container 08ab8332dcb300025e2c4899a6cd31c12f6d337846bbe859b8cb2d8066110914: Status 404 returned error can't find the container with id 08ab8332dcb300025e2c4899a6cd31c12f6d337846bbe859b8cb2d8066110914 Apr 16 16:50:32.588061 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:32.588038 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk5gr" Apr 16 16:50:32.595672 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:32.595649 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-78df5754bd-ngrn7" Apr 16 16:50:32.725379 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:32.725347 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-78df5754bd-ngrn7"] Apr 16 16:50:32.728504 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:50:32.728471 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05869c45_9aab_4f0a_93bd_06c4bba46382.slice/crio-4ca1713c3bab9e9e20808d54d88e20ceb4a923766287e6f3ddd15ce5e9dca969 WatchSource:0}: Error finding container 4ca1713c3bab9e9e20808d54d88e20ceb4a923766287e6f3ddd15ce5e9dca969: Status 404 returned error can't find the container with id 4ca1713c3bab9e9e20808d54d88e20ceb4a923766287e6f3ddd15ce5e9dca969 Apr 16 16:50:33.207693 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:33.207659 2576 generic.go:358] "Generic (PLEG): container finished" podID="26f9c3a6-a68f-4294-ad49-f50574e36b2d" containerID="d884d23c0f837a67bfa7664c496de39fc65e1412461ecb02593b51a5b9192d5f" exitCode=0 Apr 16 16:50:33.207850 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:33.207745 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-59tvd" event={"ID":"26f9c3a6-a68f-4294-ad49-f50574e36b2d","Type":"ContainerDied","Data":"d884d23c0f837a67bfa7664c496de39fc65e1412461ecb02593b51a5b9192d5f"} Apr 16 16:50:33.210068 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:33.209995 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7479c89684-8kzt5" event={"ID":"1a20b040-7ded-4aba-a389-84ea64de0827","Type":"ContainerStarted","Data":"39b8a0612cd2365acd67eb454a9a3f97728d401f18ab52eaeaf92abf3dd7c53a"} Apr 16 16:50:33.210068 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:33.210031 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7479c89684-8kzt5" event={"ID":"1a20b040-7ded-4aba-a389-84ea64de0827","Type":"ContainerStarted","Data":"04e0537533f158e59b5f9549c6ab6ce6ce43795dd6a916777d834b8ffe402d4a"} Apr 16 16:50:33.210068 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:33.210045 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7479c89684-8kzt5" event={"ID":"1a20b040-7ded-4aba-a389-84ea64de0827","Type":"ContainerStarted","Data":"0693096ed47507e65b4551292656cda53913b88fa0e6ffd3cf01354abb5b11fe"} Apr 16 16:50:33.211904 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:33.211865 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78df5754bd-ngrn7" event={"ID":"05869c45-9aab-4f0a-93bd-06c4bba46382","Type":"ContainerStarted","Data":"4ca1713c3bab9e9e20808d54d88e20ceb4a923766287e6f3ddd15ce5e9dca969"} Apr 16 16:50:33.213226 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:33.213194 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xfnx6" event={"ID":"efee44e6-2e2b-47f0-b356-6e77267f19e0","Type":"ContainerStarted","Data":"08ab8332dcb300025e2c4899a6cd31c12f6d337846bbe859b8cb2d8066110914"} Apr 16 16:50:33.215417 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:33.215344 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fb554db88-dcz4j" event={"ID":"1d9783fe-808a-4862-81c1-31dcf22df1a4","Type":"ContainerStarted","Data":"99811446d4e46a25015a2df57ff6a7babc8221367ed53d6b7496b1c70c53aec3"} Apr 16 16:50:33.216601 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:33.216571 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e1c13e2d-7bc4-4764-b62a-01c231a9b003","Type":"ContainerStarted","Data":"ce7f5d0e3fb0bfb4cb67e10f0c4a10eb16359c5065e5b4cc74f169b030d09905"} Apr 16 16:50:33.244914 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:33.244860 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7479c89684-8kzt5" podStartSLOduration=1.626064515 podStartE2EDuration="4.244843007s" podCreationTimestamp="2026-04-16 16:50:29 +0000 UTC" firstStartedPulling="2026-04-16 16:50:29.646381997 +0000 UTC m=+168.707531506" lastFinishedPulling="2026-04-16 16:50:32.265160481 +0000 UTC m=+171.326309998" observedRunningTime="2026-04-16 16:50:33.243379244 +0000 UTC m=+172.304528777" watchObservedRunningTime="2026-04-16 16:50:33.244843007 +0000 UTC m=+172.305992538" Apr 16 16:50:33.262814 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:33.261974 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-6fb554db88-dcz4j" podStartSLOduration=2.0254371349999998 podStartE2EDuration="5.261954317s" podCreationTimestamp="2026-04-16 16:50:28 +0000 UTC" firstStartedPulling="2026-04-16 16:50:29.027356035 +0000 UTC m=+168.088505545" lastFinishedPulling="2026-04-16 16:50:32.263873214 +0000 UTC m=+171.325022727" observedRunningTime="2026-04-16 16:50:33.260826699 +0000 UTC m=+172.321976231" watchObservedRunningTime="2026-04-16 16:50:33.261954317 +0000 UTC m=+172.323103851" Apr 16 16:50:33.976035 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:33.975227 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5876b4bbc7-gmxs2"] Apr 16 16:50:33.979317 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:33.979291 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5876b4bbc7-gmxs2" Apr 16 16:50:33.981780 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:33.981595 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Apr 16 16:50:33.981780 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:33.981604 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"default-dockercfg-br4mq\"" Apr 16 16:50:33.990446 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:33.990409 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5876b4bbc7-gmxs2"] Apr 16 16:50:34.038849 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:34.038810 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d620b8bc-d805-441c-864d-5f23f7621826-monitoring-plugin-cert\") pod \"monitoring-plugin-5876b4bbc7-gmxs2\" (UID: \"d620b8bc-d805-441c-864d-5f23f7621826\") " pod="openshift-monitoring/monitoring-plugin-5876b4bbc7-gmxs2" Apr 16 16:50:34.140355 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:34.140320 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d620b8bc-d805-441c-864d-5f23f7621826-monitoring-plugin-cert\") pod \"monitoring-plugin-5876b4bbc7-gmxs2\" (UID: \"d620b8bc-d805-441c-864d-5f23f7621826\") " pod="openshift-monitoring/monitoring-plugin-5876b4bbc7-gmxs2" Apr 16 16:50:34.143392 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:34.143362 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d620b8bc-d805-441c-864d-5f23f7621826-monitoring-plugin-cert\") pod \"monitoring-plugin-5876b4bbc7-gmxs2\" (UID: \"d620b8bc-d805-441c-864d-5f23f7621826\") " pod="openshift-monitoring/monitoring-plugin-5876b4bbc7-gmxs2" Apr 16 16:50:34.222000 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:34.221961 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-59tvd" event={"ID":"26f9c3a6-a68f-4294-ad49-f50574e36b2d","Type":"ContainerStarted","Data":"2b60ffb10f01b70ba63b7cace7797a8a3ba177065705e99af85c354bf52d4254"} Apr 16 16:50:34.292978 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:34.292941 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5876b4bbc7-gmxs2" Apr 16 16:50:34.792944 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:34.792602 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5876b4bbc7-gmxs2"] Apr 16 16:50:34.798788 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:50:34.798759 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd620b8bc_d805_441c_864d_5f23f7621826.slice/crio-0ecd70ac36b8f0ed5d45def8a2c491ec34e958b3f10af50d03422292fc61d286 WatchSource:0}: Error finding container 0ecd70ac36b8f0ed5d45def8a2c491ec34e958b3f10af50d03422292fc61d286: Status 404 returned error can't find the container with id 0ecd70ac36b8f0ed5d45def8a2c491ec34e958b3f10af50d03422292fc61d286 Apr 16 16:50:35.226782 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.226701 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5876b4bbc7-gmxs2" event={"ID":"d620b8bc-d805-441c-864d-5f23f7621826","Type":"ContainerStarted","Data":"0ecd70ac36b8f0ed5d45def8a2c491ec34e958b3f10af50d03422292fc61d286"} Apr 16 16:50:35.228622 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.228597 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78df5754bd-ngrn7" event={"ID":"05869c45-9aab-4f0a-93bd-06c4bba46382","Type":"ContainerStarted","Data":"1ae11d1a632c255439fbe969e8f0b23ea28147bde120ac37cc88fb3b1f9dc76b"} Apr 16 16:50:35.228756 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.228629 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78df5754bd-ngrn7" event={"ID":"05869c45-9aab-4f0a-93bd-06c4bba46382","Type":"ContainerStarted","Data":"249f4bd8e38690e43cb7e0998e2b1e6aa858362d48770ecea98874287e2e5d1c"} Apr 16 16:50:35.228756 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.228642 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78df5754bd-ngrn7" event={"ID":"05869c45-9aab-4f0a-93bd-06c4bba46382","Type":"ContainerStarted","Data":"9d0c5f458b33adfbbfd1ebe8c54d3a7d2ea2b30f832f2f6301aa56bd8eacfe65"} Apr 16 16:50:35.229822 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.229794 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xfnx6" event={"ID":"efee44e6-2e2b-47f0-b356-6e77267f19e0","Type":"ContainerStarted","Data":"a5c804348f839813d5054294f126ef135bbf16f9478f15e2f8a2efba480882ee"} Apr 16 16:50:35.232010 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.231980 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-59tvd" event={"ID":"26f9c3a6-a68f-4294-ad49-f50574e36b2d","Type":"ContainerStarted","Data":"21c026505c28f0eca27c723c72bfef11b456261287a806401def6bf6bf067ef8"} Apr 16 16:50:35.234688 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.234662 2576 generic.go:358] "Generic (PLEG): container finished" podID="e1c13e2d-7bc4-4764-b62a-01c231a9b003" containerID="a49cead03208f94dceebdf53a25c29bef47f09c9f54141502e126df70216fe43" exitCode=0 Apr 16 16:50:35.234784 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.234708 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e1c13e2d-7bc4-4764-b62a-01c231a9b003","Type":"ContainerDied","Data":"a49cead03208f94dceebdf53a25c29bef47f09c9f54141502e126df70216fe43"} Apr 16 16:50:35.244636 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.244592 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xfnx6" podStartSLOduration=138.015998544 podStartE2EDuration="2m20.244576109s" podCreationTimestamp="2026-04-16 16:48:15 +0000 UTC" firstStartedPulling="2026-04-16 16:50:32.440389238 +0000 UTC m=+171.501538753" lastFinishedPulling="2026-04-16 16:50:34.66896681 +0000 UTC m=+173.730116318" observedRunningTime="2026-04-16 16:50:35.243484335 +0000 UTC m=+174.304633878" watchObservedRunningTime="2026-04-16 16:50:35.244576109 +0000 UTC m=+174.305725641" Apr 16 16:50:35.259505 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.259073 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-59tvd" podStartSLOduration=4.126091654 podStartE2EDuration="6.259054347s" podCreationTimestamp="2026-04-16 16:50:29 +0000 UTC" firstStartedPulling="2026-04-16 16:50:30.130909482 +0000 UTC m=+169.192058997" lastFinishedPulling="2026-04-16 16:50:32.263872179 +0000 UTC m=+171.325021690" observedRunningTime="2026-04-16 16:50:35.258206203 +0000 UTC m=+174.319355735" watchObservedRunningTime="2026-04-16 16:50:35.259054347 +0000 UTC m=+174.320203878" Apr 16 16:50:35.363311 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.363213 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Apr 16 16:50:35.375272 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.375208 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.379604 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.379574 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Apr 16 16:50:35.380862 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.380773 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-f5f6q\"" Apr 16 16:50:35.382065 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.381006 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Apr 16 16:50:35.382065 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.381601 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Apr 16 16:50:35.382065 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.381796 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Apr 16 16:50:35.382065 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.381997 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Apr 16 16:50:35.382328 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.382181 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Apr 16 16:50:35.383986 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.382416 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Apr 16 16:50:35.383986 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.382652 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Apr 16 16:50:35.383986 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.382834 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Apr 16 16:50:35.383986 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.383105 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Apr 16 16:50:35.384496 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.384479 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Apr 16 16:50:35.385402 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.385379 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Apr 16 16:50:35.386002 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.385984 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-65oju51kn331h\"" Apr 16 16:50:35.387127 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.387105 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Apr 16 16:50:35.392335 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.392041 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Apr 16 16:50:35.456003 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.455841 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/05797269-685d-4822-90bf-01c7aa513b40-config-out\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.456003 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.455897 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05797269-685d-4822-90bf-01c7aa513b40-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.456003 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.455935 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/05797269-685d-4822-90bf-01c7aa513b40-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.456003 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.455964 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/05797269-685d-4822-90bf-01c7aa513b40-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.456373 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.456018 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05797269-685d-4822-90bf-01c7aa513b40-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.456373 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.456057 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05797269-685d-4822-90bf-01c7aa513b40-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.456373 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.456084 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/05797269-685d-4822-90bf-01c7aa513b40-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.456373 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.456104 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/05797269-685d-4822-90bf-01c7aa513b40-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.456373 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.456148 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4ttr\" (UniqueName: \"kubernetes.io/projected/05797269-685d-4822-90bf-01c7aa513b40-kube-api-access-d4ttr\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.456373 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.456176 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/05797269-685d-4822-90bf-01c7aa513b40-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.456373 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.456216 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/05797269-685d-4822-90bf-01c7aa513b40-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.456373 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.456265 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/05797269-685d-4822-90bf-01c7aa513b40-web-config\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.456373 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.456311 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/05797269-685d-4822-90bf-01c7aa513b40-config\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.456373 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.456335 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/05797269-685d-4822-90bf-01c7aa513b40-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.456373 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.456361 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/05797269-685d-4822-90bf-01c7aa513b40-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.456864 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.456384 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/05797269-685d-4822-90bf-01c7aa513b40-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.456864 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.456418 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/05797269-685d-4822-90bf-01c7aa513b40-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.456864 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.456478 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05797269-685d-4822-90bf-01c7aa513b40-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.561283 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.559728 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d4ttr\" (UniqueName: \"kubernetes.io/projected/05797269-685d-4822-90bf-01c7aa513b40-kube-api-access-d4ttr\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.561283 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.559789 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/05797269-685d-4822-90bf-01c7aa513b40-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.561283 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.559918 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/05797269-685d-4822-90bf-01c7aa513b40-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.561283 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.559949 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/05797269-685d-4822-90bf-01c7aa513b40-web-config\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.561283 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.559994 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/05797269-685d-4822-90bf-01c7aa513b40-config\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.561283 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.560019 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/05797269-685d-4822-90bf-01c7aa513b40-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.561283 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.560047 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/05797269-685d-4822-90bf-01c7aa513b40-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.561283 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.560072 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/05797269-685d-4822-90bf-01c7aa513b40-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.561283 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.560105 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/05797269-685d-4822-90bf-01c7aa513b40-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.561283 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.560162 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05797269-685d-4822-90bf-01c7aa513b40-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.561283 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.560204 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/05797269-685d-4822-90bf-01c7aa513b40-config-out\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.561283 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.560234 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05797269-685d-4822-90bf-01c7aa513b40-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.561283 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.560284 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/05797269-685d-4822-90bf-01c7aa513b40-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.561283 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.560318 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/05797269-685d-4822-90bf-01c7aa513b40-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.561283 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.560423 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05797269-685d-4822-90bf-01c7aa513b40-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.561283 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.560457 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05797269-685d-4822-90bf-01c7aa513b40-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.561283 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.560486 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/05797269-685d-4822-90bf-01c7aa513b40-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.562163 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.560511 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/05797269-685d-4822-90bf-01c7aa513b40-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.562163 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.561372 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05797269-685d-4822-90bf-01c7aa513b40-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.563661 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.562332 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05797269-685d-4822-90bf-01c7aa513b40-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.563661 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.563343 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05797269-685d-4822-90bf-01c7aa513b40-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.565295 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.564030 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05797269-685d-4822-90bf-01c7aa513b40-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.567831 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.567304 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/05797269-685d-4822-90bf-01c7aa513b40-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.567831 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.567703 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/05797269-685d-4822-90bf-01c7aa513b40-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.568638 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.568163 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/05797269-685d-4822-90bf-01c7aa513b40-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.568638 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.568192 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/05797269-685d-4822-90bf-01c7aa513b40-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.568638 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.568358 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/05797269-685d-4822-90bf-01c7aa513b40-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.569041 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.569021 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/05797269-685d-4822-90bf-01c7aa513b40-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.573467 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.571448 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/05797269-685d-4822-90bf-01c7aa513b40-config-out\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.573467 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.571683 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/05797269-685d-4822-90bf-01c7aa513b40-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.573467 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.571897 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/05797269-685d-4822-90bf-01c7aa513b40-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.573467 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.572582 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/05797269-685d-4822-90bf-01c7aa513b40-config\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.573467 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.572673 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/05797269-685d-4822-90bf-01c7aa513b40-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.573467 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.573364 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4ttr\" (UniqueName: \"kubernetes.io/projected/05797269-685d-4822-90bf-01c7aa513b40-kube-api-access-d4ttr\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.573467 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.573422 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/05797269-685d-4822-90bf-01c7aa513b40-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.573871 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.573747 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/05797269-685d-4822-90bf-01c7aa513b40-web-config\") pod \"prometheus-k8s-0\" (UID: \"05797269-685d-4822-90bf-01c7aa513b40\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.693964 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.693923 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:50:35.963932 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:35.963904 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Apr 16 16:50:36.181581 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:36.181492 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rzx7h" Apr 16 16:50:36.282666 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:50:36.282625 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05797269_685d_4822_90bf_01c7aa513b40.slice/crio-723db2960b96836e175e0c451ed63a4b46b58d282cb8d91c2b0c2070d98cec4e WatchSource:0}: Error finding container 723db2960b96836e175e0c451ed63a4b46b58d282cb8d91c2b0c2070d98cec4e: Status 404 returned error can't find the container with id 723db2960b96836e175e0c451ed63a4b46b58d282cb8d91c2b0c2070d98cec4e Apr 16 16:50:37.244308 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:37.244173 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5876b4bbc7-gmxs2" event={"ID":"d620b8bc-d805-441c-864d-5f23f7621826","Type":"ContainerStarted","Data":"93ed74e19725c8ef4d2d7f8c7c3a62b9cb382c3f32f5d9a339e50fb919c68dcb"} Apr 16 16:50:37.244763 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:37.244411 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5876b4bbc7-gmxs2" Apr 16 16:50:37.247530 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:37.247504 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78df5754bd-ngrn7" event={"ID":"05869c45-9aab-4f0a-93bd-06c4bba46382","Type":"ContainerStarted","Data":"1722a1083bb18c168c9ca9bdf204d8999eb735bb2f882d25b84004aecb2a0f18"} Apr 16 16:50:37.247668 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:37.247654 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78df5754bd-ngrn7" event={"ID":"05869c45-9aab-4f0a-93bd-06c4bba46382","Type":"ContainerStarted","Data":"6dddf8a1a5152f68b7447456d561e42d672a001baf39429b75c8ff6e0179f2d8"} Apr 16 16:50:37.247760 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:37.247749 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78df5754bd-ngrn7" event={"ID":"05869c45-9aab-4f0a-93bd-06c4bba46382","Type":"ContainerStarted","Data":"b1e1dd2822f3a1e5e54938e1efdf4c258f56edfcef12c4c6897c75adb9a103a4"} Apr 16 16:50:37.248626 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:37.248607 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-78df5754bd-ngrn7" Apr 16 16:50:37.250444 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:37.250424 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5876b4bbc7-gmxs2" Apr 16 16:50:37.250853 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:37.250834 2576 generic.go:358] "Generic (PLEG): container finished" podID="05797269-685d-4822-90bf-01c7aa513b40" containerID="a75ddde37a20f80fbc7e39d761c3e9e0486088ddd2b51f7d1fb014646ccbb8ec" exitCode=0 Apr 16 16:50:37.250931 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:37.250908 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"05797269-685d-4822-90bf-01c7aa513b40","Type":"ContainerDied","Data":"a75ddde37a20f80fbc7e39d761c3e9e0486088ddd2b51f7d1fb014646ccbb8ec"} Apr 16 16:50:37.250931 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:37.250928 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"05797269-685d-4822-90bf-01c7aa513b40","Type":"ContainerStarted","Data":"723db2960b96836e175e0c451ed63a4b46b58d282cb8d91c2b0c2070d98cec4e"} Apr 16 16:50:37.254206 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:37.254184 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e1c13e2d-7bc4-4764-b62a-01c231a9b003","Type":"ContainerStarted","Data":"a91d5cf24f95caf1265336521f9e286cfd118e26f697a3c83b079f2f4e18c973"} Apr 16 16:50:37.254311 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:37.254214 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e1c13e2d-7bc4-4764-b62a-01c231a9b003","Type":"ContainerStarted","Data":"17e8b688d0b07089e9ab91b48e18af14e05bad03b2bb179e7f24f10dde450ddc"} Apr 16 16:50:37.254311 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:37.254228 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e1c13e2d-7bc4-4764-b62a-01c231a9b003","Type":"ContainerStarted","Data":"66b9858980e0c9641a3fc22cfbe0a840e475168546a8b65a3f9bd5617e25be71"} Apr 16 16:50:37.254311 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:37.254259 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e1c13e2d-7bc4-4764-b62a-01c231a9b003","Type":"ContainerStarted","Data":"7a6c83b5a306228ef0342364d62376902a1b7abedddde2c9d420239b6b552efe"} Apr 16 16:50:37.254311 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:37.254273 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e1c13e2d-7bc4-4764-b62a-01c231a9b003","Type":"ContainerStarted","Data":"84516902e356168fa797c6f6f89f21abb30a5c84f5dea6216ded008b3e8803ae"} Apr 16 16:50:37.254311 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:37.254286 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e1c13e2d-7bc4-4764-b62a-01c231a9b003","Type":"ContainerStarted","Data":"25acef5e90abd20e2731d46530455df045f7b5cc50ec144568429eeb6b8810fa"} Apr 16 16:50:37.258310 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:37.258275 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5876b4bbc7-gmxs2" podStartSLOduration=2.341548136 podStartE2EDuration="4.258263925s" podCreationTimestamp="2026-04-16 16:50:33 +0000 UTC" firstStartedPulling="2026-04-16 16:50:34.802578117 +0000 UTC m=+173.863727642" lastFinishedPulling="2026-04-16 16:50:36.719293916 +0000 UTC m=+175.780443431" observedRunningTime="2026-04-16 16:50:37.257496089 +0000 UTC m=+176.318645621" watchObservedRunningTime="2026-04-16 16:50:37.258263925 +0000 UTC m=+176.319413457" Apr 16 16:50:37.314882 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:37.314834 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.026620794 podStartE2EDuration="7.314816232s" podCreationTimestamp="2026-04-16 16:50:30 +0000 UTC" firstStartedPulling="2026-04-16 16:50:32.434553576 +0000 UTC m=+171.495703085" lastFinishedPulling="2026-04-16 16:50:36.722749003 +0000 UTC m=+175.783898523" observedRunningTime="2026-04-16 16:50:37.313355867 +0000 UTC m=+176.374505398" watchObservedRunningTime="2026-04-16 16:50:37.314816232 +0000 UTC m=+176.375965762" Apr 16 16:50:37.334970 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:37.334920 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-78df5754bd-ngrn7" podStartSLOduration=1.351234201 podStartE2EDuration="5.334902852s" podCreationTimestamp="2026-04-16 16:50:32 +0000 UTC" firstStartedPulling="2026-04-16 16:50:32.730458026 +0000 UTC m=+171.791607535" lastFinishedPulling="2026-04-16 16:50:36.714126673 +0000 UTC m=+175.775276186" observedRunningTime="2026-04-16 16:50:37.332857244 +0000 UTC m=+176.394006774" watchObservedRunningTime="2026-04-16 16:50:37.334902852 +0000 UTC m=+176.396052382" Apr 16 16:50:38.885925 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:38.885890 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6fb554db88-dcz4j" Apr 16 16:50:38.886378 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:38.885969 2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6fb554db88-dcz4j" Apr 16 16:50:38.891865 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:38.891841 2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6fb554db88-dcz4j" Apr 16 16:50:39.268568 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:39.268478 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6fb554db88-dcz4j" Apr 16 16:50:39.270896 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:39.270871 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-78df5754bd-ngrn7" Apr 16 16:50:40.271080 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:40.271018 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"05797269-685d-4822-90bf-01c7aa513b40","Type":"ContainerStarted","Data":"dc9240801100763ca6e32fc1c01f9b03bd4baeb6449e6fa1efb60c5e6437b5a4"} Apr 16 16:50:40.271080 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:40.271067 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"05797269-685d-4822-90bf-01c7aa513b40","Type":"ContainerStarted","Data":"0e060f1d887cec581cfc2c171e5381b447f2a2dcd683eff2ddf42c396b750501"} Apr 16 16:50:40.271080 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:40.271081 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"05797269-685d-4822-90bf-01c7aa513b40","Type":"ContainerStarted","Data":"4beddeafb9e03e28c3a0e18ea5c7e364a5eba32a96a712bce64e0e33fab53601"} Apr 16 16:50:40.271605 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:40.271094 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"05797269-685d-4822-90bf-01c7aa513b40","Type":"ContainerStarted","Data":"3fbadd9e06629fa146f07a386eaca72bdc2d15417e719002b5cff42741cbaad0"} Apr 16 16:50:41.279306 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:41.279266 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"05797269-685d-4822-90bf-01c7aa513b40","Type":"ContainerStarted","Data":"e1647937e48a8bb70c4e50b120030e1257e2b324c7bddf866f636e3c93dde8dc"} Apr 16 16:50:41.279779 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:41.279748 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"05797269-685d-4822-90bf-01c7aa513b40","Type":"ContainerStarted","Data":"0efd30a2609662898843f8c6d6848ef251e4d79808ad53578e7903b23e50d2ab"} Apr 16 16:50:41.305051 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:41.304988 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.614302796 podStartE2EDuration="6.30496896s" podCreationTimestamp="2026-04-16 16:50:35 +0000 UTC" firstStartedPulling="2026-04-16 16:50:37.252165003 +0000 UTC m=+176.313314512" lastFinishedPulling="2026-04-16 16:50:39.942831165 +0000 UTC m=+179.003980676" observedRunningTime="2026-04-16 16:50:41.302461559 +0000 UTC m=+180.363611092" watchObservedRunningTime="2026-04-16 16:50:41.30496896 +0000 UTC m=+180.366118492" Apr 16 16:50:45.105824 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:45.105787 2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fb554db88-dcz4j"] Apr 16 16:50:45.695030 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:50:45.694990 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:51:08.361087 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:08.361000 2576 generic.go:358] "Generic (PLEG): container finished" podID="4bb5aacd-20aa-4fb6-ae32-eee3df133067" containerID="67c4a59a42cf2f73ee8aaef1c197522159a2f20d4cab8c65c7810fa1c8449fb0" exitCode=0 Apr 16 16:51:08.361496 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:08.361076 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-756bb7d76f-dzm7p" event={"ID":"4bb5aacd-20aa-4fb6-ae32-eee3df133067","Type":"ContainerDied","Data":"67c4a59a42cf2f73ee8aaef1c197522159a2f20d4cab8c65c7810fa1c8449fb0"} Apr 16 16:51:08.361496 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:08.361413 2576 scope.go:117] "RemoveContainer" containerID="67c4a59a42cf2f73ee8aaef1c197522159a2f20d4cab8c65c7810fa1c8449fb0" Apr 16 16:51:09.366009 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:09.365969 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-756bb7d76f-dzm7p" event={"ID":"4bb5aacd-20aa-4fb6-ae32-eee3df133067","Type":"ContainerStarted","Data":"a8fea9ce826742b7a555a774331c902f69d040db427d200ff64c714b5379f4e1"} Apr 16 16:51:10.125471 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:10.125431 2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-6fb554db88-dcz4j" podUID="1d9783fe-808a-4862-81c1-31dcf22df1a4" containerName="console" containerID="cri-o://99811446d4e46a25015a2df57ff6a7babc8221367ed53d6b7496b1c70c53aec3" gracePeriod=15 Apr 16 16:51:10.370403 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:10.370381 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6fb554db88-dcz4j_1d9783fe-808a-4862-81c1-31dcf22df1a4/console/0.log" Apr 16 16:51:10.370785 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:10.370417 2576 generic.go:358] "Generic (PLEG): container finished" podID="1d9783fe-808a-4862-81c1-31dcf22df1a4" containerID="99811446d4e46a25015a2df57ff6a7babc8221367ed53d6b7496b1c70c53aec3" exitCode=2 Apr 16 16:51:10.370785 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:10.370494 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fb554db88-dcz4j" event={"ID":"1d9783fe-808a-4862-81c1-31dcf22df1a4","Type":"ContainerDied","Data":"99811446d4e46a25015a2df57ff6a7babc8221367ed53d6b7496b1c70c53aec3"} Apr 16 16:51:10.370785 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:10.370526 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fb554db88-dcz4j" event={"ID":"1d9783fe-808a-4862-81c1-31dcf22df1a4","Type":"ContainerDied","Data":"a533715623674db382a8ac1871dc17464a08b689d5e875af35a22f9697561b6d"} Apr 16 16:51:10.370785 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:10.370538 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a533715623674db382a8ac1871dc17464a08b689d5e875af35a22f9697561b6d" Apr 16 16:51:10.378563 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:10.378515 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6fb554db88-dcz4j_1d9783fe-808a-4862-81c1-31dcf22df1a4/console/0.log" Apr 16 16:51:10.378672 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:10.378571 2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fb554db88-dcz4j" Apr 16 16:51:10.494897 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:10.494863 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d9783fe-808a-4862-81c1-31dcf22df1a4-trusted-ca-bundle\") pod \"1d9783fe-808a-4862-81c1-31dcf22df1a4\" (UID: \"1d9783fe-808a-4862-81c1-31dcf22df1a4\") " Apr 16 16:51:10.495058 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:10.494935 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d9783fe-808a-4862-81c1-31dcf22df1a4-console-config\") pod \"1d9783fe-808a-4862-81c1-31dcf22df1a4\" (UID: \"1d9783fe-808a-4862-81c1-31dcf22df1a4\") " Apr 16 16:51:10.495058 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:10.494971 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d9783fe-808a-4862-81c1-31dcf22df1a4-oauth-serving-cert\") pod \"1d9783fe-808a-4862-81c1-31dcf22df1a4\" (UID: \"1d9783fe-808a-4862-81c1-31dcf22df1a4\") " Apr 16 16:51:10.495058 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:10.495009 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d9783fe-808a-4862-81c1-31dcf22df1a4-console-oauth-config\") pod \"1d9783fe-808a-4862-81c1-31dcf22df1a4\" (UID: \"1d9783fe-808a-4862-81c1-31dcf22df1a4\") " Apr 16 16:51:10.495058 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:10.495047 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c69qs\" (UniqueName: \"kubernetes.io/projected/1d9783fe-808a-4862-81c1-31dcf22df1a4-kube-api-access-c69qs\") pod \"1d9783fe-808a-4862-81c1-31dcf22df1a4\" (UID: \"1d9783fe-808a-4862-81c1-31dcf22df1a4\") " Apr 16 16:51:10.495222 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:10.495076 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d9783fe-808a-4862-81c1-31dcf22df1a4-service-ca\") pod \"1d9783fe-808a-4862-81c1-31dcf22df1a4\" (UID: \"1d9783fe-808a-4862-81c1-31dcf22df1a4\") " Apr 16 16:51:10.495222 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:10.495099 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d9783fe-808a-4862-81c1-31dcf22df1a4-console-serving-cert\") pod \"1d9783fe-808a-4862-81c1-31dcf22df1a4\" (UID: \"1d9783fe-808a-4862-81c1-31dcf22df1a4\") " Apr 16 16:51:10.495499 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:10.495473 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d9783fe-808a-4862-81c1-31dcf22df1a4-console-config" (OuterVolumeSpecName: "console-config") pod "1d9783fe-808a-4862-81c1-31dcf22df1a4" (UID: "1d9783fe-808a-4862-81c1-31dcf22df1a4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 16:51:10.495581 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:10.495494 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d9783fe-808a-4862-81c1-31dcf22df1a4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1d9783fe-808a-4862-81c1-31dcf22df1a4" (UID: "1d9783fe-808a-4862-81c1-31dcf22df1a4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 16:51:10.495654 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:10.495536 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d9783fe-808a-4862-81c1-31dcf22df1a4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1d9783fe-808a-4862-81c1-31dcf22df1a4" (UID: "1d9783fe-808a-4862-81c1-31dcf22df1a4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 16:51:10.495734 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:10.495708 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d9783fe-808a-4862-81c1-31dcf22df1a4-service-ca" (OuterVolumeSpecName: "service-ca") pod "1d9783fe-808a-4862-81c1-31dcf22df1a4" (UID: "1d9783fe-808a-4862-81c1-31dcf22df1a4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 16:51:10.497636 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:10.497604 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d9783fe-808a-4862-81c1-31dcf22df1a4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1d9783fe-808a-4862-81c1-31dcf22df1a4" (UID: "1d9783fe-808a-4862-81c1-31dcf22df1a4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 16:51:10.497746 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:10.497727 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d9783fe-808a-4862-81c1-31dcf22df1a4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1d9783fe-808a-4862-81c1-31dcf22df1a4" (UID: "1d9783fe-808a-4862-81c1-31dcf22df1a4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 16:51:10.497838 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:10.497816 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d9783fe-808a-4862-81c1-31dcf22df1a4-kube-api-access-c69qs" (OuterVolumeSpecName: "kube-api-access-c69qs") pod "1d9783fe-808a-4862-81c1-31dcf22df1a4" (UID: "1d9783fe-808a-4862-81c1-31dcf22df1a4"). InnerVolumeSpecName "kube-api-access-c69qs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 16:51:10.596208 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:10.596171 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d9783fe-808a-4862-81c1-31dcf22df1a4-trusted-ca-bundle\") on node \"ip-10-0-142-37.ec2.internal\" DevicePath \"\"" Apr 16 16:51:10.596208 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:10.596203 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d9783fe-808a-4862-81c1-31dcf22df1a4-console-config\") on node \"ip-10-0-142-37.ec2.internal\" DevicePath \"\"" Apr 16 16:51:10.596208 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:10.596213 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d9783fe-808a-4862-81c1-31dcf22df1a4-oauth-serving-cert\") on node \"ip-10-0-142-37.ec2.internal\" DevicePath \"\"" Apr 16 16:51:10.596463 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:10.596222 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d9783fe-808a-4862-81c1-31dcf22df1a4-console-oauth-config\") on node \"ip-10-0-142-37.ec2.internal\" DevicePath \"\"" Apr 16 16:51:10.596463 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:10.596231 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c69qs\" (UniqueName: \"kubernetes.io/projected/1d9783fe-808a-4862-81c1-31dcf22df1a4-kube-api-access-c69qs\") on node \"ip-10-0-142-37.ec2.internal\" DevicePath \"\"" Apr 16 16:51:10.596463 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:10.596240 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d9783fe-808a-4862-81c1-31dcf22df1a4-service-ca\") on node \"ip-10-0-142-37.ec2.internal\" DevicePath \"\"" Apr 16 16:51:10.596463 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:10.596267 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d9783fe-808a-4862-81c1-31dcf22df1a4-console-serving-cert\") on node \"ip-10-0-142-37.ec2.internal\" DevicePath \"\"" Apr 16 16:51:11.373818 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:11.373787 2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fb554db88-dcz4j" Apr 16 16:51:11.393144 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:11.393120 2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fb554db88-dcz4j"] Apr 16 16:51:11.395808 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:11.395787 2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6fb554db88-dcz4j"] Apr 16 16:51:11.592430 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:11.592398 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d9783fe-808a-4862-81c1-31dcf22df1a4" path="/var/lib/kubelet/pods/1d9783fe-808a-4862-81c1-31dcf22df1a4/volumes" Apr 16 16:51:35.694585 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:35.694551 2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:51:35.710900 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:35.710875 2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:51:36.467748 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:36.467721 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:51:52.367335 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:52.367231 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/412718e2-2075-4c7b-a6a9-3b94ed57564c-metrics-certs\") pod \"network-metrics-daemon-hk5gr\" (UID: \"412718e2-2075-4c7b-a6a9-3b94ed57564c\") " pod="openshift-multus/network-metrics-daemon-hk5gr" Apr 16 16:51:52.369676 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:52.369655 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/412718e2-2075-4c7b-a6a9-3b94ed57564c-metrics-certs\") pod \"network-metrics-daemon-hk5gr\" (UID: \"412718e2-2075-4c7b-a6a9-3b94ed57564c\") " pod="openshift-multus/network-metrics-daemon-hk5gr" Apr 16 16:51:52.390766 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:52.390736 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-877l2\"" Apr 16 16:51:52.399491 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:52.399473 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk5gr" Apr 16 16:51:52.518681 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:52.518648 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hk5gr"] Apr 16 16:51:52.521102 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:51:52.521067 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod412718e2_2075_4c7b_a6a9_3b94ed57564c.slice/crio-e655550f5d8fce468f51a15b20c78e72cebc67d561baf9c47dc0703be6f8ebd7 WatchSource:0}: Error finding container e655550f5d8fce468f51a15b20c78e72cebc67d561baf9c47dc0703be6f8ebd7: Status 404 returned error can't find the container with id e655550f5d8fce468f51a15b20c78e72cebc67d561baf9c47dc0703be6f8ebd7 Apr 16 16:51:53.487974 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:53.487933 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-674cd685d5-c72ss"] Apr 16 16:51:53.488450 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:53.488434 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d9783fe-808a-4862-81c1-31dcf22df1a4" containerName="console" Apr 16 16:51:53.488450 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:53.488450 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d9783fe-808a-4862-81c1-31dcf22df1a4" containerName="console" Apr 16 16:51:53.488580 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:53.488545 2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="1d9783fe-808a-4862-81c1-31dcf22df1a4" containerName="console" Apr 16 16:51:53.498330 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:53.498306 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-674cd685d5-c72ss" Apr 16 16:51:53.500390 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:53.500341 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Apr 16 16:51:53.500390 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:53.500373 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-x8sjb\"" Apr 16 16:51:53.500972 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:53.500801 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Apr 16 16:51:53.500972 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:53.500830 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Apr 16 16:51:53.500972 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:53.500837 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Apr 16 16:51:53.501546 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:53.501526 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Apr 16 16:51:53.505938 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:53.505894 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hk5gr" event={"ID":"412718e2-2075-4c7b-a6a9-3b94ed57564c","Type":"ContainerStarted","Data":"e655550f5d8fce468f51a15b20c78e72cebc67d561baf9c47dc0703be6f8ebd7"} Apr 16 16:51:53.509771 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:53.509109 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-674cd685d5-c72ss"] Apr 16 16:51:53.522411 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:53.520657 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Apr 16 16:51:53.578144 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:53.578117 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a5a5bf5d-ebc9-47c7-a825-754a41a34908-secret-telemeter-client\") pod \"telemeter-client-674cd685d5-c72ss\" (UID: \"a5a5bf5d-ebc9-47c7-a825-754a41a34908\") " pod="openshift-monitoring/telemeter-client-674cd685d5-c72ss" Apr 16 16:51:53.578236 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:53.578169 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a5a5bf5d-ebc9-47c7-a825-754a41a34908-telemeter-client-tls\") pod \"telemeter-client-674cd685d5-c72ss\" (UID: \"a5a5bf5d-ebc9-47c7-a825-754a41a34908\") " pod="openshift-monitoring/telemeter-client-674cd685d5-c72ss" Apr 16 16:51:53.578297 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:53.578277 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a5a5bf5d-ebc9-47c7-a825-754a41a34908-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-674cd685d5-c72ss\" (UID: \"a5a5bf5d-ebc9-47c7-a825-754a41a34908\") " pod="openshift-monitoring/telemeter-client-674cd685d5-c72ss" Apr 16 16:51:53.578352 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:53.578319 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5a5bf5d-ebc9-47c7-a825-754a41a34908-serving-certs-ca-bundle\") pod \"telemeter-client-674cd685d5-c72ss\" (UID: \"a5a5bf5d-ebc9-47c7-a825-754a41a34908\") " pod="openshift-monitoring/telemeter-client-674cd685d5-c72ss" Apr 16 16:51:53.578352 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:53.578337 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a5a5bf5d-ebc9-47c7-a825-754a41a34908-federate-client-tls\") pod \"telemeter-client-674cd685d5-c72ss\" (UID: \"a5a5bf5d-ebc9-47c7-a825-754a41a34908\") " pod="openshift-monitoring/telemeter-client-674cd685d5-c72ss" Apr 16 16:51:53.578486 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:53.578431 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5a5bf5d-ebc9-47c7-a825-754a41a34908-metrics-client-ca\") pod \"telemeter-client-674cd685d5-c72ss\" (UID: \"a5a5bf5d-ebc9-47c7-a825-754a41a34908\") " pod="openshift-monitoring/telemeter-client-674cd685d5-c72ss" Apr 16 16:51:53.578486 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:53.578452 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5a5bf5d-ebc9-47c7-a825-754a41a34908-telemeter-trusted-ca-bundle\") pod \"telemeter-client-674cd685d5-c72ss\" (UID: \"a5a5bf5d-ebc9-47c7-a825-754a41a34908\") " pod="openshift-monitoring/telemeter-client-674cd685d5-c72ss" Apr 16 16:51:53.578600 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:53.578573 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj4kq\" (UniqueName: \"kubernetes.io/projected/a5a5bf5d-ebc9-47c7-a825-754a41a34908-kube-api-access-wj4kq\") pod \"telemeter-client-674cd685d5-c72ss\" (UID: \"a5a5bf5d-ebc9-47c7-a825-754a41a34908\") " pod="openshift-monitoring/telemeter-client-674cd685d5-c72ss" Apr 16 16:51:53.679372 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:53.679346 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wj4kq\" (UniqueName: \"kubernetes.io/projected/a5a5bf5d-ebc9-47c7-a825-754a41a34908-kube-api-access-wj4kq\") pod \"telemeter-client-674cd685d5-c72ss\" (UID: \"a5a5bf5d-ebc9-47c7-a825-754a41a34908\") " pod="openshift-monitoring/telemeter-client-674cd685d5-c72ss" Apr 16 16:51:53.679516 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:53.679485 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a5a5bf5d-ebc9-47c7-a825-754a41a34908-secret-telemeter-client\") pod \"telemeter-client-674cd685d5-c72ss\" (UID: \"a5a5bf5d-ebc9-47c7-a825-754a41a34908\") " pod="openshift-monitoring/telemeter-client-674cd685d5-c72ss" Apr 16 16:51:53.679641 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:53.679563 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a5a5bf5d-ebc9-47c7-a825-754a41a34908-telemeter-client-tls\") pod \"telemeter-client-674cd685d5-c72ss\" (UID: \"a5a5bf5d-ebc9-47c7-a825-754a41a34908\") " pod="openshift-monitoring/telemeter-client-674cd685d5-c72ss" Apr 16 16:51:53.679744 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:53.679724 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a5a5bf5d-ebc9-47c7-a825-754a41a34908-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-674cd685d5-c72ss\" (UID: \"a5a5bf5d-ebc9-47c7-a825-754a41a34908\") " pod="openshift-monitoring/telemeter-client-674cd685d5-c72ss" Apr 16 16:51:53.679809 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:53.679791 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5a5bf5d-ebc9-47c7-a825-754a41a34908-serving-certs-ca-bundle\") pod \"telemeter-client-674cd685d5-c72ss\" (UID: \"a5a5bf5d-ebc9-47c7-a825-754a41a34908\") " pod="openshift-monitoring/telemeter-client-674cd685d5-c72ss" Apr 16 16:51:53.679865 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:53.679826 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a5a5bf5d-ebc9-47c7-a825-754a41a34908-federate-client-tls\") pod \"telemeter-client-674cd685d5-c72ss\" (UID: \"a5a5bf5d-ebc9-47c7-a825-754a41a34908\") " pod="openshift-monitoring/telemeter-client-674cd685d5-c72ss" Apr 16 16:51:53.679918 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:53.679898 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5a5bf5d-ebc9-47c7-a825-754a41a34908-metrics-client-ca\") pod \"telemeter-client-674cd685d5-c72ss\" (UID: \"a5a5bf5d-ebc9-47c7-a825-754a41a34908\") " pod="openshift-monitoring/telemeter-client-674cd685d5-c72ss" Apr 16 16:51:53.679966 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:53.679930 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5a5bf5d-ebc9-47c7-a825-754a41a34908-telemeter-trusted-ca-bundle\") pod \"telemeter-client-674cd685d5-c72ss\" (UID: \"a5a5bf5d-ebc9-47c7-a825-754a41a34908\") " pod="openshift-monitoring/telemeter-client-674cd685d5-c72ss" Apr 16 16:51:53.680802 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:53.680690 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5a5bf5d-ebc9-47c7-a825-754a41a34908-serving-certs-ca-bundle\") pod \"telemeter-client-674cd685d5-c72ss\" (UID: \"a5a5bf5d-ebc9-47c7-a825-754a41a34908\") " pod="openshift-monitoring/telemeter-client-674cd685d5-c72ss" Apr 16 16:51:53.680802 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:53.680747 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5a5bf5d-ebc9-47c7-a825-754a41a34908-telemeter-trusted-ca-bundle\") pod \"telemeter-client-674cd685d5-c72ss\" (UID: \"a5a5bf5d-ebc9-47c7-a825-754a41a34908\") " pod="openshift-monitoring/telemeter-client-674cd685d5-c72ss" Apr 16 16:51:53.680957 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:53.680900 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5a5bf5d-ebc9-47c7-a825-754a41a34908-metrics-client-ca\") pod \"telemeter-client-674cd685d5-c72ss\" (UID: \"a5a5bf5d-ebc9-47c7-a825-754a41a34908\") " pod="openshift-monitoring/telemeter-client-674cd685d5-c72ss" Apr 16 16:51:53.682612 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:53.682582 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a5a5bf5d-ebc9-47c7-a825-754a41a34908-federate-client-tls\") pod \"telemeter-client-674cd685d5-c72ss\" (UID: \"a5a5bf5d-ebc9-47c7-a825-754a41a34908\") " pod="openshift-monitoring/telemeter-client-674cd685d5-c72ss" Apr 16 16:51:53.682705 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:53.682696 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a5a5bf5d-ebc9-47c7-a825-754a41a34908-telemeter-client-tls\") pod \"telemeter-client-674cd685d5-c72ss\" (UID: \"a5a5bf5d-ebc9-47c7-a825-754a41a34908\") " pod="openshift-monitoring/telemeter-client-674cd685d5-c72ss" Apr 16 16:51:53.682893 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:53.682873 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a5a5bf5d-ebc9-47c7-a825-754a41a34908-secret-telemeter-client\") pod \"telemeter-client-674cd685d5-c72ss\" (UID: \"a5a5bf5d-ebc9-47c7-a825-754a41a34908\") " pod="openshift-monitoring/telemeter-client-674cd685d5-c72ss" Apr 16 16:51:53.683227 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:53.683210 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a5a5bf5d-ebc9-47c7-a825-754a41a34908-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-674cd685d5-c72ss\" (UID: \"a5a5bf5d-ebc9-47c7-a825-754a41a34908\") " pod="openshift-monitoring/telemeter-client-674cd685d5-c72ss" Apr 16 16:51:53.687395 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:53.687375 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj4kq\" (UniqueName: \"kubernetes.io/projected/a5a5bf5d-ebc9-47c7-a825-754a41a34908-kube-api-access-wj4kq\") pod \"telemeter-client-674cd685d5-c72ss\" (UID: \"a5a5bf5d-ebc9-47c7-a825-754a41a34908\") " pod="openshift-monitoring/telemeter-client-674cd685d5-c72ss" Apr 16 16:51:53.824665 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:53.824632 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-674cd685d5-c72ss" Apr 16 16:51:53.947174 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:53.947056 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-674cd685d5-c72ss"] Apr 16 16:51:53.950376 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:51:53.950349 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5a5bf5d_ebc9_47c7_a825_754a41a34908.slice/crio-edcfe08323940054f5806b369e6cbd991581e4b099783de831fb042d0fa87a8e WatchSource:0}: Error finding container edcfe08323940054f5806b369e6cbd991581e4b099783de831fb042d0fa87a8e: Status 404 returned error can't find the container with id edcfe08323940054f5806b369e6cbd991581e4b099783de831fb042d0fa87a8e Apr 16 16:51:54.511128 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:54.511077 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hk5gr" event={"ID":"412718e2-2075-4c7b-a6a9-3b94ed57564c","Type":"ContainerStarted","Data":"e0f9be235f208484473bef8679e8fec6c53eacbdcf1a20bfd4426785c0eb05ee"} Apr 16 16:51:54.511128 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:54.511126 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hk5gr" event={"ID":"412718e2-2075-4c7b-a6a9-3b94ed57564c","Type":"ContainerStarted","Data":"55527920fb0c28415ffa961535d08e87364199d506eae763ef64a4cf5209819c"} Apr 16 16:51:54.512365 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:54.512337 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-674cd685d5-c72ss" event={"ID":"a5a5bf5d-ebc9-47c7-a825-754a41a34908","Type":"ContainerStarted","Data":"edcfe08323940054f5806b369e6cbd991581e4b099783de831fb042d0fa87a8e"} Apr 16 16:51:54.529077 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:54.529015 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hk5gr" podStartSLOduration=252.634945556 podStartE2EDuration="4m13.529000817s" podCreationTimestamp="2026-04-16 16:47:41 +0000 UTC" firstStartedPulling="2026-04-16 16:51:52.523294047 +0000 UTC m=+251.584443557" lastFinishedPulling="2026-04-16 16:51:53.417349295 +0000 UTC m=+252.478498818" observedRunningTime="2026-04-16 16:51:54.527111938 +0000 UTC m=+253.588261470" watchObservedRunningTime="2026-04-16 16:51:54.529000817 +0000 UTC m=+253.590150411" Apr 16 16:51:55.516943 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:55.516904 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-674cd685d5-c72ss" event={"ID":"a5a5bf5d-ebc9-47c7-a825-754a41a34908","Type":"ContainerStarted","Data":"8a40d77354440367b459dd4920c13b263326478495b926678d4f322cc21d8965"} Apr 16 16:51:55.516943 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:55.516944 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-674cd685d5-c72ss" event={"ID":"a5a5bf5d-ebc9-47c7-a825-754a41a34908","Type":"ContainerStarted","Data":"e83e53d1794ffd6a8fb7bdb10987db73d49c62d3723fb0490abc6a9f32f04951"} Apr 16 16:51:55.517443 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:55.516959 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-674cd685d5-c72ss" event={"ID":"a5a5bf5d-ebc9-47c7-a825-754a41a34908","Type":"ContainerStarted","Data":"5b78e5c709732a4d9a31ea52dc660d4104ef4d1132794ce57ce729725ac63323"} Apr 16 16:51:55.538199 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:51:55.538150 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-674cd685d5-c72ss" podStartSLOduration=1.194456974 podStartE2EDuration="2.538135422s" podCreationTimestamp="2026-04-16 16:51:53 +0000 UTC" firstStartedPulling="2026-04-16 16:51:53.952278575 +0000 UTC m=+253.013428084" lastFinishedPulling="2026-04-16 16:51:55.295957023 +0000 UTC m=+254.357106532" observedRunningTime="2026-04-16 16:51:55.537791286 +0000 UTC m=+254.598940818" watchObservedRunningTime="2026-04-16 16:51:55.538135422 +0000 UTC m=+254.599284985" Apr 16 16:52:41.463729 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:52:41.463697 2576 kubelet.go:1628] "Image garbage collection succeeded" Apr 16 16:56:14.902500 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:14.902464 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/s3-init-fxl9h"] Apr 16 16:56:14.905980 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:14.905960 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-fxl9h" Apr 16 16:56:14.908111 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:14.908087 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Apr 16 16:56:14.908111 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:14.908098 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Apr 16 16:56:14.908277 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:14.908159 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-9c52w\"" Apr 16 16:56:14.908543 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:14.908524 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Apr 16 16:56:14.911298 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:14.911276 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-fxl9h"] Apr 16 16:56:14.989362 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:14.989318 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfhdx\" (UniqueName: \"kubernetes.io/projected/182c48d6-e302-4868-b398-a0447e32a794-kube-api-access-hfhdx\") pod \"s3-init-fxl9h\" (UID: \"182c48d6-e302-4868-b398-a0447e32a794\") " pod="kserve/s3-init-fxl9h" Apr 16 16:56:15.090717 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:15.090673 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hfhdx\" (UniqueName: \"kubernetes.io/projected/182c48d6-e302-4868-b398-a0447e32a794-kube-api-access-hfhdx\") pod \"s3-init-fxl9h\" (UID: \"182c48d6-e302-4868-b398-a0447e32a794\") " pod="kserve/s3-init-fxl9h" Apr 16 16:56:15.098531 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:15.098500 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfhdx\" (UniqueName: \"kubernetes.io/projected/182c48d6-e302-4868-b398-a0447e32a794-kube-api-access-hfhdx\") pod \"s3-init-fxl9h\" (UID: \"182c48d6-e302-4868-b398-a0447e32a794\") " pod="kserve/s3-init-fxl9h" Apr 16 16:56:15.232121 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:15.232025 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-fxl9h" Apr 16 16:56:15.349768 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:15.349743 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-fxl9h"] Apr 16 16:56:15.352024 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:56:15.351996 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod182c48d6_e302_4868_b398_a0447e32a794.slice/crio-e4b330363636611bd88da63b359f9952be075de54e1a3cdf100bf0910a432939 WatchSource:0}: Error finding container e4b330363636611bd88da63b359f9952be075de54e1a3cdf100bf0910a432939: Status 404 returned error can't find the container with id e4b330363636611bd88da63b359f9952be075de54e1a3cdf100bf0910a432939 Apr 16 16:56:15.353750 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:15.353729 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Apr 16 16:56:16.297955 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:16.297911 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-fxl9h" event={"ID":"182c48d6-e302-4868-b398-a0447e32a794","Type":"ContainerStarted","Data":"e4b330363636611bd88da63b359f9952be075de54e1a3cdf100bf0910a432939"} Apr 16 16:56:20.311775 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:20.311705 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-fxl9h" event={"ID":"182c48d6-e302-4868-b398-a0447e32a794","Type":"ContainerStarted","Data":"dad561c2a3f8059ab3d8e630a80c00af1a60a1ffb2af928c58cebb48d3135d6e"} Apr 16 16:56:20.326154 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:20.326103 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/s3-init-fxl9h" podStartSLOduration=1.867314747 podStartE2EDuration="6.326087588s" podCreationTimestamp="2026-04-16 16:56:14 +0000 UTC" firstStartedPulling="2026-04-16 16:56:15.353915253 +0000 UTC m=+514.415064778" lastFinishedPulling="2026-04-16 16:56:19.812688108 +0000 UTC m=+518.873837619" observedRunningTime="2026-04-16 16:56:20.324775267 +0000 UTC m=+519.385924797" watchObservedRunningTime="2026-04-16 16:56:20.326087588 +0000 UTC m=+519.387237118" Apr 16 16:56:23.322483 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:23.322453 2576 generic.go:358] "Generic (PLEG): container finished" podID="182c48d6-e302-4868-b398-a0447e32a794" containerID="dad561c2a3f8059ab3d8e630a80c00af1a60a1ffb2af928c58cebb48d3135d6e" exitCode=0 Apr 16 16:56:23.322870 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:23.322536 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-fxl9h" event={"ID":"182c48d6-e302-4868-b398-a0447e32a794","Type":"ContainerDied","Data":"dad561c2a3f8059ab3d8e630a80c00af1a60a1ffb2af928c58cebb48d3135d6e"} Apr 16 16:56:24.448061 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:24.448038 2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-fxl9h" Apr 16 16:56:24.577932 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:24.577845 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfhdx\" (UniqueName: \"kubernetes.io/projected/182c48d6-e302-4868-b398-a0447e32a794-kube-api-access-hfhdx\") pod \"182c48d6-e302-4868-b398-a0447e32a794\" (UID: \"182c48d6-e302-4868-b398-a0447e32a794\") " Apr 16 16:56:24.580183 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:24.580159 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/182c48d6-e302-4868-b398-a0447e32a794-kube-api-access-hfhdx" (OuterVolumeSpecName: "kube-api-access-hfhdx") pod "182c48d6-e302-4868-b398-a0447e32a794" (UID: "182c48d6-e302-4868-b398-a0447e32a794"). InnerVolumeSpecName "kube-api-access-hfhdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 16:56:24.678567 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:24.678517 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfhdx\" (UniqueName: \"kubernetes.io/projected/182c48d6-e302-4868-b398-a0447e32a794-kube-api-access-hfhdx\") on node \"ip-10-0-142-37.ec2.internal\" DevicePath \"\"" Apr 16 16:56:25.329920 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:25.329886 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-fxl9h" event={"ID":"182c48d6-e302-4868-b398-a0447e32a794","Type":"ContainerDied","Data":"e4b330363636611bd88da63b359f9952be075de54e1a3cdf100bf0910a432939"} Apr 16 16:56:25.329920 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:25.329902 2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-fxl9h" Apr 16 16:56:25.329920 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:25.329918 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4b330363636611bd88da63b359f9952be075de54e1a3cdf100bf0910a432939" Apr 16 16:56:35.443081 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:35.443047 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-vqqz2"] Apr 16 16:56:35.443469 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:35.443434 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="182c48d6-e302-4868-b398-a0447e32a794" containerName="s3-init" Apr 16 16:56:35.443469 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:35.443447 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="182c48d6-e302-4868-b398-a0447e32a794" containerName="s3-init" Apr 16 16:56:35.443536 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:35.443497 2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="182c48d6-e302-4868-b398-a0447e32a794" containerName="s3-init" Apr 16 16:56:35.446736 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:35.446717 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-vqqz2" Apr 16 16:56:35.448690 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:35.448651 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Apr 16 16:56:35.448811 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:35.448658 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Apr 16 16:56:35.448811 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:35.448718 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\"" Apr 16 16:56:35.449271 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:35.449235 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-9c52w\"" Apr 16 16:56:35.452302 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:35.452283 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-vqqz2"] Apr 16 16:56:35.569959 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:35.569923 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5xjg\" (UniqueName: \"kubernetes.io/projected/14b0808e-16e3-4b10-993d-c7c6d9fd96d2-kube-api-access-p5xjg\") pod \"s3-tls-init-custom-vqqz2\" (UID: \"14b0808e-16e3-4b10-993d-c7c6d9fd96d2\") " pod="kserve/s3-tls-init-custom-vqqz2" Apr 16 16:56:35.670864 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:35.670822 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p5xjg\" (UniqueName: \"kubernetes.io/projected/14b0808e-16e3-4b10-993d-c7c6d9fd96d2-kube-api-access-p5xjg\") pod \"s3-tls-init-custom-vqqz2\" (UID: \"14b0808e-16e3-4b10-993d-c7c6d9fd96d2\") " pod="kserve/s3-tls-init-custom-vqqz2" Apr 16 16:56:35.679324 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:35.679298 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5xjg\" (UniqueName: \"kubernetes.io/projected/14b0808e-16e3-4b10-993d-c7c6d9fd96d2-kube-api-access-p5xjg\") pod \"s3-tls-init-custom-vqqz2\" (UID: \"14b0808e-16e3-4b10-993d-c7c6d9fd96d2\") " pod="kserve/s3-tls-init-custom-vqqz2" Apr 16 16:56:35.768822 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:35.768735 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-vqqz2" Apr 16 16:56:35.890393 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:35.890371 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-vqqz2"] Apr 16 16:56:35.892718 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:56:35.892694 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14b0808e_16e3_4b10_993d_c7c6d9fd96d2.slice/crio-469e49ef398a3d696e5f32a37f7eb6d85239836599ecaa6f372b991d53814665 WatchSource:0}: Error finding container 469e49ef398a3d696e5f32a37f7eb6d85239836599ecaa6f372b991d53814665: Status 404 returned error can't find the container with id 469e49ef398a3d696e5f32a37f7eb6d85239836599ecaa6f372b991d53814665 Apr 16 16:56:36.364573 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:36.364542 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-vqqz2" event={"ID":"14b0808e-16e3-4b10-993d-c7c6d9fd96d2","Type":"ContainerStarted","Data":"d5a4ded35a14ba383dbae6839407797632adef006c453814410dbb919f4afdad"} Apr 16 16:56:36.364573 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:36.364577 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-vqqz2" event={"ID":"14b0808e-16e3-4b10-993d-c7c6d9fd96d2","Type":"ContainerStarted","Data":"469e49ef398a3d696e5f32a37f7eb6d85239836599ecaa6f372b991d53814665"} Apr 16 16:56:36.378555 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:36.378496 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-vqqz2" podStartSLOduration=1.3784789800000001 podStartE2EDuration="1.37847898s" podCreationTimestamp="2026-04-16 16:56:35 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:56:36.377897134 +0000 UTC m=+535.439046666" watchObservedRunningTime="2026-04-16 16:56:36.37847898 +0000 UTC m=+535.439628515" Apr 16 16:56:41.475869 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:41.475841 2576 scope.go:117] "RemoveContainer" containerID="99811446d4e46a25015a2df57ff6a7babc8221367ed53d6b7496b1c70c53aec3" Apr 16 16:56:42.387100 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:42.387066 2576 generic.go:358] "Generic (PLEG): container finished" podID="14b0808e-16e3-4b10-993d-c7c6d9fd96d2" containerID="d5a4ded35a14ba383dbae6839407797632adef006c453814410dbb919f4afdad" exitCode=0 Apr 16 16:56:42.387296 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:42.387143 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-vqqz2" event={"ID":"14b0808e-16e3-4b10-993d-c7c6d9fd96d2","Type":"ContainerDied","Data":"d5a4ded35a14ba383dbae6839407797632adef006c453814410dbb919f4afdad"} Apr 16 16:56:43.512431 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:43.512409 2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-vqqz2" Apr 16 16:56:43.643829 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:43.643723 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5xjg\" (UniqueName: \"kubernetes.io/projected/14b0808e-16e3-4b10-993d-c7c6d9fd96d2-kube-api-access-p5xjg\") pod \"14b0808e-16e3-4b10-993d-c7c6d9fd96d2\" (UID: \"14b0808e-16e3-4b10-993d-c7c6d9fd96d2\") " Apr 16 16:56:43.645933 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:43.645907 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14b0808e-16e3-4b10-993d-c7c6d9fd96d2-kube-api-access-p5xjg" (OuterVolumeSpecName: "kube-api-access-p5xjg") pod "14b0808e-16e3-4b10-993d-c7c6d9fd96d2" (UID: "14b0808e-16e3-4b10-993d-c7c6d9fd96d2"). InnerVolumeSpecName "kube-api-access-p5xjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 16:56:43.745310 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:43.745234 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5xjg\" (UniqueName: \"kubernetes.io/projected/14b0808e-16e3-4b10-993d-c7c6d9fd96d2-kube-api-access-p5xjg\") on node \"ip-10-0-142-37.ec2.internal\" DevicePath \"\"" Apr 16 16:56:44.395037 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:44.395012 2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-vqqz2" Apr 16 16:56:44.395209 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:44.395011 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-vqqz2" event={"ID":"14b0808e-16e3-4b10-993d-c7c6d9fd96d2","Type":"ContainerDied","Data":"469e49ef398a3d696e5f32a37f7eb6d85239836599ecaa6f372b991d53814665"} Apr 16 16:56:44.395209 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:44.395117 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="469e49ef398a3d696e5f32a37f7eb6d85239836599ecaa6f372b991d53814665" Apr 16 16:56:46.963465 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:46.963437 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-nwhqq"] Apr 16 16:56:46.963986 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:46.963817 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14b0808e-16e3-4b10-993d-c7c6d9fd96d2" containerName="s3-tls-init-custom" Apr 16 16:56:46.963986 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:46.963831 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="14b0808e-16e3-4b10-993d-c7c6d9fd96d2" containerName="s3-tls-init-custom" Apr 16 16:56:46.963986 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:46.963898 2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="14b0808e-16e3-4b10-993d-c7c6d9fd96d2" containerName="s3-tls-init-custom" Apr 16 16:56:46.973618 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:46.973584 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-nwhqq"] Apr 16 16:56:46.973788 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:46.973677 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-nwhqq" Apr 16 16:56:46.975888 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:46.975858 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\"" Apr 16 16:56:46.976034 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:46.975901 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Apr 16 16:56:46.976034 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:46.975920 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Apr 16 16:56:46.976445 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:46.976411 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-9c52w\"" Apr 16 16:56:47.071533 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:47.071491 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lh4d\" (UniqueName: \"kubernetes.io/projected/42b09675-9fe9-480a-8470-a70374427d0d-kube-api-access-8lh4d\") pod \"s3-tls-init-serving-nwhqq\" (UID: \"42b09675-9fe9-480a-8470-a70374427d0d\") " pod="kserve/s3-tls-init-serving-nwhqq" Apr 16 16:56:47.172431 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:47.172395 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8lh4d\" (UniqueName: \"kubernetes.io/projected/42b09675-9fe9-480a-8470-a70374427d0d-kube-api-access-8lh4d\") pod \"s3-tls-init-serving-nwhqq\" (UID: \"42b09675-9fe9-480a-8470-a70374427d0d\") " pod="kserve/s3-tls-init-serving-nwhqq" Apr 16 16:56:47.181427 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:47.181406 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lh4d\" (UniqueName: \"kubernetes.io/projected/42b09675-9fe9-480a-8470-a70374427d0d-kube-api-access-8lh4d\") pod \"s3-tls-init-serving-nwhqq\" (UID: \"42b09675-9fe9-480a-8470-a70374427d0d\") " pod="kserve/s3-tls-init-serving-nwhqq" Apr 16 16:56:47.292146 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:47.292114 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-nwhqq" Apr 16 16:56:47.411202 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:47.411114 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-nwhqq"] Apr 16 16:56:47.413694 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:56:47.413655 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42b09675_9fe9_480a_8470_a70374427d0d.slice/crio-c3b550bb13218b66bfcd5bbec20dd34337645fb7c8bf9a65b78a2a5261d5d9f8 WatchSource:0}: Error finding container c3b550bb13218b66bfcd5bbec20dd34337645fb7c8bf9a65b78a2a5261d5d9f8: Status 404 returned error can't find the container with id c3b550bb13218b66bfcd5bbec20dd34337645fb7c8bf9a65b78a2a5261d5d9f8 Apr 16 16:56:48.407901 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:48.407866 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-nwhqq" event={"ID":"42b09675-9fe9-480a-8470-a70374427d0d","Type":"ContainerStarted","Data":"f59b488356e3f75cdd1eabe9581d06e985023e47436cf1db77af03f246964cc6"} Apr 16 16:56:48.407901 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:48.407901 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-nwhqq" event={"ID":"42b09675-9fe9-480a-8470-a70374427d0d","Type":"ContainerStarted","Data":"c3b550bb13218b66bfcd5bbec20dd34337645fb7c8bf9a65b78a2a5261d5d9f8"} Apr 16 16:56:48.423343 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:48.423298 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-nwhqq" podStartSLOduration=2.423285309 podStartE2EDuration="2.423285309s" podCreationTimestamp="2026-04-16 16:56:46 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:56:48.421973887 +0000 UTC m=+547.483123419" watchObservedRunningTime="2026-04-16 16:56:48.423285309 +0000 UTC m=+547.484434838" Apr 16 16:56:53.424154 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:53.424121 2576 generic.go:358] "Generic (PLEG): container finished" podID="42b09675-9fe9-480a-8470-a70374427d0d" containerID="f59b488356e3f75cdd1eabe9581d06e985023e47436cf1db77af03f246964cc6" exitCode=0 Apr 16 16:56:53.424552 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:53.424181 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-nwhqq" event={"ID":"42b09675-9fe9-480a-8470-a70374427d0d","Type":"ContainerDied","Data":"f59b488356e3f75cdd1eabe9581d06e985023e47436cf1db77af03f246964cc6"} Apr 16 16:56:54.550288 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:54.550258 2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-nwhqq" Apr 16 16:56:54.738234 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:54.738161 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lh4d\" (UniqueName: \"kubernetes.io/projected/42b09675-9fe9-480a-8470-a70374427d0d-kube-api-access-8lh4d\") pod \"42b09675-9fe9-480a-8470-a70374427d0d\" (UID: \"42b09675-9fe9-480a-8470-a70374427d0d\") " Apr 16 16:56:54.740395 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:54.740370 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42b09675-9fe9-480a-8470-a70374427d0d-kube-api-access-8lh4d" (OuterVolumeSpecName: "kube-api-access-8lh4d") pod "42b09675-9fe9-480a-8470-a70374427d0d" (UID: "42b09675-9fe9-480a-8470-a70374427d0d"). InnerVolumeSpecName "kube-api-access-8lh4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 16:56:54.839821 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:54.839790 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lh4d\" (UniqueName: \"kubernetes.io/projected/42b09675-9fe9-480a-8470-a70374427d0d-kube-api-access-8lh4d\") on node \"ip-10-0-142-37.ec2.internal\" DevicePath \"\"" Apr 16 16:56:55.433654 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:55.433616 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-nwhqq" event={"ID":"42b09675-9fe9-480a-8470-a70374427d0d","Type":"ContainerDied","Data":"c3b550bb13218b66bfcd5bbec20dd34337645fb7c8bf9a65b78a2a5261d5d9f8"} Apr 16 16:56:55.433654 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:55.433650 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3b550bb13218b66bfcd5bbec20dd34337645fb7c8bf9a65b78a2a5261d5d9f8" Apr 16 16:56:55.433654 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:56:55.433629 2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-nwhqq" Apr 16 16:57:04.693636 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:57:04.693596 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r"] Apr 16 16:57:04.694122 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:57:04.694103 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42b09675-9fe9-480a-8470-a70374427d0d" containerName="s3-tls-init-serving" Apr 16 16:57:04.694202 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:57:04.694126 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="42b09675-9fe9-480a-8470-a70374427d0d" containerName="s3-tls-init-serving" Apr 16 16:57:04.694295 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:57:04.694206 2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="42b09675-9fe9-480a-8470-a70374427d0d" containerName="s3-tls-init-serving" Apr 16 16:57:04.711631 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:57:04.711599 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r"] Apr 16 16:57:04.711769 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:57:04.711745 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r" Apr 16 16:57:04.713783 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:57:04.713756 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5ftrw\"" Apr 16 16:57:04.816639 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:57:04.816606 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e190ead5-ee2d-4785-9efa-b6f854dfee7f-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r\" (UID: \"e190ead5-ee2d-4785-9efa-b6f854dfee7f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r" Apr 16 16:57:04.918029 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:57:04.917990 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e190ead5-ee2d-4785-9efa-b6f854dfee7f-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r\" (UID: \"e190ead5-ee2d-4785-9efa-b6f854dfee7f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r" Apr 16 16:57:04.918397 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:57:04.918377 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e190ead5-ee2d-4785-9efa-b6f854dfee7f-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r\" (UID: \"e190ead5-ee2d-4785-9efa-b6f854dfee7f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r" Apr 16 16:57:05.022319 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:57:05.022225 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r" Apr 16 16:57:05.146619 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:57:05.146592 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r"] Apr 16 16:57:05.148921 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:57:05.148895 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode190ead5_ee2d_4785_9efa_b6f854dfee7f.slice/crio-f63962a5a72bd4634f8cdc9a37cb1ceee188860dd298569b787e1b2817f1e71c WatchSource:0}: Error finding container f63962a5a72bd4634f8cdc9a37cb1ceee188860dd298569b787e1b2817f1e71c: Status 404 returned error can't find the container with id f63962a5a72bd4634f8cdc9a37cb1ceee188860dd298569b787e1b2817f1e71c Apr 16 16:57:05.467294 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:57:05.467238 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r" event={"ID":"e190ead5-ee2d-4785-9efa-b6f854dfee7f","Type":"ContainerStarted","Data":"f63962a5a72bd4634f8cdc9a37cb1ceee188860dd298569b787e1b2817f1e71c"} Apr 16 16:57:09.485003 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:57:09.484966 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r" event={"ID":"e190ead5-ee2d-4785-9efa-b6f854dfee7f","Type":"ContainerStarted","Data":"76d84ad46148527e515f8329d1a0de4f5f48fbb0b8e844df78747364ba76364a"} Apr 16 16:57:12.497084 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:57:12.497000 2576 generic.go:358] "Generic (PLEG): container finished" podID="e190ead5-ee2d-4785-9efa-b6f854dfee7f" containerID="76d84ad46148527e515f8329d1a0de4f5f48fbb0b8e844df78747364ba76364a" exitCode=0 Apr 16 16:57:12.497084 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:57:12.497047 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r" event={"ID":"e190ead5-ee2d-4785-9efa-b6f854dfee7f","Type":"ContainerDied","Data":"76d84ad46148527e515f8329d1a0de4f5f48fbb0b8e844df78747364ba76364a"} Apr 16 16:57:25.554692 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:57:25.554659 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r" event={"ID":"e190ead5-ee2d-4785-9efa-b6f854dfee7f","Type":"ContainerStarted","Data":"92739c0ea5d241724f7d21ffc7bbaa60f417ffe905ff10836b1645d8bc246a12"} Apr 16 16:57:29.569785 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:57:29.569743 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r" event={"ID":"e190ead5-ee2d-4785-9efa-b6f854dfee7f","Type":"ContainerStarted","Data":"bfa3b0fa7568facd94e5d1f29384295034180eef1a28021adfe960914a0475ba"} Apr 16 16:57:29.570185 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:57:29.569945 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r" Apr 16 16:57:29.571212 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:57:29.571185 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r" podUID="e190ead5-ee2d-4785-9efa-b6f854dfee7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Apr 16 16:57:29.585888 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:57:29.585840 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r" podStartSLOduration=2.13366284 podStartE2EDuration="25.58582671s" podCreationTimestamp="2026-04-16 16:57:04 +0000 UTC" firstStartedPulling="2026-04-16 16:57:05.150929483 +0000 UTC m=+564.212078995" lastFinishedPulling="2026-04-16 16:57:28.603093357 +0000 UTC m=+587.664242865" observedRunningTime="2026-04-16 16:57:29.584638565 +0000 UTC m=+588.645788096" watchObservedRunningTime="2026-04-16 16:57:29.58582671 +0000 UTC m=+588.646976242" Apr 16 16:57:30.573568 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:57:30.573533 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r" Apr 16 16:57:30.574004 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:57:30.573698 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r" podUID="e190ead5-ee2d-4785-9efa-b6f854dfee7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Apr 16 16:57:30.574731 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:57:30.574704 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r" podUID="e190ead5-ee2d-4785-9efa-b6f854dfee7f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Apr 16 16:57:31.577619 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:57:31.577563 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r" podUID="e190ead5-ee2d-4785-9efa-b6f854dfee7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Apr 16 16:57:31.578012 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:57:31.577967 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r" podUID="e190ead5-ee2d-4785-9efa-b6f854dfee7f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Apr 16 16:57:41.578181 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:57:41.578134 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r" podUID="e190ead5-ee2d-4785-9efa-b6f854dfee7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Apr 16 16:57:41.578664 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:57:41.578638 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r" podUID="e190ead5-ee2d-4785-9efa-b6f854dfee7f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Apr 16 16:57:51.578295 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:57:51.578175 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r" podUID="e190ead5-ee2d-4785-9efa-b6f854dfee7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Apr 16 16:57:51.578765 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:57:51.578710 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r" podUID="e190ead5-ee2d-4785-9efa-b6f854dfee7f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Apr 16 16:58:01.577788 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:58:01.577731 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r" podUID="e190ead5-ee2d-4785-9efa-b6f854dfee7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Apr 16 16:58:01.578288 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:58:01.578239 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r" podUID="e190ead5-ee2d-4785-9efa-b6f854dfee7f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Apr 16 16:58:11.578391 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:58:11.578349 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r" podUID="e190ead5-ee2d-4785-9efa-b6f854dfee7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Apr 16 16:58:11.578819 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:58:11.578792 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r" podUID="e190ead5-ee2d-4785-9efa-b6f854dfee7f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Apr 16 16:58:21.577724 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:58:21.577672 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r" podUID="e190ead5-ee2d-4785-9efa-b6f854dfee7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Apr 16 16:58:21.580278 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:58:21.578091 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r" podUID="e190ead5-ee2d-4785-9efa-b6f854dfee7f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Apr 16 16:58:31.578508 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:58:31.578474 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r" Apr 16 16:58:31.578960 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:58:31.578554 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r" Apr 16 16:58:35.108966 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:58:35.108931 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84"] Apr 16 16:58:35.112693 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:58:35.112675 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84" Apr 16 16:58:35.120802 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:58:35.120775 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84"] Apr 16 16:58:35.220113 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:58:35.220061 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df201950-65db-4d8e-a187-b29ae219ea5d-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84\" (UID: \"df201950-65db-4d8e-a187-b29ae219ea5d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84" Apr 16 16:58:35.321410 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:58:35.321376 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df201950-65db-4d8e-a187-b29ae219ea5d-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84\" (UID: \"df201950-65db-4d8e-a187-b29ae219ea5d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84" Apr 16 16:58:35.321823 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:58:35.321800 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df201950-65db-4d8e-a187-b29ae219ea5d-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84\" (UID: \"df201950-65db-4d8e-a187-b29ae219ea5d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84" Apr 16 16:58:35.424263 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:58:35.424156 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84" Apr 16 16:58:35.544109 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:58:35.544084 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84"] Apr 16 16:58:35.545775 ip-10-0-142-37 kubenswrapper[2576]: W0416 16:58:35.545745 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf201950_65db_4d8e_a187_b29ae219ea5d.slice/crio-c7ee3f7e5c263b771caebb60a2d9c01dc82d13102b2142b1460319fdf4659efb WatchSource:0}: Error finding container c7ee3f7e5c263b771caebb60a2d9c01dc82d13102b2142b1460319fdf4659efb: Status 404 returned error can't find the container with id c7ee3f7e5c263b771caebb60a2d9c01dc82d13102b2142b1460319fdf4659efb Apr 16 16:58:35.802744 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:58:35.802703 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84" event={"ID":"df201950-65db-4d8e-a187-b29ae219ea5d","Type":"ContainerStarted","Data":"0a1c68f2717649ef13547d9b4ff1e8c027e3f641f000ae0de2516acd2b16cfc6"} Apr 16 16:58:35.802900 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:58:35.802747 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84" event={"ID":"df201950-65db-4d8e-a187-b29ae219ea5d","Type":"ContainerStarted","Data":"c7ee3f7e5c263b771caebb60a2d9c01dc82d13102b2142b1460319fdf4659efb"} Apr 16 16:58:39.818571 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:58:39.818537 2576 generic.go:358] "Generic (PLEG): container finished" podID="df201950-65db-4d8e-a187-b29ae219ea5d" containerID="0a1c68f2717649ef13547d9b4ff1e8c027e3f641f000ae0de2516acd2b16cfc6" exitCode=0 Apr 16 16:58:39.818977 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:58:39.818600 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84" event={"ID":"df201950-65db-4d8e-a187-b29ae219ea5d","Type":"ContainerDied","Data":"0a1c68f2717649ef13547d9b4ff1e8c027e3f641f000ae0de2516acd2b16cfc6"} Apr 16 16:58:40.824538 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:58:40.824506 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84" event={"ID":"df201950-65db-4d8e-a187-b29ae219ea5d","Type":"ContainerStarted","Data":"54d4ebc8e9a107c7b0fd0fe84938276a6578bf453c86edb1f1e7e41ffb9aff14"} Apr 16 16:58:40.824964 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:58:40.824546 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84" event={"ID":"df201950-65db-4d8e-a187-b29ae219ea5d","Type":"ContainerStarted","Data":"5677f04d8e7d6afc5ad9e6244e0cd01a35a90e4af87582c76265870a716c452e"} Apr 16 16:58:40.824964 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:58:40.824892 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84" Apr 16 16:58:40.826309 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:58:40.826283 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84" podUID="df201950-65db-4d8e-a187-b29ae219ea5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:5000: connect: connection refused" Apr 16 16:58:40.840806 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:58:40.840753 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84" podStartSLOduration=5.840739693 podStartE2EDuration="5.840739693s" podCreationTimestamp="2026-04-16 16:58:35 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:58:40.838910877 +0000 UTC m=+659.900060407" watchObservedRunningTime="2026-04-16 16:58:40.840739693 +0000 UTC m=+659.901889225" Apr 16 16:58:41.827834 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:58:41.827796 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84" Apr 16 16:58:41.828235 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:58:41.827961 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84" podUID="df201950-65db-4d8e-a187-b29ae219ea5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:5000: connect: connection refused" Apr 16 16:58:41.828899 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:58:41.828874 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84" podUID="df201950-65db-4d8e-a187-b29ae219ea5d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Apr 16 16:58:42.831506 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:58:42.831468 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84" podUID="df201950-65db-4d8e-a187-b29ae219ea5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:5000: connect: connection refused" Apr 16 16:58:42.831895 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:58:42.831765 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84" podUID="df201950-65db-4d8e-a187-b29ae219ea5d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Apr 16 16:58:52.831831 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:58:52.831785 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84" podUID="df201950-65db-4d8e-a187-b29ae219ea5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:5000: connect: connection refused" Apr 16 16:58:52.832322 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:58:52.832221 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84" podUID="df201950-65db-4d8e-a187-b29ae219ea5d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Apr 16 16:59:02.836871 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:59:02.836803 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84" podUID="df201950-65db-4d8e-a187-b29ae219ea5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:5000: connect: connection refused" Apr 16 16:59:02.837528 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:59:02.837496 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84" podUID="df201950-65db-4d8e-a187-b29ae219ea5d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Apr 16 16:59:12.831412 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:59:12.831359 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84" podUID="df201950-65db-4d8e-a187-b29ae219ea5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:5000: connect: connection refused" Apr 16 16:59:12.831823 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:59:12.831774 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84" podUID="df201950-65db-4d8e-a187-b29ae219ea5d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Apr 16 16:59:22.831532 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:59:22.831436 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84" podUID="df201950-65db-4d8e-a187-b29ae219ea5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:5000: connect: connection refused" Apr 16 16:59:22.831905 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:59:22.831824 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84" podUID="df201950-65db-4d8e-a187-b29ae219ea5d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Apr 16 16:59:32.831921 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:59:32.831875 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84" podUID="df201950-65db-4d8e-a187-b29ae219ea5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:5000: connect: connection refused" Apr 16 16:59:32.832478 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:59:32.832453 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84" podUID="df201950-65db-4d8e-a187-b29ae219ea5d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Apr 16 16:59:42.831824 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:59:42.831781 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84" podUID="df201950-65db-4d8e-a187-b29ae219ea5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:5000: connect: connection refused" Apr 16 16:59:42.832314 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:59:42.832290 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84" podUID="df201950-65db-4d8e-a187-b29ae219ea5d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Apr 16 16:59:52.589402 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:59:52.589371 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84" Apr 16 16:59:52.589883 ip-10-0-142-37 kubenswrapper[2576]: I0416 16:59:52.589526 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84" Apr 16 17:00:05.253049 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:00:05.253003 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-7fc456f8b8-qzxjk"] Apr 16 17:00:05.257079 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:00:05.257050 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fc456f8b8-qzxjk" Apr 16 17:00:05.264550 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:00:05.264518 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-7fc456f8b8-qzxjk"] Apr 16 17:00:05.395977 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:00:05.395943 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59298cc7-7cfd-49cd-b675-a5784a4851b6-kserve-provision-location\") pod \"isvc-logger-predictor-7fc456f8b8-qzxjk\" (UID: \"59298cc7-7cfd-49cd-b675-a5784a4851b6\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7fc456f8b8-qzxjk" Apr 16 17:00:05.496993 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:00:05.496949 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59298cc7-7cfd-49cd-b675-a5784a4851b6-kserve-provision-location\") pod \"isvc-logger-predictor-7fc456f8b8-qzxjk\" (UID: \"59298cc7-7cfd-49cd-b675-a5784a4851b6\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7fc456f8b8-qzxjk" Apr 16 17:00:05.497399 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:00:05.497380 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59298cc7-7cfd-49cd-b675-a5784a4851b6-kserve-provision-location\") pod \"isvc-logger-predictor-7fc456f8b8-qzxjk\" (UID: \"59298cc7-7cfd-49cd-b675-a5784a4851b6\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7fc456f8b8-qzxjk" Apr 16 17:00:05.569169 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:00:05.569133 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fc456f8b8-qzxjk" Apr 16 17:00:05.696230 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:00:05.696206 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-7fc456f8b8-qzxjk"] Apr 16 17:00:05.698120 ip-10-0-142-37 kubenswrapper[2576]: W0416 17:00:05.698080 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59298cc7_7cfd_49cd_b675_a5784a4851b6.slice/crio-debc0b3f15f220285deb3d7dfb653e526aa46ce17486f00caaaab5acad710776 WatchSource:0}: Error finding container debc0b3f15f220285deb3d7dfb653e526aa46ce17486f00caaaab5acad710776: Status 404 returned error can't find the container with id debc0b3f15f220285deb3d7dfb653e526aa46ce17486f00caaaab5acad710776 Apr 16 17:00:06.121638 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:00:06.121606 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fc456f8b8-qzxjk" event={"ID":"59298cc7-7cfd-49cd-b675-a5784a4851b6","Type":"ContainerStarted","Data":"5874327c13d74c8125f6fb42f00ed3544221a60a7e4eaa02a449d038030c7a21"} Apr 16 17:00:06.121811 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:00:06.121646 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fc456f8b8-qzxjk" event={"ID":"59298cc7-7cfd-49cd-b675-a5784a4851b6","Type":"ContainerStarted","Data":"debc0b3f15f220285deb3d7dfb653e526aa46ce17486f00caaaab5acad710776"} Apr 16 17:00:10.136728 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:00:10.136697 2576 generic.go:358] "Generic (PLEG): container finished" podID="59298cc7-7cfd-49cd-b675-a5784a4851b6" containerID="5874327c13d74c8125f6fb42f00ed3544221a60a7e4eaa02a449d038030c7a21" exitCode=0 Apr 16 17:00:10.137482 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:00:10.136772 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fc456f8b8-qzxjk" event={"ID":"59298cc7-7cfd-49cd-b675-a5784a4851b6","Type":"ContainerDied","Data":"5874327c13d74c8125f6fb42f00ed3544221a60a7e4eaa02a449d038030c7a21"} Apr 16 17:00:11.142134 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:00:11.142096 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fc456f8b8-qzxjk" event={"ID":"59298cc7-7cfd-49cd-b675-a5784a4851b6","Type":"ContainerStarted","Data":"8a2574b4a238daea1585dd7f0b075982a4290a4e0722cb8f83784836cc70be2c"} Apr 16 17:00:11.142134 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:00:11.142138 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fc456f8b8-qzxjk" event={"ID":"59298cc7-7cfd-49cd-b675-a5784a4851b6","Type":"ContainerStarted","Data":"1164558764d22d33395e054873e7ffbc12c7a58dfe49c3275a4dfc1fff60e98f"} Apr 16 17:00:11.142609 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:00:11.142506 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fc456f8b8-qzxjk" Apr 16 17:00:11.142609 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:00:11.142540 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fc456f8b8-qzxjk" Apr 16 17:00:11.143906 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:00:11.143873 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fc456f8b8-qzxjk" podUID="59298cc7-7cfd-49cd-b675-a5784a4851b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused" Apr 16 17:00:11.144620 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:00:11.144593 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fc456f8b8-qzxjk" podUID="59298cc7-7cfd-49cd-b675-a5784a4851b6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Apr 16 17:00:11.157977 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:00:11.157931 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fc456f8b8-qzxjk" podStartSLOduration=6.157918657 podStartE2EDuration="6.157918657s" podCreationTimestamp="2026-04-16 17:00:05 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 17:00:11.155894634 +0000 UTC m=+750.217044164" watchObservedRunningTime="2026-04-16 17:00:11.157918657 +0000 UTC m=+750.219068188" Apr 16 17:00:12.146739 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:00:12.146699 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fc456f8b8-qzxjk" podUID="59298cc7-7cfd-49cd-b675-a5784a4851b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused" Apr 16 17:00:12.147172 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:00:12.147051 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fc456f8b8-qzxjk" podUID="59298cc7-7cfd-49cd-b675-a5784a4851b6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Apr 16 17:00:22.146735 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:00:22.146677 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fc456f8b8-qzxjk" podUID="59298cc7-7cfd-49cd-b675-a5784a4851b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused" Apr 16 17:00:22.147211 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:00:22.147162 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fc456f8b8-qzxjk" podUID="59298cc7-7cfd-49cd-b675-a5784a4851b6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Apr 16 17:00:32.147448 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:00:32.147400 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fc456f8b8-qzxjk" podUID="59298cc7-7cfd-49cd-b675-a5784a4851b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused" Apr 16 17:00:32.147882 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:00:32.147788 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fc456f8b8-qzxjk" podUID="59298cc7-7cfd-49cd-b675-a5784a4851b6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Apr 16 17:00:42.146739 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:00:42.146695 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fc456f8b8-qzxjk" podUID="59298cc7-7cfd-49cd-b675-a5784a4851b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused" Apr 16 17:00:42.147227 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:00:42.147162 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fc456f8b8-qzxjk" podUID="59298cc7-7cfd-49cd-b675-a5784a4851b6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Apr 16 17:00:52.147317 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:00:52.147189 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fc456f8b8-qzxjk" podUID="59298cc7-7cfd-49cd-b675-a5784a4851b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused" Apr 16 17:00:52.147727 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:00:52.147705 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fc456f8b8-qzxjk" podUID="59298cc7-7cfd-49cd-b675-a5784a4851b6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Apr 16 17:01:02.147417 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:01:02.147366 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fc456f8b8-qzxjk" podUID="59298cc7-7cfd-49cd-b675-a5784a4851b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused" Apr 16 17:01:02.147895 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:01:02.147815 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fc456f8b8-qzxjk" podUID="59298cc7-7cfd-49cd-b675-a5784a4851b6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Apr 16 17:01:12.146972 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:01:12.146922 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fc456f8b8-qzxjk" podUID="59298cc7-7cfd-49cd-b675-a5784a4851b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused" Apr 16 17:01:12.147466 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:01:12.147344 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fc456f8b8-qzxjk" podUID="59298cc7-7cfd-49cd-b675-a5784a4851b6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Apr 16 17:01:22.147438 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:01:22.147402 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fc456f8b8-qzxjk" Apr 16 17:01:22.147918 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:01:22.147846 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7fc456f8b8-qzxjk" Apr 16 17:08:26.570496 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:08:26.570410 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b8d64fb98-vw8rk"] Apr 16 17:08:26.574150 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:08:26.574130 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b8d64fb98-vw8rk" Apr 16 17:08:26.584375 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:08:26.584344 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b8d64fb98-vw8rk"] Apr 16 17:08:26.657184 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:08:26.657149 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c6ad62b-3888-4cc4-8471-79ebb62b2f1b-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-6b8d64fb98-vw8rk\" (UID: \"6c6ad62b-3888-4cc4-8471-79ebb62b2f1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b8d64fb98-vw8rk" Apr 16 17:08:26.758637 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:08:26.758593 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c6ad62b-3888-4cc4-8471-79ebb62b2f1b-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-6b8d64fb98-vw8rk\" (UID: \"6c6ad62b-3888-4cc4-8471-79ebb62b2f1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b8d64fb98-vw8rk" Apr 16 17:08:26.758990 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:08:26.758969 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c6ad62b-3888-4cc4-8471-79ebb62b2f1b-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-6b8d64fb98-vw8rk\" (UID: \"6c6ad62b-3888-4cc4-8471-79ebb62b2f1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b8d64fb98-vw8rk" Apr 16 17:08:26.885397 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:08:26.885291 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b8d64fb98-vw8rk" Apr 16 17:08:27.007728 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:08:27.007702 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b8d64fb98-vw8rk"] Apr 16 17:08:27.009899 ip-10-0-142-37 kubenswrapper[2576]: W0416 17:08:27.009873 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c6ad62b_3888_4cc4_8471_79ebb62b2f1b.slice/crio-8bf3814f67c0a3949224c93c9bfd7dc82c6c17f3cad264e51c794b3c77ab3671 WatchSource:0}: Error finding container 8bf3814f67c0a3949224c93c9bfd7dc82c6c17f3cad264e51c794b3c77ab3671: Status 404 returned error can't find the container with id 8bf3814f67c0a3949224c93c9bfd7dc82c6c17f3cad264e51c794b3c77ab3671 Apr 16 17:08:27.011670 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:08:27.011651 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Apr 16 17:08:27.879816 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:08:27.879783 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b8d64fb98-vw8rk" event={"ID":"6c6ad62b-3888-4cc4-8471-79ebb62b2f1b","Type":"ContainerStarted","Data":"faa0a2dbea5932c82fcb8b5d4306811f455b4f78988e00b6d0adc3f50c551875"} Apr 16 17:08:27.879816 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:08:27.879822 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b8d64fb98-vw8rk" event={"ID":"6c6ad62b-3888-4cc4-8471-79ebb62b2f1b","Type":"ContainerStarted","Data":"8bf3814f67c0a3949224c93c9bfd7dc82c6c17f3cad264e51c794b3c77ab3671"} Apr 16 17:08:30.892979 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:08:30.892884 2576 generic.go:358] "Generic (PLEG): container finished" podID="6c6ad62b-3888-4cc4-8471-79ebb62b2f1b" containerID="faa0a2dbea5932c82fcb8b5d4306811f455b4f78988e00b6d0adc3f50c551875" exitCode=0 Apr 16 17:08:30.892979 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:08:30.892945 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b8d64fb98-vw8rk" event={"ID":"6c6ad62b-3888-4cc4-8471-79ebb62b2f1b","Type":"ContainerDied","Data":"faa0a2dbea5932c82fcb8b5d4306811f455b4f78988e00b6d0adc3f50c551875"} Apr 16 17:08:31.898504 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:08:31.898469 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b8d64fb98-vw8rk" event={"ID":"6c6ad62b-3888-4cc4-8471-79ebb62b2f1b","Type":"ContainerStarted","Data":"7cdb18a021e3d731c84fe9db018cd40c099e4acca669f177c4d23259a72fda1f"} Apr 16 17:08:34.911304 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:08:34.911267 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b8d64fb98-vw8rk" event={"ID":"6c6ad62b-3888-4cc4-8471-79ebb62b2f1b","Type":"ContainerStarted","Data":"7ff7a9ac126ddb4b139671c6a3f41454f0f8992d886150137c556663375f0cc2"} Apr 16 17:08:34.911673 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:08:34.911395 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b8d64fb98-vw8rk" Apr 16 17:08:34.911673 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:08:34.911476 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b8d64fb98-vw8rk" Apr 16 17:08:34.930163 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:08:34.930101 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b8d64fb98-vw8rk" podStartSLOduration=5.792910451 podStartE2EDuration="8.930087432s" podCreationTimestamp="2026-04-16 17:08:26 +0000 UTC" firstStartedPulling="2026-04-16 17:08:30.961117644 +0000 UTC m=+1250.022267160" lastFinishedPulling="2026-04-16 17:08:34.098294629 +0000 UTC m=+1253.159444141" observedRunningTime="2026-04-16 17:08:34.928115088 +0000 UTC m=+1253.989264631" watchObservedRunningTime="2026-04-16 17:08:34.930087432 +0000 UTC m=+1253.991236964" Apr 16 17:09:05.918678 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:09:05.918632 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b8d64fb98-vw8rk" Apr 16 17:09:35.920265 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:09:35.920208 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b8d64fb98-vw8rk" Apr 16 17:09:36.671910 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:09:36.671876 2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b8d64fb98-vw8rk"] Apr 16 17:09:36.672352 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:09:36.672292 2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b8d64fb98-vw8rk" podUID="6c6ad62b-3888-4cc4-8471-79ebb62b2f1b" containerName="kserve-container" containerID="cri-o://7cdb18a021e3d731c84fe9db018cd40c099e4acca669f177c4d23259a72fda1f" gracePeriod=30 Apr 16 17:09:36.672546 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:09:36.672491 2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b8d64fb98-vw8rk" podUID="6c6ad62b-3888-4cc4-8471-79ebb62b2f1b" containerName="kserve-agent" containerID="cri-o://7ff7a9ac126ddb4b139671c6a3f41454f0f8992d886150137c556663375f0cc2" gracePeriod=30 Apr 16 17:09:36.717204 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:09:36.717175 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-z562j"] Apr 16 17:09:36.720944 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:09:36.720926 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-z562j" Apr 16 17:09:36.727834 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:09:36.727807 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-z562j"] Apr 16 17:09:36.854078 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:09:36.854046 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2183d52a-e000-4a8f-9d4b-3d289a306e23-kserve-provision-location\") pod \"isvc-paddle-predictor-7dddcb4bd4-z562j\" (UID: \"2183d52a-e000-4a8f-9d4b-3d289a306e23\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-z562j" Apr 16 17:09:36.954683 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:09:36.954587 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2183d52a-e000-4a8f-9d4b-3d289a306e23-kserve-provision-location\") pod \"isvc-paddle-predictor-7dddcb4bd4-z562j\" (UID: \"2183d52a-e000-4a8f-9d4b-3d289a306e23\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-z562j" Apr 16 17:09:36.955033 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:09:36.954953 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2183d52a-e000-4a8f-9d4b-3d289a306e23-kserve-provision-location\") pod \"isvc-paddle-predictor-7dddcb4bd4-z562j\" (UID: \"2183d52a-e000-4a8f-9d4b-3d289a306e23\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-z562j" Apr 16 17:09:37.032470 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:09:37.032426 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-z562j" Apr 16 17:09:37.157837 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:09:37.157813 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-z562j"] Apr 16 17:09:37.160696 ip-10-0-142-37 kubenswrapper[2576]: W0416 17:09:37.160667 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2183d52a_e000_4a8f_9d4b_3d289a306e23.slice/crio-2db627b66a6db572f8dd5b9c94c76dbf54340bf60b2d1a4e9fcee2b8b529ec75 WatchSource:0}: Error finding container 2db627b66a6db572f8dd5b9c94c76dbf54340bf60b2d1a4e9fcee2b8b529ec75: Status 404 returned error can't find the container with id 2db627b66a6db572f8dd5b9c94c76dbf54340bf60b2d1a4e9fcee2b8b529ec75 Apr 16 17:09:38.142051 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:09:38.142011 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-z562j" event={"ID":"2183d52a-e000-4a8f-9d4b-3d289a306e23","Type":"ContainerStarted","Data":"96893ea58750f7ab2df85c68a61d6394d233115942c96f5853170860868a9123"} Apr 16 17:09:38.142445 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:09:38.142059 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-z562j" event={"ID":"2183d52a-e000-4a8f-9d4b-3d289a306e23","Type":"ContainerStarted","Data":"2db627b66a6db572f8dd5b9c94c76dbf54340bf60b2d1a4e9fcee2b8b529ec75"} Apr 16 17:09:39.147097 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:09:39.147064 2576 generic.go:358] "Generic (PLEG): container finished" podID="6c6ad62b-3888-4cc4-8471-79ebb62b2f1b" containerID="7cdb18a021e3d731c84fe9db018cd40c099e4acca669f177c4d23259a72fda1f" exitCode=0 Apr 16 17:09:39.147486 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:09:39.147139 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b8d64fb98-vw8rk" event={"ID":"6c6ad62b-3888-4cc4-8471-79ebb62b2f1b","Type":"ContainerDied","Data":"7cdb18a021e3d731c84fe9db018cd40c099e4acca669f177c4d23259a72fda1f"} Apr 16 17:09:42.160147 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:09:42.160067 2576 generic.go:358] "Generic (PLEG): container finished" podID="2183d52a-e000-4a8f-9d4b-3d289a306e23" containerID="96893ea58750f7ab2df85c68a61d6394d233115942c96f5853170860868a9123" exitCode=0 Apr 16 17:09:42.160147 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:09:42.160117 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-z562j" event={"ID":"2183d52a-e000-4a8f-9d4b-3d289a306e23","Type":"ContainerDied","Data":"96893ea58750f7ab2df85c68a61d6394d233115942c96f5853170860868a9123"} Apr 16 17:09:45.916023 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:09:45.915922 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b8d64fb98-vw8rk" podUID="6c6ad62b-3888-4cc4-8471-79ebb62b2f1b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.30:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.30:8080: connect: connection refused" Apr 16 17:09:55.211591 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:09:55.211556 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-z562j" event={"ID":"2183d52a-e000-4a8f-9d4b-3d289a306e23","Type":"ContainerStarted","Data":"683eee4604e9a7fbc25576ddac547764e1a386af70eb987df6d83a7005102a43"} Apr 16 17:09:55.211996 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:09:55.211836 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-z562j" Apr 16 17:09:55.212963 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:09:55.212932 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-z562j" podUID="2183d52a-e000-4a8f-9d4b-3d289a306e23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused" Apr 16 17:09:55.230234 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:09:55.230173 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-z562j" podStartSLOduration=6.367555477 podStartE2EDuration="19.230153523s" podCreationTimestamp="2026-04-16 17:09:36 +0000 UTC" firstStartedPulling="2026-04-16 17:09:42.161362207 +0000 UTC m=+1321.222511716" lastFinishedPulling="2026-04-16 17:09:55.023960249 +0000 UTC m=+1334.085109762" observedRunningTime="2026-04-16 17:09:55.226363544 +0000 UTC m=+1334.287513113" watchObservedRunningTime="2026-04-16 17:09:55.230153523 +0000 UTC m=+1334.291303063" Apr 16 17:09:55.916355 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:09:55.916316 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b8d64fb98-vw8rk" podUID="6c6ad62b-3888-4cc4-8471-79ebb62b2f1b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.30:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.30:8080: connect: connection refused" Apr 16 17:09:56.215900 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:09:56.215810 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-z562j" podUID="2183d52a-e000-4a8f-9d4b-3d289a306e23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused" Apr 16 17:10:05.915978 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:05.915940 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b8d64fb98-vw8rk" podUID="6c6ad62b-3888-4cc4-8471-79ebb62b2f1b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.30:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.30:8080: connect: connection refused" Apr 16 17:10:05.916451 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:05.916057 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b8d64fb98-vw8rk" Apr 16 17:10:06.216413 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:06.216324 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-z562j" podUID="2183d52a-e000-4a8f-9d4b-3d289a306e23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused" Apr 16 17:10:06.818091 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:06.818066 2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b8d64fb98-vw8rk" Apr 16 17:10:06.919489 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:06.919447 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c6ad62b-3888-4cc4-8471-79ebb62b2f1b-kserve-provision-location\") pod \"6c6ad62b-3888-4cc4-8471-79ebb62b2f1b\" (UID: \"6c6ad62b-3888-4cc4-8471-79ebb62b2f1b\") " Apr 16 17:10:06.919864 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:06.919776 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c6ad62b-3888-4cc4-8471-79ebb62b2f1b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6c6ad62b-3888-4cc4-8471-79ebb62b2f1b" (UID: "6c6ad62b-3888-4cc4-8471-79ebb62b2f1b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 16 17:10:07.020380 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:07.020289 2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c6ad62b-3888-4cc4-8471-79ebb62b2f1b-kserve-provision-location\") on node \"ip-10-0-142-37.ec2.internal\" DevicePath \"\"" Apr 16 17:10:07.258240 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:07.258209 2576 generic.go:358] "Generic (PLEG): container finished" podID="6c6ad62b-3888-4cc4-8471-79ebb62b2f1b" containerID="7ff7a9ac126ddb4b139671c6a3f41454f0f8992d886150137c556663375f0cc2" exitCode=0 Apr 16 17:10:07.258429 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:07.258314 2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b8d64fb98-vw8rk" Apr 16 17:10:07.258429 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:07.258335 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b8d64fb98-vw8rk" event={"ID":"6c6ad62b-3888-4cc4-8471-79ebb62b2f1b","Type":"ContainerDied","Data":"7ff7a9ac126ddb4b139671c6a3f41454f0f8992d886150137c556663375f0cc2"} Apr 16 17:10:07.258429 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:07.258377 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b8d64fb98-vw8rk" event={"ID":"6c6ad62b-3888-4cc4-8471-79ebb62b2f1b","Type":"ContainerDied","Data":"8bf3814f67c0a3949224c93c9bfd7dc82c6c17f3cad264e51c794b3c77ab3671"} Apr 16 17:10:07.258429 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:07.258393 2576 scope.go:117] "RemoveContainer" containerID="7ff7a9ac126ddb4b139671c6a3f41454f0f8992d886150137c556663375f0cc2" Apr 16 17:10:07.268016 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:07.267998 2576 scope.go:117] "RemoveContainer" containerID="7cdb18a021e3d731c84fe9db018cd40c099e4acca669f177c4d23259a72fda1f" Apr 16 17:10:07.277832 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:07.277387 2576 scope.go:117] "RemoveContainer" containerID="faa0a2dbea5932c82fcb8b5d4306811f455b4f78988e00b6d0adc3f50c551875" Apr 16 17:10:07.283464 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:07.281655 2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b8d64fb98-vw8rk"] Apr 16 17:10:07.285191 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:07.285167 2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b8d64fb98-vw8rk"] Apr 16 17:10:07.290501 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:07.290480 2576 scope.go:117] "RemoveContainer" containerID="7ff7a9ac126ddb4b139671c6a3f41454f0f8992d886150137c556663375f0cc2" Apr 16 17:10:07.290812 ip-10-0-142-37 kubenswrapper[2576]: E0416 17:10:07.290791 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ff7a9ac126ddb4b139671c6a3f41454f0f8992d886150137c556663375f0cc2\": container with ID starting with 7ff7a9ac126ddb4b139671c6a3f41454f0f8992d886150137c556663375f0cc2 not found: ID does not exist" containerID="7ff7a9ac126ddb4b139671c6a3f41454f0f8992d886150137c556663375f0cc2" Apr 16 17:10:07.290886 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:07.290820 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ff7a9ac126ddb4b139671c6a3f41454f0f8992d886150137c556663375f0cc2"} err="failed to get container status \"7ff7a9ac126ddb4b139671c6a3f41454f0f8992d886150137c556663375f0cc2\": rpc error: code = NotFound desc = could not find container \"7ff7a9ac126ddb4b139671c6a3f41454f0f8992d886150137c556663375f0cc2\": container with ID starting with 7ff7a9ac126ddb4b139671c6a3f41454f0f8992d886150137c556663375f0cc2 not found: ID does not exist" Apr 16 17:10:07.290886 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:07.290860 2576 scope.go:117] "RemoveContainer" containerID="7cdb18a021e3d731c84fe9db018cd40c099e4acca669f177c4d23259a72fda1f" Apr 16 17:10:07.291113 ip-10-0-142-37 kubenswrapper[2576]: E0416 17:10:07.291093 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cdb18a021e3d731c84fe9db018cd40c099e4acca669f177c4d23259a72fda1f\": container with ID starting with 7cdb18a021e3d731c84fe9db018cd40c099e4acca669f177c4d23259a72fda1f not found: ID does not exist" containerID="7cdb18a021e3d731c84fe9db018cd40c099e4acca669f177c4d23259a72fda1f" Apr 16 17:10:07.291156 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:07.291119 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cdb18a021e3d731c84fe9db018cd40c099e4acca669f177c4d23259a72fda1f"} err="failed to get container status \"7cdb18a021e3d731c84fe9db018cd40c099e4acca669f177c4d23259a72fda1f\": rpc error: code = NotFound desc = could not find container \"7cdb18a021e3d731c84fe9db018cd40c099e4acca669f177c4d23259a72fda1f\": container with ID starting with 7cdb18a021e3d731c84fe9db018cd40c099e4acca669f177c4d23259a72fda1f not found: ID does not exist" Apr 16 17:10:07.291156 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:07.291135 2576 scope.go:117] "RemoveContainer" containerID="faa0a2dbea5932c82fcb8b5d4306811f455b4f78988e00b6d0adc3f50c551875" Apr 16 17:10:07.291477 ip-10-0-142-37 kubenswrapper[2576]: E0416 17:10:07.291460 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faa0a2dbea5932c82fcb8b5d4306811f455b4f78988e00b6d0adc3f50c551875\": container with ID starting with faa0a2dbea5932c82fcb8b5d4306811f455b4f78988e00b6d0adc3f50c551875 not found: ID does not exist" containerID="faa0a2dbea5932c82fcb8b5d4306811f455b4f78988e00b6d0adc3f50c551875" Apr 16 17:10:07.291530 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:07.291482 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faa0a2dbea5932c82fcb8b5d4306811f455b4f78988e00b6d0adc3f50c551875"} err="failed to get container status \"faa0a2dbea5932c82fcb8b5d4306811f455b4f78988e00b6d0adc3f50c551875\": rpc error: code = NotFound desc = could not find container \"faa0a2dbea5932c82fcb8b5d4306811f455b4f78988e00b6d0adc3f50c551875\": container with ID starting with faa0a2dbea5932c82fcb8b5d4306811f455b4f78988e00b6d0adc3f50c551875 not found: ID does not exist" Apr 16 17:10:07.593774 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:07.593740 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c6ad62b-3888-4cc4-8471-79ebb62b2f1b" path="/var/lib/kubelet/pods/6c6ad62b-3888-4cc4-8471-79ebb62b2f1b/volumes" Apr 16 17:10:16.216299 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:16.216227 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-z562j" podUID="2183d52a-e000-4a8f-9d4b-3d289a306e23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused" Apr 16 17:10:26.216779 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:26.216732 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-z562j" podUID="2183d52a-e000-4a8f-9d4b-3d289a306e23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused" Apr 16 17:10:36.215786 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:36.215741 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-z562j" podUID="2183d52a-e000-4a8f-9d4b-3d289a306e23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused" Apr 16 17:10:46.217368 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:46.217282 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-z562j" Apr 16 17:10:47.410743 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:47.410712 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-65rhw"] Apr 16 17:10:47.411125 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:47.411111 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c6ad62b-3888-4cc4-8471-79ebb62b2f1b" containerName="kserve-agent" Apr 16 17:10:47.411167 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:47.411128 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c6ad62b-3888-4cc4-8471-79ebb62b2f1b" containerName="kserve-agent" Apr 16 17:10:47.411167 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:47.411156 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c6ad62b-3888-4cc4-8471-79ebb62b2f1b" containerName="storage-initializer" Apr 16 17:10:47.411167 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:47.411162 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c6ad62b-3888-4cc4-8471-79ebb62b2f1b" containerName="storage-initializer" Apr 16 17:10:47.411167 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:47.411167 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c6ad62b-3888-4cc4-8471-79ebb62b2f1b" containerName="kserve-container" Apr 16 17:10:47.411315 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:47.411173 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c6ad62b-3888-4cc4-8471-79ebb62b2f1b" containerName="kserve-container" Apr 16 17:10:47.411315 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:47.411230 2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="6c6ad62b-3888-4cc4-8471-79ebb62b2f1b" containerName="kserve-agent" Apr 16 17:10:47.411315 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:47.411265 2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="6c6ad62b-3888-4cc4-8471-79ebb62b2f1b" containerName="kserve-container" Apr 16 17:10:47.426026 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:47.425996 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-65rhw"] Apr 16 17:10:47.426173 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:47.426122 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-65rhw" Apr 16 17:10:47.484799 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:47.484763 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41e6c8cc-bfa4-441f-9a7d-189270eafb7a-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-86b49c4466-65rhw\" (UID: \"41e6c8cc-bfa4-441f-9a7d-189270eafb7a\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-65rhw" Apr 16 17:10:47.586149 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:47.586112 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41e6c8cc-bfa4-441f-9a7d-189270eafb7a-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-86b49c4466-65rhw\" (UID: \"41e6c8cc-bfa4-441f-9a7d-189270eafb7a\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-65rhw" Apr 16 17:10:47.586517 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:47.586496 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41e6c8cc-bfa4-441f-9a7d-189270eafb7a-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-86b49c4466-65rhw\" (UID: \"41e6c8cc-bfa4-441f-9a7d-189270eafb7a\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-65rhw" Apr 16 17:10:47.736087 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:47.735977 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-65rhw" Apr 16 17:10:47.860768 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:47.860740 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-65rhw"] Apr 16 17:10:47.863023 ip-10-0-142-37 kubenswrapper[2576]: W0416 17:10:47.862989 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41e6c8cc_bfa4_441f_9a7d_189270eafb7a.slice/crio-8afe5bf2f11d49459e4819690bfbb8583c220b30cc30aabf23ea931bcfaac487 WatchSource:0}: Error finding container 8afe5bf2f11d49459e4819690bfbb8583c220b30cc30aabf23ea931bcfaac487: Status 404 returned error can't find the container with id 8afe5bf2f11d49459e4819690bfbb8583c220b30cc30aabf23ea931bcfaac487 Apr 16 17:10:48.415020 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:48.414978 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-65rhw" event={"ID":"41e6c8cc-bfa4-441f-9a7d-189270eafb7a","Type":"ContainerStarted","Data":"c1e1be8edfdbb9b012a3218d78e058ced52ecd3474f8322f640be6cc6c73ccd0"} Apr 16 17:10:48.415020 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:48.415019 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-65rhw" event={"ID":"41e6c8cc-bfa4-441f-9a7d-189270eafb7a","Type":"ContainerStarted","Data":"8afe5bf2f11d49459e4819690bfbb8583c220b30cc30aabf23ea931bcfaac487"} Apr 16 17:10:52.430991 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:52.430954 2576 generic.go:358] "Generic (PLEG): container finished" podID="41e6c8cc-bfa4-441f-9a7d-189270eafb7a" containerID="c1e1be8edfdbb9b012a3218d78e058ced52ecd3474f8322f640be6cc6c73ccd0" exitCode=0 Apr 16 17:10:52.431383 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:52.431028 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-65rhw" event={"ID":"41e6c8cc-bfa4-441f-9a7d-189270eafb7a","Type":"ContainerDied","Data":"c1e1be8edfdbb9b012a3218d78e058ced52ecd3474f8322f640be6cc6c73ccd0"} Apr 16 17:10:53.437121 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:53.437085 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-65rhw" event={"ID":"41e6c8cc-bfa4-441f-9a7d-189270eafb7a","Type":"ContainerStarted","Data":"b04abe9fa427013cf45634ccb05a1f92fed0826eb7613ffa6f4f76406aeef0bf"} Apr 16 17:10:53.437618 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:53.437477 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-65rhw" Apr 16 17:10:53.438820 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:53.438792 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-65rhw" podUID="41e6c8cc-bfa4-441f-9a7d-189270eafb7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Apr 16 17:10:53.452387 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:53.452348 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-65rhw" podStartSLOduration=6.452336756 podStartE2EDuration="6.452336756s" podCreationTimestamp="2026-04-16 17:10:47 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 17:10:53.451845256 +0000 UTC m=+1392.512994786" watchObservedRunningTime="2026-04-16 17:10:53.452336756 +0000 UTC m=+1392.513486288" Apr 16 17:10:54.440784 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:10:54.440746 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-65rhw" podUID="41e6c8cc-bfa4-441f-9a7d-189270eafb7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Apr 16 17:11:04.441447 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:11:04.441396 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-65rhw" podUID="41e6c8cc-bfa4-441f-9a7d-189270eafb7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Apr 16 17:11:14.441130 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:11:14.441077 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-65rhw" podUID="41e6c8cc-bfa4-441f-9a7d-189270eafb7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Apr 16 17:11:24.440803 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:11:24.440705 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-65rhw" podUID="41e6c8cc-bfa4-441f-9a7d-189270eafb7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Apr 16 17:11:34.442327 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:11:34.442295 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-65rhw" Apr 16 17:11:37.994386 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:11:37.994349 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452"] Apr 16 17:11:37.998049 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:11:37.998029 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452" Apr 16 17:11:38.004018 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:11:38.003990 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452"] Apr 16 17:11:38.112588 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:11:38.112555 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa6e2817-5d34-4c5e-9436-ab23a1de66a7-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452\" (UID: \"fa6e2817-5d34-4c5e-9436-ab23a1de66a7\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452" Apr 16 17:11:38.213814 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:11:38.213771 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa6e2817-5d34-4c5e-9436-ab23a1de66a7-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452\" (UID: \"fa6e2817-5d34-4c5e-9436-ab23a1de66a7\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452" Apr 16 17:11:38.214175 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:11:38.214157 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa6e2817-5d34-4c5e-9436-ab23a1de66a7-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452\" (UID: \"fa6e2817-5d34-4c5e-9436-ab23a1de66a7\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452" Apr 16 17:11:38.309868 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:11:38.309836 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452" Apr 16 17:11:38.434429 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:11:38.434396 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452"] Apr 16 17:11:38.437878 ip-10-0-142-37 kubenswrapper[2576]: W0416 17:11:38.437848 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa6e2817_5d34_4c5e_9436_ab23a1de66a7.slice/crio-1fdb5ae2276b9a82583edd433491bb2485e66131431c189f51f609d3d8e1fa62 WatchSource:0}: Error finding container 1fdb5ae2276b9a82583edd433491bb2485e66131431c189f51f609d3d8e1fa62: Status 404 returned error can't find the container with id 1fdb5ae2276b9a82583edd433491bb2485e66131431c189f51f609d3d8e1fa62 Apr 16 17:11:38.605033 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:11:38.604947 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452" event={"ID":"fa6e2817-5d34-4c5e-9436-ab23a1de66a7","Type":"ContainerStarted","Data":"2aa2a9833ffa3ffbf912a0f8263eb9807579094cc6e6c46425b06fec35373b7a"} Apr 16 17:11:38.605033 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:11:38.604982 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452" event={"ID":"fa6e2817-5d34-4c5e-9436-ab23a1de66a7","Type":"ContainerStarted","Data":"1fdb5ae2276b9a82583edd433491bb2485e66131431c189f51f609d3d8e1fa62"} Apr 16 17:11:43.623386 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:11:43.623352 2576 generic.go:358] "Generic (PLEG): container finished" podID="fa6e2817-5d34-4c5e-9436-ab23a1de66a7" containerID="2aa2a9833ffa3ffbf912a0f8263eb9807579094cc6e6c46425b06fec35373b7a" exitCode=0 Apr 16 17:11:43.623764 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:11:43.623394 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452" event={"ID":"fa6e2817-5d34-4c5e-9436-ab23a1de66a7","Type":"ContainerDied","Data":"2aa2a9833ffa3ffbf912a0f8263eb9807579094cc6e6c46425b06fec35373b7a"} Apr 16 17:11:44.628779 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:11:44.628732 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452" event={"ID":"fa6e2817-5d34-4c5e-9436-ab23a1de66a7","Type":"ContainerStarted","Data":"0af04a6524b71bbd61048ca2f6fc083985c71286dc509b8914f6bf0ef1fab4a0"} Apr 16 17:11:44.629202 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:11:44.629053 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452" Apr 16 17:11:44.630574 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:11:44.630550 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452" podUID="fa6e2817-5d34-4c5e-9436-ab23a1de66a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused" Apr 16 17:11:44.642625 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:11:44.642575 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452" podStartSLOduration=7.642558949 podStartE2EDuration="7.642558949s" podCreationTimestamp="2026-04-16 17:11:37 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 17:11:44.64145468 +0000 UTC m=+1443.702604212" watchObservedRunningTime="2026-04-16 17:11:44.642558949 +0000 UTC m=+1443.703708482" Apr 16 17:11:45.632564 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:11:45.632522 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452" podUID="fa6e2817-5d34-4c5e-9436-ab23a1de66a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused" Apr 16 17:11:55.633383 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:11:55.633339 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452" podUID="fa6e2817-5d34-4c5e-9436-ab23a1de66a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused" Apr 16 17:12:05.633085 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:12:05.633035 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452" podUID="fa6e2817-5d34-4c5e-9436-ab23a1de66a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused" Apr 16 17:12:15.633571 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:12:15.633518 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452" podUID="fa6e2817-5d34-4c5e-9436-ab23a1de66a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused" Apr 16 17:12:25.633072 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:12:25.633025 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452" podUID="fa6e2817-5d34-4c5e-9436-ab23a1de66a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused" Apr 16 17:12:35.633445 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:12:35.633410 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452" Apr 16 17:15:29.032346 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:15:29.032298 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl"] Apr 16 17:15:29.036014 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:15:29.035999 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl" Apr 16 17:15:29.043354 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:15:29.043167 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl"] Apr 16 17:15:29.095492 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:15:29.095458 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ef0fefe-cad8-48c9-b622-e02732895f5d-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl\" (UID: \"6ef0fefe-cad8-48c9-b622-e02732895f5d\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl" Apr 16 17:15:29.196684 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:15:29.196647 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ef0fefe-cad8-48c9-b622-e02732895f5d-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl\" (UID: \"6ef0fefe-cad8-48c9-b622-e02732895f5d\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl" Apr 16 17:15:29.197127 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:15:29.197102 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ef0fefe-cad8-48c9-b622-e02732895f5d-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl\" (UID: \"6ef0fefe-cad8-48c9-b622-e02732895f5d\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl" Apr 16 17:15:29.346836 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:15:29.346788 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl" Apr 16 17:15:29.463834 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:15:29.463806 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl"] Apr 16 17:15:29.465198 ip-10-0-142-37 kubenswrapper[2576]: W0416 17:15:29.465173 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef0fefe_cad8_48c9_b622_e02732895f5d.slice/crio-91641e9d64f69642b481d11c42f8f3d0b9175cd470cae9f934305af0b4098a0e WatchSource:0}: Error finding container 91641e9d64f69642b481d11c42f8f3d0b9175cd470cae9f934305af0b4098a0e: Status 404 returned error can't find the container with id 91641e9d64f69642b481d11c42f8f3d0b9175cd470cae9f934305af0b4098a0e Apr 16 17:15:29.467274 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:15:29.467234 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Apr 16 17:15:30.441026 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:15:30.440990 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl" event={"ID":"6ef0fefe-cad8-48c9-b622-e02732895f5d","Type":"ContainerStarted","Data":"ffcb93905a9e3d373c37db174e2d113f4a272eecc4cbdd0b3c21a4e1e3e7d818"} Apr 16 17:15:30.441449 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:15:30.441030 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl" event={"ID":"6ef0fefe-cad8-48c9-b622-e02732895f5d","Type":"ContainerStarted","Data":"91641e9d64f69642b481d11c42f8f3d0b9175cd470cae9f934305af0b4098a0e"} Apr 16 17:15:33.453620 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:15:33.453587 2576 generic.go:358] "Generic (PLEG): container finished" podID="6ef0fefe-cad8-48c9-b622-e02732895f5d" containerID="ffcb93905a9e3d373c37db174e2d113f4a272eecc4cbdd0b3c21a4e1e3e7d818" exitCode=0 Apr 16 17:15:33.453962 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:15:33.453661 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl" event={"ID":"6ef0fefe-cad8-48c9-b622-e02732895f5d","Type":"ContainerDied","Data":"ffcb93905a9e3d373c37db174e2d113f4a272eecc4cbdd0b3c21a4e1e3e7d818"} Apr 16 17:15:41.497095 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:15:41.497060 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl" event={"ID":"6ef0fefe-cad8-48c9-b622-e02732895f5d","Type":"ContainerStarted","Data":"0a8cb2548cfa5a6b7f7b3c8db638de6753a9c7ec250bb13c9a1b766d2ac6ce7c"} Apr 16 17:15:41.497549 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:15:41.497409 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl" Apr 16 17:15:41.498638 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:15:41.498611 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl" podUID="6ef0fefe-cad8-48c9-b622-e02732895f5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Apr 16 17:15:41.512465 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:15:41.512423 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl" podStartSLOduration=5.532463094 podStartE2EDuration="12.512410912s" podCreationTimestamp="2026-04-16 17:15:29 +0000 UTC" firstStartedPulling="2026-04-16 17:15:33.4547641 +0000 UTC m=+1672.515913609" lastFinishedPulling="2026-04-16 17:15:40.434711909 +0000 UTC m=+1679.495861427" observedRunningTime="2026-04-16 17:15:41.510115815 +0000 UTC m=+1680.571265347" watchObservedRunningTime="2026-04-16 17:15:41.512410912 +0000 UTC m=+1680.573560442" Apr 16 17:15:42.501913 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:15:42.501875 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl" podUID="6ef0fefe-cad8-48c9-b622-e02732895f5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Apr 16 17:15:52.502178 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:15:52.502076 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl" podUID="6ef0fefe-cad8-48c9-b622-e02732895f5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Apr 16 17:16:02.502202 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:16:02.502154 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl" podUID="6ef0fefe-cad8-48c9-b622-e02732895f5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Apr 16 17:16:12.502535 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:16:12.502478 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl" podUID="6ef0fefe-cad8-48c9-b622-e02732895f5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Apr 16 17:16:22.502766 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:16:22.502718 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl" podUID="6ef0fefe-cad8-48c9-b622-e02732895f5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Apr 16 17:16:32.502709 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:16:32.502663 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl" podUID="6ef0fefe-cad8-48c9-b622-e02732895f5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Apr 16 17:16:42.502316 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:16:42.502273 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl" podUID="6ef0fefe-cad8-48c9-b622-e02732895f5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Apr 16 17:16:52.502367 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:16:52.502321 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl" podUID="6ef0fefe-cad8-48c9-b622-e02732895f5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Apr 16 17:17:02.503445 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:17:02.503414 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl" Apr 16 17:17:09.303394 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:17:09.303358 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d8b668-predictor-57457558f8-f87zp"] Apr 16 17:17:09.307042 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:17:09.307026 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-d8b668-predictor-57457558f8-f87zp" Apr 16 17:17:09.314091 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:17:09.314065 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d8b668-predictor-57457558f8-f87zp"] Apr 16 17:17:09.488983 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:17:09.488945 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d27f89f4-30a7-46a4-8ff0-a6cf7879a4b2-kserve-provision-location\") pod \"isvc-primary-d8b668-predictor-57457558f8-f87zp\" (UID: \"d27f89f4-30a7-46a4-8ff0-a6cf7879a4b2\") " pod="kserve-ci-e2e-test/isvc-primary-d8b668-predictor-57457558f8-f87zp" Apr 16 17:17:09.589665 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:17:09.589543 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d27f89f4-30a7-46a4-8ff0-a6cf7879a4b2-kserve-provision-location\") pod \"isvc-primary-d8b668-predictor-57457558f8-f87zp\" (UID: \"d27f89f4-30a7-46a4-8ff0-a6cf7879a4b2\") " pod="kserve-ci-e2e-test/isvc-primary-d8b668-predictor-57457558f8-f87zp" Apr 16 17:17:09.590019 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:17:09.589995 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d27f89f4-30a7-46a4-8ff0-a6cf7879a4b2-kserve-provision-location\") pod \"isvc-primary-d8b668-predictor-57457558f8-f87zp\" (UID: \"d27f89f4-30a7-46a4-8ff0-a6cf7879a4b2\") " pod="kserve-ci-e2e-test/isvc-primary-d8b668-predictor-57457558f8-f87zp" Apr 16 17:17:09.619015 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:17:09.618983 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-d8b668-predictor-57457558f8-f87zp" Apr 16 17:17:09.739272 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:17:09.739230 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d8b668-predictor-57457558f8-f87zp"] Apr 16 17:17:09.741553 ip-10-0-142-37 kubenswrapper[2576]: W0416 17:17:09.741527 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd27f89f4_30a7_46a4_8ff0_a6cf7879a4b2.slice/crio-3d69f49b7efd2dd2bd6a31eb49207cddec26c61990c38bf9062a77f901409562 WatchSource:0}: Error finding container 3d69f49b7efd2dd2bd6a31eb49207cddec26c61990c38bf9062a77f901409562: Status 404 returned error can't find the container with id 3d69f49b7efd2dd2bd6a31eb49207cddec26c61990c38bf9062a77f901409562 Apr 16 17:17:09.844229 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:17:09.844144 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d8b668-predictor-57457558f8-f87zp" event={"ID":"d27f89f4-30a7-46a4-8ff0-a6cf7879a4b2","Type":"ContainerStarted","Data":"f1b08608ced9b273025cd1661dfa125f5e435d6e8edd9bdd7b9b46b48eb743ec"} Apr 16 17:17:09.844229 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:17:09.844181 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d8b668-predictor-57457558f8-f87zp" event={"ID":"d27f89f4-30a7-46a4-8ff0-a6cf7879a4b2","Type":"ContainerStarted","Data":"3d69f49b7efd2dd2bd6a31eb49207cddec26c61990c38bf9062a77f901409562"} Apr 16 17:17:13.860817 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:17:13.860773 2576 generic.go:358] "Generic (PLEG): container finished" podID="d27f89f4-30a7-46a4-8ff0-a6cf7879a4b2" containerID="f1b08608ced9b273025cd1661dfa125f5e435d6e8edd9bdd7b9b46b48eb743ec" exitCode=0 Apr 16 17:17:13.860817 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:17:13.860817 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d8b668-predictor-57457558f8-f87zp" event={"ID":"d27f89f4-30a7-46a4-8ff0-a6cf7879a4b2","Type":"ContainerDied","Data":"f1b08608ced9b273025cd1661dfa125f5e435d6e8edd9bdd7b9b46b48eb743ec"} Apr 16 17:17:14.866308 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:17:14.866272 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d8b668-predictor-57457558f8-f87zp" event={"ID":"d27f89f4-30a7-46a4-8ff0-a6cf7879a4b2","Type":"ContainerStarted","Data":"e949e78a664b03c83e80d9fef88c7b4070de33018ac08722f7e0fb8924ce863c"} Apr 16 17:17:14.866782 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:17:14.866615 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-d8b668-predictor-57457558f8-f87zp" Apr 16 17:17:14.867962 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:17:14.867939 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d8b668-predictor-57457558f8-f87zp" podUID="d27f89f4-30a7-46a4-8ff0-a6cf7879a4b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused" Apr 16 17:17:14.879706 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:17:14.879667 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-d8b668-predictor-57457558f8-f87zp" podStartSLOduration=5.879655546 podStartE2EDuration="5.879655546s" podCreationTimestamp="2026-04-16 17:17:09 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 17:17:14.879114535 +0000 UTC m=+1773.940264067" watchObservedRunningTime="2026-04-16 17:17:14.879655546 +0000 UTC m=+1773.940805076" Apr 16 17:17:15.870521 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:17:15.870438 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d8b668-predictor-57457558f8-f87zp" podUID="d27f89f4-30a7-46a4-8ff0-a6cf7879a4b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused" Apr 16 17:17:25.871365 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:17:25.871320 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d8b668-predictor-57457558f8-f87zp" podUID="d27f89f4-30a7-46a4-8ff0-a6cf7879a4b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused" Apr 16 17:17:35.871140 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:17:35.871095 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d8b668-predictor-57457558f8-f87zp" podUID="d27f89f4-30a7-46a4-8ff0-a6cf7879a4b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused" Apr 16 17:17:45.871311 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:17:45.871235 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d8b668-predictor-57457558f8-f87zp" podUID="d27f89f4-30a7-46a4-8ff0-a6cf7879a4b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused" Apr 16 17:17:55.871240 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:17:55.871194 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d8b668-predictor-57457558f8-f87zp" podUID="d27f89f4-30a7-46a4-8ff0-a6cf7879a4b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused" Apr 16 17:18:05.870600 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:05.870552 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d8b668-predictor-57457558f8-f87zp" podUID="d27f89f4-30a7-46a4-8ff0-a6cf7879a4b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused" Apr 16 17:18:15.871236 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:15.871196 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d8b668-predictor-57457558f8-f87zp" podUID="d27f89f4-30a7-46a4-8ff0-a6cf7879a4b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused" Apr 16 17:18:25.871940 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:25.871904 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-d8b668-predictor-57457558f8-f87zp" Apr 16 17:18:29.421269 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:29.421219 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d8b668-predictor-6f9994ff99-wdvzg"] Apr 16 17:18:29.424822 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:29.424804 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-d8b668-predictor-6f9994ff99-wdvzg" Apr 16 17:18:29.427507 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:29.427483 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-d8b668\"" Apr 16 17:18:29.427614 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:29.427505 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Apr 16 17:18:29.427614 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:29.427506 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-d8b668-dockercfg-xlvxh\"" Apr 16 17:18:29.435144 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:29.435114 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d8b668-predictor-6f9994ff99-wdvzg"] Apr 16 17:18:29.445055 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:29.445025 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1-kserve-provision-location\") pod \"isvc-secondary-d8b668-predictor-6f9994ff99-wdvzg\" (UID: \"ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1\") " pod="kserve-ci-e2e-test/isvc-secondary-d8b668-predictor-6f9994ff99-wdvzg" Apr 16 17:18:29.445206 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:29.445093 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1-cabundle-cert\") pod \"isvc-secondary-d8b668-predictor-6f9994ff99-wdvzg\" (UID: \"ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1\") " pod="kserve-ci-e2e-test/isvc-secondary-d8b668-predictor-6f9994ff99-wdvzg" Apr 16 17:18:29.545610 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:29.545574 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1-cabundle-cert\") pod \"isvc-secondary-d8b668-predictor-6f9994ff99-wdvzg\" (UID: \"ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1\") " pod="kserve-ci-e2e-test/isvc-secondary-d8b668-predictor-6f9994ff99-wdvzg" Apr 16 17:18:29.545802 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:29.545661 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1-kserve-provision-location\") pod \"isvc-secondary-d8b668-predictor-6f9994ff99-wdvzg\" (UID: \"ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1\") " pod="kserve-ci-e2e-test/isvc-secondary-d8b668-predictor-6f9994ff99-wdvzg" Apr 16 17:18:29.545995 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:29.545975 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1-kserve-provision-location\") pod \"isvc-secondary-d8b668-predictor-6f9994ff99-wdvzg\" (UID: \"ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1\") " pod="kserve-ci-e2e-test/isvc-secondary-d8b668-predictor-6f9994ff99-wdvzg" Apr 16 17:18:29.546304 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:29.546282 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1-cabundle-cert\") pod \"isvc-secondary-d8b668-predictor-6f9994ff99-wdvzg\" (UID: \"ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1\") " pod="kserve-ci-e2e-test/isvc-secondary-d8b668-predictor-6f9994ff99-wdvzg" Apr 16 17:18:29.736946 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:29.736832 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-d8b668-predictor-6f9994ff99-wdvzg" Apr 16 17:18:29.865316 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:29.865281 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d8b668-predictor-6f9994ff99-wdvzg"] Apr 16 17:18:29.868615 ip-10-0-142-37 kubenswrapper[2576]: W0416 17:18:29.868586 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded6bb25e_ebcb_4c82_833d_3322c1ca1ec1.slice/crio-c13e3ccef4cfa9a9030dc6b9cb2b32fcc835bea27eb12148aeb206a6b2f53547 WatchSource:0}: Error finding container c13e3ccef4cfa9a9030dc6b9cb2b32fcc835bea27eb12148aeb206a6b2f53547: Status 404 returned error can't find the container with id c13e3ccef4cfa9a9030dc6b9cb2b32fcc835bea27eb12148aeb206a6b2f53547 Apr 16 17:18:30.150040 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:30.150001 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d8b668-predictor-6f9994ff99-wdvzg" event={"ID":"ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1","Type":"ContainerStarted","Data":"370d968ec177ea9d60068aa63e424527fcc4e1dbf9bc5d1735c85ec07fd313a0"} Apr 16 17:18:30.150040 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:30.150043 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d8b668-predictor-6f9994ff99-wdvzg" event={"ID":"ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1","Type":"ContainerStarted","Data":"c13e3ccef4cfa9a9030dc6b9cb2b32fcc835bea27eb12148aeb206a6b2f53547"} Apr 16 17:18:35.174306 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:35.174277 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d8b668-predictor-6f9994ff99-wdvzg_ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1/storage-initializer/0.log" Apr 16 17:18:35.174686 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:35.174314 2576 generic.go:358] "Generic (PLEG): container finished" podID="ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1" containerID="370d968ec177ea9d60068aa63e424527fcc4e1dbf9bc5d1735c85ec07fd313a0" exitCode=1 Apr 16 17:18:35.174686 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:35.174378 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d8b668-predictor-6f9994ff99-wdvzg" event={"ID":"ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1","Type":"ContainerDied","Data":"370d968ec177ea9d60068aa63e424527fcc4e1dbf9bc5d1735c85ec07fd313a0"} Apr 16 17:18:36.179548 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:36.179465 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d8b668-predictor-6f9994ff99-wdvzg_ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1/storage-initializer/0.log" Apr 16 17:18:36.180021 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:36.179549 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d8b668-predictor-6f9994ff99-wdvzg" event={"ID":"ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1","Type":"ContainerStarted","Data":"cc2264f822cd589fe2db170c25389cf23c0e0cfc2e1e73fb7ef3788bee31a5bd"} Apr 16 17:18:40.195615 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:40.195588 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d8b668-predictor-6f9994ff99-wdvzg_ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1/storage-initializer/1.log" Apr 16 17:18:40.196030 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:40.195915 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d8b668-predictor-6f9994ff99-wdvzg_ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1/storage-initializer/0.log" Apr 16 17:18:40.196030 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:40.195952 2576 generic.go:358] "Generic (PLEG): container finished" podID="ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1" containerID="cc2264f822cd589fe2db170c25389cf23c0e0cfc2e1e73fb7ef3788bee31a5bd" exitCode=1 Apr 16 17:18:40.196030 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:40.196021 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d8b668-predictor-6f9994ff99-wdvzg" event={"ID":"ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1","Type":"ContainerDied","Data":"cc2264f822cd589fe2db170c25389cf23c0e0cfc2e1e73fb7ef3788bee31a5bd"} Apr 16 17:18:40.196133 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:40.196060 2576 scope.go:117] "RemoveContainer" containerID="370d968ec177ea9d60068aa63e424527fcc4e1dbf9bc5d1735c85ec07fd313a0" Apr 16 17:18:40.196467 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:40.196447 2576 scope.go:117] "RemoveContainer" containerID="370d968ec177ea9d60068aa63e424527fcc4e1dbf9bc5d1735c85ec07fd313a0" Apr 16 17:18:40.207959 ip-10-0-142-37 kubenswrapper[2576]: E0416 17:18:40.207932 2576 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-d8b668-predictor-6f9994ff99-wdvzg_kserve-ci-e2e-test_ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1_0 in pod sandbox c13e3ccef4cfa9a9030dc6b9cb2b32fcc835bea27eb12148aeb206a6b2f53547 from index: no such id: '370d968ec177ea9d60068aa63e424527fcc4e1dbf9bc5d1735c85ec07fd313a0'" containerID="370d968ec177ea9d60068aa63e424527fcc4e1dbf9bc5d1735c85ec07fd313a0" Apr 16 17:18:40.208022 ip-10-0-142-37 kubenswrapper[2576]: E0416 17:18:40.207985 2576 kuberuntime_container.go:951] "Unhandled Error" err="failed to remove pod init container \"storage-initializer\": rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-d8b668-predictor-6f9994ff99-wdvzg_kserve-ci-e2e-test_ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1_0 in pod sandbox c13e3ccef4cfa9a9030dc6b9cb2b32fcc835bea27eb12148aeb206a6b2f53547 from index: no such id: '370d968ec177ea9d60068aa63e424527fcc4e1dbf9bc5d1735c85ec07fd313a0'; Skipping pod \"isvc-secondary-d8b668-predictor-6f9994ff99-wdvzg_kserve-ci-e2e-test(ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1)\"" logger="UnhandledError" Apr 16 17:18:40.209466 ip-10-0-142-37 kubenswrapper[2576]: E0416 17:18:40.209430 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-d8b668-predictor-6f9994ff99-wdvzg_kserve-ci-e2e-test(ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1)\"" pod="kserve-ci-e2e-test/isvc-secondary-d8b668-predictor-6f9994ff99-wdvzg" podUID="ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1" Apr 16 17:18:41.201864 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:41.201836 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d8b668-predictor-6f9994ff99-wdvzg_ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1/storage-initializer/1.log" Apr 16 17:18:45.497019 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:45.496926 2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d8b668-predictor-6f9994ff99-wdvzg"] Apr 16 17:18:45.545196 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:45.545159 2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d8b668-predictor-57457558f8-f87zp"] Apr 16 17:18:45.545864 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:45.545806 2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-d8b668-predictor-57457558f8-f87zp" podUID="d27f89f4-30a7-46a4-8ff0-a6cf7879a4b2" containerName="kserve-container" containerID="cri-o://e949e78a664b03c83e80d9fef88c7b4070de33018ac08722f7e0fb8924ce863c" gracePeriod=30 Apr 16 17:18:45.640850 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:45.640829 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d8b668-predictor-6f9994ff99-wdvzg_ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1/storage-initializer/1.log" Apr 16 17:18:45.640998 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:45.640894 2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-d8b668-predictor-6f9994ff99-wdvzg" Apr 16 17:18:45.645898 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:45.645871 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-8e007e-predictor-f8956bf75-84p8l"] Apr 16 17:18:45.646399 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:45.646382 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1" containerName="storage-initializer" Apr 16 17:18:45.646399 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:45.646401 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1" containerName="storage-initializer" Apr 16 17:18:45.646554 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:45.646422 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1" containerName="storage-initializer" Apr 16 17:18:45.646554 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:45.646430 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1" containerName="storage-initializer" Apr 16 17:18:45.646554 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:45.646527 2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1" containerName="storage-initializer" Apr 16 17:18:45.649174 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:45.647322 2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1" containerName="storage-initializer" Apr 16 17:18:45.653485 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:45.653443 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-8e007e-predictor-f8956bf75-84p8l" Apr 16 17:18:45.656633 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:45.656611 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-8e007e-dockercfg-dvc8j\"" Apr 16 17:18:45.657398 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:45.657379 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-8e007e\"" Apr 16 17:18:45.658683 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:45.658665 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-8e007e-predictor-f8956bf75-84p8l"] Apr 16 17:18:45.683553 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:45.683525 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1-kserve-provision-location\") pod \"ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1\" (UID: \"ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1\") " Apr 16 17:18:45.683716 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:45.683577 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1-cabundle-cert\") pod \"ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1\" (UID: \"ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1\") " Apr 16 17:18:45.683861 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:45.683804 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1" (UID: "ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 16 17:18:45.683939 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:45.683892 2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1-kserve-provision-location\") on node \"ip-10-0-142-37.ec2.internal\" DevicePath \"\"" Apr 16 17:18:45.683981 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:45.683933 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1" (UID: "ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 17:18:45.784954 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:45.784856 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec47cb42-ff06-4144-9049-d48e8b2da896-kserve-provision-location\") pod \"isvc-init-fail-8e007e-predictor-f8956bf75-84p8l\" (UID: \"ec47cb42-ff06-4144-9049-d48e8b2da896\") " pod="kserve-ci-e2e-test/isvc-init-fail-8e007e-predictor-f8956bf75-84p8l" Apr 16 17:18:45.784954 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:45.784904 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec47cb42-ff06-4144-9049-d48e8b2da896-cabundle-cert\") pod \"isvc-init-fail-8e007e-predictor-f8956bf75-84p8l\" (UID: \"ec47cb42-ff06-4144-9049-d48e8b2da896\") " pod="kserve-ci-e2e-test/isvc-init-fail-8e007e-predictor-f8956bf75-84p8l" Apr 16 17:18:45.785134 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:45.785043 2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1-cabundle-cert\") on node \"ip-10-0-142-37.ec2.internal\" DevicePath \"\"" Apr 16 17:18:45.871321 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:45.871235 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d8b668-predictor-57457558f8-f87zp" podUID="d27f89f4-30a7-46a4-8ff0-a6cf7879a4b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused" Apr 16 17:18:45.886408 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:45.886377 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec47cb42-ff06-4144-9049-d48e8b2da896-kserve-provision-location\") pod \"isvc-init-fail-8e007e-predictor-f8956bf75-84p8l\" (UID: \"ec47cb42-ff06-4144-9049-d48e8b2da896\") " pod="kserve-ci-e2e-test/isvc-init-fail-8e007e-predictor-f8956bf75-84p8l" Apr 16 17:18:45.886537 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:45.886416 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec47cb42-ff06-4144-9049-d48e8b2da896-cabundle-cert\") pod \"isvc-init-fail-8e007e-predictor-f8956bf75-84p8l\" (UID: \"ec47cb42-ff06-4144-9049-d48e8b2da896\") " pod="kserve-ci-e2e-test/isvc-init-fail-8e007e-predictor-f8956bf75-84p8l" Apr 16 17:18:45.886762 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:45.886744 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec47cb42-ff06-4144-9049-d48e8b2da896-kserve-provision-location\") pod \"isvc-init-fail-8e007e-predictor-f8956bf75-84p8l\" (UID: \"ec47cb42-ff06-4144-9049-d48e8b2da896\") " pod="kserve-ci-e2e-test/isvc-init-fail-8e007e-predictor-f8956bf75-84p8l" Apr 16 17:18:45.887011 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:45.886994 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec47cb42-ff06-4144-9049-d48e8b2da896-cabundle-cert\") pod \"isvc-init-fail-8e007e-predictor-f8956bf75-84p8l\" (UID: \"ec47cb42-ff06-4144-9049-d48e8b2da896\") " pod="kserve-ci-e2e-test/isvc-init-fail-8e007e-predictor-f8956bf75-84p8l" Apr 16 17:18:45.968109 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:45.968071 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-8e007e-predictor-f8956bf75-84p8l" Apr 16 17:18:46.097860 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:46.097822 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-8e007e-predictor-f8956bf75-84p8l"] Apr 16 17:18:46.101197 ip-10-0-142-37 kubenswrapper[2576]: W0416 17:18:46.101167 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec47cb42_ff06_4144_9049_d48e8b2da896.slice/crio-954332426a988f34e86a3c11566189479f77b7986562c20da54ed2374fd632db WatchSource:0}: Error finding container 954332426a988f34e86a3c11566189479f77b7986562c20da54ed2374fd632db: Status 404 returned error can't find the container with id 954332426a988f34e86a3c11566189479f77b7986562c20da54ed2374fd632db Apr 16 17:18:46.223567 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:46.223540 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d8b668-predictor-6f9994ff99-wdvzg_ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1/storage-initializer/1.log" Apr 16 17:18:46.223893 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:46.223642 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d8b668-predictor-6f9994ff99-wdvzg" event={"ID":"ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1","Type":"ContainerDied","Data":"c13e3ccef4cfa9a9030dc6b9cb2b32fcc835bea27eb12148aeb206a6b2f53547"} Apr 16 17:18:46.223893 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:46.223684 2576 scope.go:117] "RemoveContainer" containerID="cc2264f822cd589fe2db170c25389cf23c0e0cfc2e1e73fb7ef3788bee31a5bd" Apr 16 17:18:46.223893 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:46.223717 2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-d8b668-predictor-6f9994ff99-wdvzg" Apr 16 17:18:46.225415 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:46.225387 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-8e007e-predictor-f8956bf75-84p8l" event={"ID":"ec47cb42-ff06-4144-9049-d48e8b2da896","Type":"ContainerStarted","Data":"bf296e0ff2e00fb9ab2183d0dda0a211d576e30f0d662a6e869150f07acf503e"} Apr 16 17:18:46.225532 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:46.225423 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-8e007e-predictor-f8956bf75-84p8l" event={"ID":"ec47cb42-ff06-4144-9049-d48e8b2da896","Type":"ContainerStarted","Data":"954332426a988f34e86a3c11566189479f77b7986562c20da54ed2374fd632db"} Apr 16 17:18:46.261365 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:46.261328 2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d8b668-predictor-6f9994ff99-wdvzg"] Apr 16 17:18:46.266395 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:46.266373 2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d8b668-predictor-6f9994ff99-wdvzg"] Apr 16 17:18:47.593080 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:47.593046 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1" path="/var/lib/kubelet/pods/ed6bb25e-ebcb-4c82-833d-3322c1ca1ec1/volumes" Apr 16 17:18:49.899276 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:49.899234 2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-d8b668-predictor-57457558f8-f87zp" Apr 16 17:18:50.021122 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:50.021051 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d27f89f4-30a7-46a4-8ff0-a6cf7879a4b2-kserve-provision-location\") pod \"d27f89f4-30a7-46a4-8ff0-a6cf7879a4b2\" (UID: \"d27f89f4-30a7-46a4-8ff0-a6cf7879a4b2\") " Apr 16 17:18:50.021417 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:50.021392 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d27f89f4-30a7-46a4-8ff0-a6cf7879a4b2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d27f89f4-30a7-46a4-8ff0-a6cf7879a4b2" (UID: "d27f89f4-30a7-46a4-8ff0-a6cf7879a4b2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 16 17:18:50.121784 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:50.121750 2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d27f89f4-30a7-46a4-8ff0-a6cf7879a4b2-kserve-provision-location\") on node \"ip-10-0-142-37.ec2.internal\" DevicePath \"\"" Apr 16 17:18:50.252616 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:50.252589 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-8e007e-predictor-f8956bf75-84p8l_ec47cb42-ff06-4144-9049-d48e8b2da896/storage-initializer/0.log" Apr 16 17:18:50.252793 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:50.252629 2576 generic.go:358] "Generic (PLEG): container finished" podID="ec47cb42-ff06-4144-9049-d48e8b2da896" containerID="bf296e0ff2e00fb9ab2183d0dda0a211d576e30f0d662a6e869150f07acf503e" exitCode=1 Apr 16 17:18:50.252793 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:50.252712 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-8e007e-predictor-f8956bf75-84p8l" event={"ID":"ec47cb42-ff06-4144-9049-d48e8b2da896","Type":"ContainerDied","Data":"bf296e0ff2e00fb9ab2183d0dda0a211d576e30f0d662a6e869150f07acf503e"} Apr 16 17:18:50.254312 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:50.254281 2576 generic.go:358] "Generic (PLEG): container finished" podID="d27f89f4-30a7-46a4-8ff0-a6cf7879a4b2" containerID="e949e78a664b03c83e80d9fef88c7b4070de33018ac08722f7e0fb8924ce863c" exitCode=0 Apr 16 17:18:50.254427 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:50.254371 2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-d8b668-predictor-57457558f8-f87zp" Apr 16 17:18:50.254427 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:50.254370 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d8b668-predictor-57457558f8-f87zp" event={"ID":"d27f89f4-30a7-46a4-8ff0-a6cf7879a4b2","Type":"ContainerDied","Data":"e949e78a664b03c83e80d9fef88c7b4070de33018ac08722f7e0fb8924ce863c"} Apr 16 17:18:50.254427 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:50.254412 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d8b668-predictor-57457558f8-f87zp" event={"ID":"d27f89f4-30a7-46a4-8ff0-a6cf7879a4b2","Type":"ContainerDied","Data":"3d69f49b7efd2dd2bd6a31eb49207cddec26c61990c38bf9062a77f901409562"} Apr 16 17:18:50.254574 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:50.254434 2576 scope.go:117] "RemoveContainer" containerID="e949e78a664b03c83e80d9fef88c7b4070de33018ac08722f7e0fb8924ce863c" Apr 16 17:18:50.263498 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:50.263471 2576 scope.go:117] "RemoveContainer" containerID="f1b08608ced9b273025cd1661dfa125f5e435d6e8edd9bdd7b9b46b48eb743ec" Apr 16 17:18:50.271868 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:50.271848 2576 scope.go:117] "RemoveContainer" containerID="e949e78a664b03c83e80d9fef88c7b4070de33018ac08722f7e0fb8924ce863c" Apr 16 17:18:50.272122 ip-10-0-142-37 kubenswrapper[2576]: E0416 17:18:50.272104 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e949e78a664b03c83e80d9fef88c7b4070de33018ac08722f7e0fb8924ce863c\": container with ID starting with e949e78a664b03c83e80d9fef88c7b4070de33018ac08722f7e0fb8924ce863c not found: ID does not exist" containerID="e949e78a664b03c83e80d9fef88c7b4070de33018ac08722f7e0fb8924ce863c" Apr 16 17:18:50.272166 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:50.272131 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e949e78a664b03c83e80d9fef88c7b4070de33018ac08722f7e0fb8924ce863c"} err="failed to get container status \"e949e78a664b03c83e80d9fef88c7b4070de33018ac08722f7e0fb8924ce863c\": rpc error: code = NotFound desc = could not find container \"e949e78a664b03c83e80d9fef88c7b4070de33018ac08722f7e0fb8924ce863c\": container with ID starting with e949e78a664b03c83e80d9fef88c7b4070de33018ac08722f7e0fb8924ce863c not found: ID does not exist" Apr 16 17:18:50.272166 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:50.272147 2576 scope.go:117] "RemoveContainer" containerID="f1b08608ced9b273025cd1661dfa125f5e435d6e8edd9bdd7b9b46b48eb743ec" Apr 16 17:18:50.272435 ip-10-0-142-37 kubenswrapper[2576]: E0416 17:18:50.272412 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1b08608ced9b273025cd1661dfa125f5e435d6e8edd9bdd7b9b46b48eb743ec\": container with ID starting with f1b08608ced9b273025cd1661dfa125f5e435d6e8edd9bdd7b9b46b48eb743ec not found: ID does not exist" containerID="f1b08608ced9b273025cd1661dfa125f5e435d6e8edd9bdd7b9b46b48eb743ec" Apr 16 17:18:50.272492 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:50.272447 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1b08608ced9b273025cd1661dfa125f5e435d6e8edd9bdd7b9b46b48eb743ec"} err="failed to get container status \"f1b08608ced9b273025cd1661dfa125f5e435d6e8edd9bdd7b9b46b48eb743ec\": rpc error: code = NotFound desc = could not find container \"f1b08608ced9b273025cd1661dfa125f5e435d6e8edd9bdd7b9b46b48eb743ec\": container with ID starting with f1b08608ced9b273025cd1661dfa125f5e435d6e8edd9bdd7b9b46b48eb743ec not found: ID does not exist" Apr 16 17:18:50.277976 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:50.277952 2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d8b668-predictor-57457558f8-f87zp"] Apr 16 17:18:50.281880 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:50.281859 2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d8b668-predictor-57457558f8-f87zp"] Apr 16 17:18:50.638459 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:50.638423 2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-8e007e-predictor-f8956bf75-84p8l"] Apr 16 17:18:51.260045 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:51.260019 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-8e007e-predictor-f8956bf75-84p8l_ec47cb42-ff06-4144-9049-d48e8b2da896/storage-initializer/0.log" Apr 16 17:18:51.260507 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:51.260147 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-8e007e-predictor-f8956bf75-84p8l" event={"ID":"ec47cb42-ff06-4144-9049-d48e8b2da896","Type":"ContainerStarted","Data":"1369f4b09be71b1b446e795a920fe635bf966a4814d93e05cf868f11fe7232bf"} Apr 16 17:18:51.260507 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:51.260269 2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-8e007e-predictor-f8956bf75-84p8l" podUID="ec47cb42-ff06-4144-9049-d48e8b2da896" containerName="storage-initializer" containerID="cri-o://1369f4b09be71b1b446e795a920fe635bf966a4814d93e05cf868f11fe7232bf" gracePeriod=30 Apr 16 17:18:51.594161 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:51.594131 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d27f89f4-30a7-46a4-8ff0-a6cf7879a4b2" path="/var/lib/kubelet/pods/d27f89f4-30a7-46a4-8ff0-a6cf7879a4b2/volumes" Apr 16 17:18:56.113031 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:56.113008 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-8e007e-predictor-f8956bf75-84p8l_ec47cb42-ff06-4144-9049-d48e8b2da896/storage-initializer/1.log" Apr 16 17:18:56.113403 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:56.113382 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-8e007e-predictor-f8956bf75-84p8l_ec47cb42-ff06-4144-9049-d48e8b2da896/storage-initializer/0.log" Apr 16 17:18:56.113446 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:56.113438 2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-8e007e-predictor-f8956bf75-84p8l" Apr 16 17:18:56.178092 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:56.178053 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec47cb42-ff06-4144-9049-d48e8b2da896-cabundle-cert\") pod \"ec47cb42-ff06-4144-9049-d48e8b2da896\" (UID: \"ec47cb42-ff06-4144-9049-d48e8b2da896\") " Apr 16 17:18:56.178279 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:56.178104 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec47cb42-ff06-4144-9049-d48e8b2da896-kserve-provision-location\") pod \"ec47cb42-ff06-4144-9049-d48e8b2da896\" (UID: \"ec47cb42-ff06-4144-9049-d48e8b2da896\") " Apr 16 17:18:56.178422 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:56.178399 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec47cb42-ff06-4144-9049-d48e8b2da896-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ec47cb42-ff06-4144-9049-d48e8b2da896" (UID: "ec47cb42-ff06-4144-9049-d48e8b2da896"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 16 17:18:56.178481 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:56.178469 2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec47cb42-ff06-4144-9049-d48e8b2da896-kserve-provision-location\") on node \"ip-10-0-142-37.ec2.internal\" DevicePath \"\"" Apr 16 17:18:56.178519 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:56.178476 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec47cb42-ff06-4144-9049-d48e8b2da896-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ec47cb42-ff06-4144-9049-d48e8b2da896" (UID: "ec47cb42-ff06-4144-9049-d48e8b2da896"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 17:18:56.278941 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:56.278860 2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec47cb42-ff06-4144-9049-d48e8b2da896-cabundle-cert\") on node \"ip-10-0-142-37.ec2.internal\" DevicePath \"\"" Apr 16 17:18:56.280735 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:56.280710 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-8e007e-predictor-f8956bf75-84p8l_ec47cb42-ff06-4144-9049-d48e8b2da896/storage-initializer/1.log" Apr 16 17:18:56.281076 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:56.281060 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-8e007e-predictor-f8956bf75-84p8l_ec47cb42-ff06-4144-9049-d48e8b2da896/storage-initializer/0.log" Apr 16 17:18:56.281144 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:56.281104 2576 generic.go:358] "Generic (PLEG): container finished" podID="ec47cb42-ff06-4144-9049-d48e8b2da896" containerID="1369f4b09be71b1b446e795a920fe635bf966a4814d93e05cf868f11fe7232bf" exitCode=1 Apr 16 17:18:56.281386 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:56.281165 2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-8e007e-predictor-f8956bf75-84p8l" Apr 16 17:18:56.281386 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:56.281191 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-8e007e-predictor-f8956bf75-84p8l" event={"ID":"ec47cb42-ff06-4144-9049-d48e8b2da896","Type":"ContainerDied","Data":"1369f4b09be71b1b446e795a920fe635bf966a4814d93e05cf868f11fe7232bf"} Apr 16 17:18:56.281386 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:56.281233 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-8e007e-predictor-f8956bf75-84p8l" event={"ID":"ec47cb42-ff06-4144-9049-d48e8b2da896","Type":"ContainerDied","Data":"954332426a988f34e86a3c11566189479f77b7986562c20da54ed2374fd632db"} Apr 16 17:18:56.281386 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:56.281271 2576 scope.go:117] "RemoveContainer" containerID="1369f4b09be71b1b446e795a920fe635bf966a4814d93e05cf868f11fe7232bf" Apr 16 17:18:56.292988 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:56.292967 2576 scope.go:117] "RemoveContainer" containerID="bf296e0ff2e00fb9ab2183d0dda0a211d576e30f0d662a6e869150f07acf503e" Apr 16 17:18:56.301017 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:56.300990 2576 scope.go:117] "RemoveContainer" containerID="1369f4b09be71b1b446e795a920fe635bf966a4814d93e05cf868f11fe7232bf" Apr 16 17:18:56.301281 ip-10-0-142-37 kubenswrapper[2576]: E0416 17:18:56.301257 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1369f4b09be71b1b446e795a920fe635bf966a4814d93e05cf868f11fe7232bf\": container with ID starting with 1369f4b09be71b1b446e795a920fe635bf966a4814d93e05cf868f11fe7232bf not found: ID does not exist" containerID="1369f4b09be71b1b446e795a920fe635bf966a4814d93e05cf868f11fe7232bf" Apr 16 17:18:56.301356 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:56.301289 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1369f4b09be71b1b446e795a920fe635bf966a4814d93e05cf868f11fe7232bf"} err="failed to get container status \"1369f4b09be71b1b446e795a920fe635bf966a4814d93e05cf868f11fe7232bf\": rpc error: code = NotFound desc = could not find container \"1369f4b09be71b1b446e795a920fe635bf966a4814d93e05cf868f11fe7232bf\": container with ID starting with 1369f4b09be71b1b446e795a920fe635bf966a4814d93e05cf868f11fe7232bf not found: ID does not exist" Apr 16 17:18:56.301356 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:56.301318 2576 scope.go:117] "RemoveContainer" containerID="bf296e0ff2e00fb9ab2183d0dda0a211d576e30f0d662a6e869150f07acf503e" Apr 16 17:18:56.301586 ip-10-0-142-37 kubenswrapper[2576]: E0416 17:18:56.301561 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf296e0ff2e00fb9ab2183d0dda0a211d576e30f0d662a6e869150f07acf503e\": container with ID starting with bf296e0ff2e00fb9ab2183d0dda0a211d576e30f0d662a6e869150f07acf503e not found: ID does not exist" containerID="bf296e0ff2e00fb9ab2183d0dda0a211d576e30f0d662a6e869150f07acf503e" Apr 16 17:18:56.301646 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:56.301585 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf296e0ff2e00fb9ab2183d0dda0a211d576e30f0d662a6e869150f07acf503e"} err="failed to get container status \"bf296e0ff2e00fb9ab2183d0dda0a211d576e30f0d662a6e869150f07acf503e\": rpc error: code = NotFound desc = could not find container \"bf296e0ff2e00fb9ab2183d0dda0a211d576e30f0d662a6e869150f07acf503e\": container with ID starting with bf296e0ff2e00fb9ab2183d0dda0a211d576e30f0d662a6e869150f07acf503e not found: ID does not exist" Apr 16 17:18:56.312871 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:56.312849 2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-8e007e-predictor-f8956bf75-84p8l"] Apr 16 17:18:56.316146 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:56.316124 2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-8e007e-predictor-f8956bf75-84p8l"] Apr 16 17:18:57.593324 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:18:57.593294 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec47cb42-ff06-4144-9049-d48e8b2da896" path="/var/lib/kubelet/pods/ec47cb42-ff06-4144-9049-d48e8b2da896/volumes" Apr 16 17:20:31.146474 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:20:31.146436 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7"] Apr 16 17:20:31.148927 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:20:31.146865 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d27f89f4-30a7-46a4-8ff0-a6cf7879a4b2" containerName="kserve-container" Apr 16 17:20:31.148927 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:20:31.146878 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d27f89f4-30a7-46a4-8ff0-a6cf7879a4b2" containerName="kserve-container" Apr 16 17:20:31.148927 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:20:31.146904 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec47cb42-ff06-4144-9049-d48e8b2da896" containerName="storage-initializer" Apr 16 17:20:31.148927 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:20:31.146910 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec47cb42-ff06-4144-9049-d48e8b2da896" containerName="storage-initializer" Apr 16 17:20:31.148927 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:20:31.146919 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d27f89f4-30a7-46a4-8ff0-a6cf7879a4b2" containerName="storage-initializer" Apr 16 17:20:31.148927 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:20:31.146924 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d27f89f4-30a7-46a4-8ff0-a6cf7879a4b2" containerName="storage-initializer" Apr 16 17:20:31.148927 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:20:31.146933 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec47cb42-ff06-4144-9049-d48e8b2da896" containerName="storage-initializer" Apr 16 17:20:31.148927 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:20:31.146939 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec47cb42-ff06-4144-9049-d48e8b2da896" containerName="storage-initializer" Apr 16 17:20:31.148927 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:20:31.146987 2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="ec47cb42-ff06-4144-9049-d48e8b2da896" containerName="storage-initializer" Apr 16 17:20:31.148927 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:20:31.146997 2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="d27f89f4-30a7-46a4-8ff0-a6cf7879a4b2" containerName="kserve-container" Apr 16 17:20:31.148927 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:20:31.147004 2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="ec47cb42-ff06-4144-9049-d48e8b2da896" containerName="storage-initializer" Apr 16 17:20:31.150010 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:20:31.149994 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7" Apr 16 17:20:31.158129 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:20:31.157834 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7"] Apr 16 17:20:31.215654 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:20:31.215612 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/969ea101-644f-4d51-9566-7e8cf20aa33f-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7\" (UID: \"969ea101-644f-4d51-9566-7e8cf20aa33f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7" Apr 16 17:20:31.316939 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:20:31.316888 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/969ea101-644f-4d51-9566-7e8cf20aa33f-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7\" (UID: \"969ea101-644f-4d51-9566-7e8cf20aa33f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7" Apr 16 17:20:31.317314 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:20:31.317292 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/969ea101-644f-4d51-9566-7e8cf20aa33f-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7\" (UID: \"969ea101-644f-4d51-9566-7e8cf20aa33f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7" Apr 16 17:20:31.461829 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:20:31.461729 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7" Apr 16 17:20:31.592835 ip-10-0-142-37 kubenswrapper[2576]: W0416 17:20:31.592793 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod969ea101_644f_4d51_9566_7e8cf20aa33f.slice/crio-5f92ef6d03a327e6578c18dae068f20f3563febf4e00268e1e5ebb3af5774307 WatchSource:0}: Error finding container 5f92ef6d03a327e6578c18dae068f20f3563febf4e00268e1e5ebb3af5774307: Status 404 returned error can't find the container with id 5f92ef6d03a327e6578c18dae068f20f3563febf4e00268e1e5ebb3af5774307 Apr 16 17:20:31.594709 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:20:31.594691 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Apr 16 17:20:31.595052 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:20:31.595032 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7"] Apr 16 17:20:31.643368 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:20:31.642960 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7" event={"ID":"969ea101-644f-4d51-9566-7e8cf20aa33f","Type":"ContainerStarted","Data":"5f92ef6d03a327e6578c18dae068f20f3563febf4e00268e1e5ebb3af5774307"} Apr 16 17:20:32.648315 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:20:32.648271 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7" event={"ID":"969ea101-644f-4d51-9566-7e8cf20aa33f","Type":"ContainerStarted","Data":"f52c549e6ee4506eeb493b177491ee41c16be0c9e329d5795ef7714935d98faf"} Apr 16 17:20:36.663848 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:20:36.663815 2576 generic.go:358] "Generic (PLEG): container finished" podID="969ea101-644f-4d51-9566-7e8cf20aa33f" containerID="f52c549e6ee4506eeb493b177491ee41c16be0c9e329d5795ef7714935d98faf" exitCode=0 Apr 16 17:20:36.664223 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:20:36.663862 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7" event={"ID":"969ea101-644f-4d51-9566-7e8cf20aa33f","Type":"ContainerDied","Data":"f52c549e6ee4506eeb493b177491ee41c16be0c9e329d5795ef7714935d98faf"} Apr 16 17:20:57.754235 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:20:57.754143 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7" event={"ID":"969ea101-644f-4d51-9566-7e8cf20aa33f","Type":"ContainerStarted","Data":"97d907d69716b1deccee274b503d0f650b9f74f0a23fa94c2882a5ba5ef035dc"} Apr 16 17:20:57.754664 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:20:57.754489 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7" Apr 16 17:20:57.755833 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:20:57.755806 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7" podUID="969ea101-644f-4d51-9566-7e8cf20aa33f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Apr 16 17:20:57.770881 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:20:57.770829 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7" podStartSLOduration=5.998928843 podStartE2EDuration="26.770814441s" podCreationTimestamp="2026-04-16 17:20:31 +0000 UTC" firstStartedPulling="2026-04-16 17:20:36.665049051 +0000 UTC m=+1975.726198563" lastFinishedPulling="2026-04-16 17:20:57.436934653 +0000 UTC m=+1996.498084161" observedRunningTime="2026-04-16 17:20:57.767753946 +0000 UTC m=+1996.828903476" watchObservedRunningTime="2026-04-16 17:20:57.770814441 +0000 UTC m=+1996.831964033" Apr 16 17:20:58.758708 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:20:58.758664 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7" podUID="969ea101-644f-4d51-9566-7e8cf20aa33f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Apr 16 17:21:08.758868 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:21:08.758828 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7" podUID="969ea101-644f-4d51-9566-7e8cf20aa33f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Apr 16 17:21:18.758907 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:21:18.758864 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7" podUID="969ea101-644f-4d51-9566-7e8cf20aa33f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Apr 16 17:21:28.759396 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:21:28.759352 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7" podUID="969ea101-644f-4d51-9566-7e8cf20aa33f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Apr 16 17:21:38.758847 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:21:38.758793 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7" podUID="969ea101-644f-4d51-9566-7e8cf20aa33f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Apr 16 17:21:48.758965 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:21:48.758873 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7" podUID="969ea101-644f-4d51-9566-7e8cf20aa33f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Apr 16 17:21:58.759358 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:21:58.759307 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7" podUID="969ea101-644f-4d51-9566-7e8cf20aa33f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Apr 16 17:22:02.588943 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:22:02.588898 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7" podUID="969ea101-644f-4d51-9566-7e8cf20aa33f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Apr 16 17:22:12.590290 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:22:12.590228 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7" Apr 16 17:23:41.527057 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:23:41.527019 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb"] Apr 16 17:23:41.531002 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:23:41.530980 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb" Apr 16 17:23:41.540091 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:23:41.540063 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb"] Apr 16 17:23:41.629951 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:23:41.629917 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1117ed58-9128-467f-93b0-8c8777987c99-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb\" (UID: \"1117ed58-9128-467f-93b0-8c8777987c99\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb" Apr 16 17:23:41.730771 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:23:41.730734 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1117ed58-9128-467f-93b0-8c8777987c99-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb\" (UID: \"1117ed58-9128-467f-93b0-8c8777987c99\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb" Apr 16 17:23:41.731135 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:23:41.731115 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1117ed58-9128-467f-93b0-8c8777987c99-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb\" (UID: \"1117ed58-9128-467f-93b0-8c8777987c99\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb" Apr 16 17:23:41.842812 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:23:41.842783 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb" Apr 16 17:23:41.968437 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:23:41.968412 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb"] Apr 16 17:23:41.970322 ip-10-0-142-37 kubenswrapper[2576]: W0416 17:23:41.970292 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1117ed58_9128_467f_93b0_8c8777987c99.slice/crio-266e2af533ca3c1afabe685678ba0d465b29057dc481af09e68abfa7891f101f WatchSource:0}: Error finding container 266e2af533ca3c1afabe685678ba0d465b29057dc481af09e68abfa7891f101f: Status 404 returned error can't find the container with id 266e2af533ca3c1afabe685678ba0d465b29057dc481af09e68abfa7891f101f Apr 16 17:23:42.347572 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:23:42.347535 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb" event={"ID":"1117ed58-9128-467f-93b0-8c8777987c99","Type":"ContainerStarted","Data":"08834bc2f98999765c3571e3bbc0e79921f504fb27c50fbfc35c6b24d8bbf45d"} Apr 16 17:23:42.347572 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:23:42.347570 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb" event={"ID":"1117ed58-9128-467f-93b0-8c8777987c99","Type":"ContainerStarted","Data":"266e2af533ca3c1afabe685678ba0d465b29057dc481af09e68abfa7891f101f"} Apr 16 17:23:46.365233 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:23:46.365198 2576 generic.go:358] "Generic (PLEG): container finished" podID="1117ed58-9128-467f-93b0-8c8777987c99" containerID="08834bc2f98999765c3571e3bbc0e79921f504fb27c50fbfc35c6b24d8bbf45d" exitCode=0 Apr 16 17:23:46.365617 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:23:46.365276 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb" event={"ID":"1117ed58-9128-467f-93b0-8c8777987c99","Type":"ContainerDied","Data":"08834bc2f98999765c3571e3bbc0e79921f504fb27c50fbfc35c6b24d8bbf45d"} Apr 16 17:23:47.371459 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:23:47.371421 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb" event={"ID":"1117ed58-9128-467f-93b0-8c8777987c99","Type":"ContainerStarted","Data":"787041a67c3ef550fbfc21f6c9064b52ee0d0a2a655ae350cbf02f805a5f8b23"} Apr 16 17:23:47.371914 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:23:47.371637 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb" Apr 16 17:23:47.387141 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:23:47.387097 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb" podStartSLOduration=6.387059189 podStartE2EDuration="6.387059189s" podCreationTimestamp="2026-04-16 17:23:41 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 17:23:47.385620545 +0000 UTC m=+2166.446770100" watchObservedRunningTime="2026-04-16 17:23:47.387059189 +0000 UTC m=+2166.448208721" Apr 16 17:24:18.377464 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:24:18.377417 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb" podUID="1117ed58-9128-467f-93b0-8c8777987c99" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.39:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.39:8080: connect: connection refused" Apr 16 17:24:28.376322 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:24:28.376278 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb" podUID="1117ed58-9128-467f-93b0-8c8777987c99" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.39:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.39:8080: connect: connection refused" Apr 16 17:24:38.376032 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:24:38.375990 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb" podUID="1117ed58-9128-467f-93b0-8c8777987c99" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.39:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.39:8080: connect: connection refused" Apr 16 17:24:48.376008 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:24:48.375921 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb" podUID="1117ed58-9128-467f-93b0-8c8777987c99" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.39:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.39:8080: connect: connection refused" Apr 16 17:24:58.375866 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:24:58.375820 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb" podUID="1117ed58-9128-467f-93b0-8c8777987c99" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.39:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.39:8080: connect: connection refused" Apr 16 17:25:01.596457 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:25:01.596425 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb" Apr 16 17:28:24.292464 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:28:24.292387 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-66fb7c44cc-gzszm"] Apr 16 17:28:24.296503 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:28:24.296482 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-66fb7c44cc-gzszm" Apr 16 17:28:24.301826 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:28:24.301790 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-66fb7c44cc-gzszm"] Apr 16 17:28:24.395463 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:28:24.395426 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/907fcecd-6d8b-4ac3-9eb5-92055fe5f000-kserve-provision-location\") pod \"isvc-sklearn-predictor-66fb7c44cc-gzszm\" (UID: \"907fcecd-6d8b-4ac3-9eb5-92055fe5f000\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-66fb7c44cc-gzszm" Apr 16 17:28:24.496384 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:28:24.496350 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/907fcecd-6d8b-4ac3-9eb5-92055fe5f000-kserve-provision-location\") pod \"isvc-sklearn-predictor-66fb7c44cc-gzszm\" (UID: \"907fcecd-6d8b-4ac3-9eb5-92055fe5f000\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-66fb7c44cc-gzszm" Apr 16 17:28:24.496757 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:28:24.496734 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/907fcecd-6d8b-4ac3-9eb5-92055fe5f000-kserve-provision-location\") pod \"isvc-sklearn-predictor-66fb7c44cc-gzszm\" (UID: \"907fcecd-6d8b-4ac3-9eb5-92055fe5f000\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-66fb7c44cc-gzszm" Apr 16 17:28:24.608380 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:28:24.608286 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-66fb7c44cc-gzszm" Apr 16 17:28:24.729881 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:28:24.729856 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-66fb7c44cc-gzszm"] Apr 16 17:28:24.731552 ip-10-0-142-37 kubenswrapper[2576]: W0416 17:28:24.731511 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod907fcecd_6d8b_4ac3_9eb5_92055fe5f000.slice/crio-8d8c05524f752c5236a5b7e2777a717f307e80ed3816c5ece8b6cb69f6a4f7a9 WatchSource:0}: Error finding container 8d8c05524f752c5236a5b7e2777a717f307e80ed3816c5ece8b6cb69f6a4f7a9: Status 404 returned error can't find the container with id 8d8c05524f752c5236a5b7e2777a717f307e80ed3816c5ece8b6cb69f6a4f7a9 Apr 16 17:28:24.733392 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:28:24.733376 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Apr 16 17:28:25.403998 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:28:25.403959 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-66fb7c44cc-gzszm" event={"ID":"907fcecd-6d8b-4ac3-9eb5-92055fe5f000","Type":"ContainerStarted","Data":"c27a0607ed8cdd377b8a51f0681537233528e6885cbd7a39528aaef55e474d65"} Apr 16 17:28:25.404404 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:28:25.404005 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-66fb7c44cc-gzszm" event={"ID":"907fcecd-6d8b-4ac3-9eb5-92055fe5f000","Type":"ContainerStarted","Data":"8d8c05524f752c5236a5b7e2777a717f307e80ed3816c5ece8b6cb69f6a4f7a9"} Apr 16 17:28:29.419329 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:28:29.419294 2576 generic.go:358] "Generic (PLEG): container finished" podID="907fcecd-6d8b-4ac3-9eb5-92055fe5f000" containerID="c27a0607ed8cdd377b8a51f0681537233528e6885cbd7a39528aaef55e474d65" exitCode=0 Apr 16 17:28:29.419698 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:28:29.419363 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-66fb7c44cc-gzszm" event={"ID":"907fcecd-6d8b-4ac3-9eb5-92055fe5f000","Type":"ContainerDied","Data":"c27a0607ed8cdd377b8a51f0681537233528e6885cbd7a39528aaef55e474d65"} Apr 16 17:28:30.425983 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:28:30.425936 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-66fb7c44cc-gzszm" event={"ID":"907fcecd-6d8b-4ac3-9eb5-92055fe5f000","Type":"ContainerStarted","Data":"3a3cb2b2661bad9dce075324326b990e202cca68b708b4cdf40cf0d141f3f7ef"} Apr 16 17:28:30.426468 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:28:30.426370 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-66fb7c44cc-gzszm" Apr 16 17:28:30.427797 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:28:30.427767 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-66fb7c44cc-gzszm" podUID="907fcecd-6d8b-4ac3-9eb5-92055fe5f000" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Apr 16 17:28:30.441739 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:28:30.441693 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-66fb7c44cc-gzszm" podStartSLOduration=6.441677743 podStartE2EDuration="6.441677743s" podCreationTimestamp="2026-04-16 17:28:24 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 17:28:30.440393147 +0000 UTC m=+2449.501542678" watchObservedRunningTime="2026-04-16 17:28:30.441677743 +0000 UTC m=+2449.502827274" Apr 16 17:28:31.430920 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:28:31.430874 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-66fb7c44cc-gzszm" podUID="907fcecd-6d8b-4ac3-9eb5-92055fe5f000" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Apr 16 17:28:41.430828 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:28:41.430786 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-66fb7c44cc-gzszm" podUID="907fcecd-6d8b-4ac3-9eb5-92055fe5f000" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Apr 16 17:28:51.431690 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:28:51.431641 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-66fb7c44cc-gzszm" podUID="907fcecd-6d8b-4ac3-9eb5-92055fe5f000" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Apr 16 17:29:01.430952 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:29:01.430905 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-66fb7c44cc-gzszm" podUID="907fcecd-6d8b-4ac3-9eb5-92055fe5f000" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Apr 16 17:29:11.431642 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:29:11.431599 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-66fb7c44cc-gzszm" podUID="907fcecd-6d8b-4ac3-9eb5-92055fe5f000" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Apr 16 17:29:21.431282 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:29:21.431174 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-66fb7c44cc-gzszm" podUID="907fcecd-6d8b-4ac3-9eb5-92055fe5f000" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Apr 16 17:29:31.431632 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:29:31.431586 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-66fb7c44cc-gzszm" podUID="907fcecd-6d8b-4ac3-9eb5-92055fe5f000" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Apr 16 17:29:41.432418 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:29:41.432382 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-66fb7c44cc-gzszm" Apr 16 17:30:34.644815 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:30:34.644775 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf744c55f-54m8x"] Apr 16 17:30:34.648584 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:30:34.648566 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf744c55f-54m8x" Apr 16 17:30:34.654036 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:30:34.654007 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf744c55f-54m8x"] Apr 16 17:30:34.763957 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:30:34.763918 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04655ab7-174e-4cf3-b5cf-52106e407e9c-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-cf744c55f-54m8x\" (UID: \"04655ab7-174e-4cf3-b5cf-52106e407e9c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf744c55f-54m8x" Apr 16 17:30:34.864627 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:30:34.864591 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04655ab7-174e-4cf3-b5cf-52106e407e9c-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-cf744c55f-54m8x\" (UID: \"04655ab7-174e-4cf3-b5cf-52106e407e9c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf744c55f-54m8x" Apr 16 17:30:34.864971 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:30:34.864951 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04655ab7-174e-4cf3-b5cf-52106e407e9c-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-cf744c55f-54m8x\" (UID: \"04655ab7-174e-4cf3-b5cf-52106e407e9c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf744c55f-54m8x" Apr 16 17:30:34.960210 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:30:34.960124 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf744c55f-54m8x" Apr 16 17:30:35.088035 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:30:35.087999 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf744c55f-54m8x"] Apr 16 17:30:35.089898 ip-10-0-142-37 kubenswrapper[2576]: W0416 17:30:35.089867 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04655ab7_174e_4cf3_b5cf_52106e407e9c.slice/crio-b1d119e70306d07c61b0e5f6483f6f18ba063930e4d23e6144295be375f9bd7d WatchSource:0}: Error finding container b1d119e70306d07c61b0e5f6483f6f18ba063930e4d23e6144295be375f9bd7d: Status 404 returned error can't find the container with id b1d119e70306d07c61b0e5f6483f6f18ba063930e4d23e6144295be375f9bd7d Apr 16 17:30:35.895979 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:30:35.895945 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf744c55f-54m8x" event={"ID":"04655ab7-174e-4cf3-b5cf-52106e407e9c","Type":"ContainerStarted","Data":"37804a8d7db223c271e90f6d6f93517b1ed930d62f74d84d759a87d3262d6772"} Apr 16 17:30:35.895979 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:30:35.895980 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf744c55f-54m8x" event={"ID":"04655ab7-174e-4cf3-b5cf-52106e407e9c","Type":"ContainerStarted","Data":"b1d119e70306d07c61b0e5f6483f6f18ba063930e4d23e6144295be375f9bd7d"} Apr 16 17:30:42.924292 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:30:42.924230 2576 generic.go:358] "Generic (PLEG): container finished" podID="04655ab7-174e-4cf3-b5cf-52106e407e9c" containerID="37804a8d7db223c271e90f6d6f93517b1ed930d62f74d84d759a87d3262d6772" exitCode=0 Apr 16 17:30:42.924678 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:30:42.924308 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf744c55f-54m8x" event={"ID":"04655ab7-174e-4cf3-b5cf-52106e407e9c","Type":"ContainerDied","Data":"37804a8d7db223c271e90f6d6f93517b1ed930d62f74d84d759a87d3262d6772"} Apr 16 17:30:43.933043 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:30:43.933010 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf744c55f-54m8x" event={"ID":"04655ab7-174e-4cf3-b5cf-52106e407e9c","Type":"ContainerStarted","Data":"7997f050347657743e300852bc48dbcca99ab8428b2ab1d89bcd1304e2725a34"} Apr 16 17:30:43.933483 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:30:43.933368 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf744c55f-54m8x" Apr 16 17:30:43.934753 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:30:43.934726 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf744c55f-54m8x" podUID="04655ab7-174e-4cf3-b5cf-52106e407e9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused" Apr 16 17:30:43.949991 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:30:43.949946 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf744c55f-54m8x" podStartSLOduration=9.949934763 podStartE2EDuration="9.949934763s" podCreationTimestamp="2026-04-16 17:30:34 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 17:30:43.947435242 +0000 UTC m=+2583.008584772" watchObservedRunningTime="2026-04-16 17:30:43.949934763 +0000 UTC m=+2583.011084293" Apr 16 17:30:44.938343 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:30:44.938294 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf744c55f-54m8x" podUID="04655ab7-174e-4cf3-b5cf-52106e407e9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused" Apr 16 17:30:54.938938 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:30:54.938886 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf744c55f-54m8x" podUID="04655ab7-174e-4cf3-b5cf-52106e407e9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused" Apr 16 17:31:04.939134 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:31:04.939101 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf744c55f-54m8x" Apr 16 17:49:40.563645 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:40.563605 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:49:40.583456 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:40.583425 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:49:40.591471 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:40.591452 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:49:40.624362 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:40.624338 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:49:40.630588 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:40.630567 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:49:40.643268 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:40.643231 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:49:40.649506 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:40.649488 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:49:40.662869 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:40.662829 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:49:40.668473 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:40.668456 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:49:40.721499 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:40.721475 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:49:40.729564 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:40.729540 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:49:40.843174 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:40.843094 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:49:40.849639 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:40.849605 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:49:40.860445 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:40.860419 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:49:40.868852 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:40.868814 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:49:40.902477 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:40.902450 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:49:40.911060 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:40.911038 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:49:40.919783 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:40.919757 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:49:40.931624 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:40.931602 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:49:40.940110 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:40.940086 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:49:40.949404 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:40.949386 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:49:40.960015 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:40.959988 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:49:40.968578 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:40.968555 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:49:40.980941 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:40.980921 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:49:40.988257 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:40.988221 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:49:42.831084 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:42.831053 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:49:42.839940 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:42.839916 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:49:42.847829 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:42.847797 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:49:42.882091 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:42.882061 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:49:42.889027 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:42.889005 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:49:42.902345 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:42.902322 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:49:42.909284 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:42.909261 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:49:42.922831 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:42.922809 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:49:42.929651 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:42.929633 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:49:42.981967 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:42.981936 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:49:42.990683 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:42.990662 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:49:43.079753 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:43.079718 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:49:43.087018 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:43.086924 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:49:43.098435 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:43.098409 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:49:43.106392 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:43.106372 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:49:43.142927 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:43.142898 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:49:43.151975 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:43.151953 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:49:43.160493 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:43.160475 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:49:43.172604 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:43.172587 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:49:43.185765 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:43.185742 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:49:43.194693 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:43.194670 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:49:43.206551 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:43.206526 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:49:43.215281 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:43.215234 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:49:43.230515 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:43.230494 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:49:43.238150 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:43.238133 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:49:44.999671 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:44.999640 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:49:45.008786 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:45.008760 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:49:45.017552 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:45.017526 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:49:45.053289 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:45.053261 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:49:45.060074 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:45.060051 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:49:45.073042 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:45.073018 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:49:45.080195 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:45.080168 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:49:45.092641 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:45.092621 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:49:45.099685 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:45.099660 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:49:45.150974 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:45.150949 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:49:45.159461 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:45.159439 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:49:45.253808 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:45.253731 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:49:45.259297 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:45.259277 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:49:45.270923 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:45.270897 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:49:45.279312 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:45.279288 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:49:45.313849 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:45.313812 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:49:45.322412 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:45.322387 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:49:45.331215 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:45.331192 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:49:45.343169 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:45.343148 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:49:45.352148 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:45.352126 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:49:45.360721 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:45.360669 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:49:45.372135 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:45.372115 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:49:45.380975 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:45.380956 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:49:45.393403 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:45.393380 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:49:45.403343 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:45.403327 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:49:47.155071 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:47.155046 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:49:47.164787 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:47.164763 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:49:47.173296 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:47.173270 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:49:47.209734 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:47.209701 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:49:47.216728 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:47.216708 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:49:47.230745 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:47.230712 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:49:47.237839 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:47.237815 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:49:47.251290 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:47.251230 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:49:47.258354 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:47.258332 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:49:47.312308 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:47.312281 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:49:47.321059 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:47.321037 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:49:47.410906 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:47.410818 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:49:47.415970 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:47.415940 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:49:47.428617 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:47.428591 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:49:47.437742 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:47.437715 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:49:47.481665 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:47.481625 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:49:47.492455 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:47.492429 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:49:47.501188 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:47.501165 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:49:47.513053 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:47.513025 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:49:47.521734 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:47.521712 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:49:47.530104 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:47.530079 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:49:47.541145 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:47.541115 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:49:47.549727 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:47.549704 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:49:47.562371 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:47.562351 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:49:47.570215 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:47.570193 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:49:49.340203 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:49.340174 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:49:49.348820 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:49.348794 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:49:49.356942 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:49.356922 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:49:49.392453 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:49.392424 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:49:49.399210 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:49.399181 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:49:49.412173 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:49.412147 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:49:49.418847 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:49.418828 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:49:49.431700 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:49.431666 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:49:49.438439 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:49.438423 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:49:49.491847 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:49.491819 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:49:49.497102 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:49.497083 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:49:49.582552 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:49.582527 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:49:49.587063 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:49.587045 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:49:49.597974 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:49.597908 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:49:49.606108 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:49.606090 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:49:49.638910 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:49.638873 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:49:49.647442 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:49.647422 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:49:49.655590 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:49.655568 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:49:49.667342 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:49.667311 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:49:49.676104 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:49.676085 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:49:49.691349 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:49.691307 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:49:49.703075 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:49.703055 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:49:49.711689 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:49.711667 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:49:49.723679 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:49.723653 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:49:49.731968 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:49.731944 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:49:51.560365 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:51.560338 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:49:51.569426 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:51.569400 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:49:51.578086 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:51.578067 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:49:51.617461 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:51.617432 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:49:51.624410 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:51.624384 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:49:51.638898 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:51.638881 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:49:51.646440 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:51.646422 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:49:51.660173 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:51.660153 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:49:51.667610 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:51.667586 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:49:51.722564 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:51.722541 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:49:51.731206 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:51.731186 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:49:51.820220 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:51.820142 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:49:51.826912 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:51.826886 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:49:51.843596 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:51.843572 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:49:51.852094 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:51.852071 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:49:51.891261 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:51.891216 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:49:51.900388 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:51.900359 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:49:51.909029 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:51.909010 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:49:51.921822 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:51.921799 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:49:51.931571 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:51.931553 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:49:51.939934 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:51.939915 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:49:51.952170 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:51.952129 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:49:51.961625 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:51.961605 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:49:51.974516 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:51.974496 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:49:51.982629 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:51.982610 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:49:53.924939 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:53.924892 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:49:53.935183 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:53.935153 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:49:53.954113 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:53.954084 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:49:53.997337 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:53.997299 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:49:54.006029 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:54.005987 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:49:54.021859 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:54.021836 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:49:54.030832 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:54.030809 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:49:54.046053 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:54.046027 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:49:54.053448 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:54.053418 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:49:54.112471 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:54.112440 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:49:54.124659 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:54.124638 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:49:54.222676 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:54.222590 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:49:54.230156 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:54.230136 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:49:54.244616 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:54.244592 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:49:54.258395 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:54.258374 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:49:54.295748 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:54.295714 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:49:54.304734 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:54.304712 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:49:54.313620 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:54.313601 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:49:54.326107 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:54.326088 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:49:54.335556 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:54.335538 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:49:54.346058 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:54.346039 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:49:54.359536 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:54.359515 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:49:54.369125 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:54.369103 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:49:54.381138 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:54.381117 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:49:54.388999 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:54.388973 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:49:56.274432 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:56.274393 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:49:56.285001 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:56.284975 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:49:56.296439 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:56.296418 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:49:56.335746 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:56.335726 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:49:56.345160 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:56.345136 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:49:56.360412 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:56.360394 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:49:56.371892 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:56.371872 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:49:56.386711 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:56.386689 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:49:56.394617 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:56.394597 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:49:56.455379 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:56.455348 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:49:56.465211 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:56.465187 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:49:56.560416 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:56.560386 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:49:56.567684 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:56.567654 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:49:56.582041 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:56.582012 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:49:56.592359 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:56.592334 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:49:56.632908 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:56.632883 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:49:56.642879 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:56.642856 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:49:56.656855 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:56.656827 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:49:56.673611 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:56.673593 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:49:56.688565 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:56.688544 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:49:56.699965 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:56.699946 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:49:56.719144 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:56.719029 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:49:56.732940 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:56.732909 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:49:56.749480 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:56.749456 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:49:56.758765 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:56.758738 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:49:58.591993 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:58.591955 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:49:58.611417 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:58.611381 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:49:58.626399 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:58.626372 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:49:58.669611 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:58.669583 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:49:58.678967 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:58.678942 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:49:58.695746 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:58.695724 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:49:58.705510 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:58.705492 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:49:58.719099 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:58.719065 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:49:58.725993 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:58.725972 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:49:58.786883 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:58.786850 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:49:58.796355 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:58.796329 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:49:58.893730 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:58.893648 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:49:58.898765 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:58.898745 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:49:58.910725 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:58.910701 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:49:58.919857 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:58.919840 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:49:58.958402 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:58.958377 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:49:58.967929 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:58.967906 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:49:58.976829 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:58.976807 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:49:58.989486 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:58.989466 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:49:58.998621 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:58.998597 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:49:59.029528 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:59.029504 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:49:59.059350 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:59.059312 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:49:59.069167 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:59.069137 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:49:59.085375 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:59.085354 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:49:59.095571 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:49:59.095544 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:50:00.961801 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:00.961769 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:50:00.972303 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:00.972282 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:50:00.981336 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:00.981305 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:50:01.021079 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:01.021051 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:50:01.029930 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:01.029909 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:50:01.047233 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:01.047212 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:50:01.053948 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:01.053932 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:50:01.080696 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:01.080675 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:50:01.090746 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:01.090723 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:50:01.166271 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:01.166223 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:50:01.179302 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:01.179279 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:50:01.296729 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:01.296700 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:50:01.307082 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:01.307064 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:50:01.320428 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:01.320395 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:50:01.331069 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:01.331046 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:50:01.372418 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:01.372389 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:50:01.385632 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:01.385603 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:50:01.395735 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:01.395707 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:50:01.408593 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:01.408568 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:50:01.420653 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:01.420634 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:50:01.430429 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:01.430410 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:50:01.442859 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:01.442843 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:50:01.454389 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:01.454366 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:50:01.469440 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:01.469414 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:50:01.477524 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:01.477507 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:50:03.329178 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:03.329148 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:50:03.338700 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:03.338677 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:50:03.347597 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:03.347575 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:50:03.382141 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:03.382109 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:50:03.389351 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:03.389327 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:50:03.403290 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:03.403261 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:50:03.412046 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:03.412022 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:50:03.426531 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:03.426504 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:50:03.433746 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:03.433728 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:50:03.490910 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:03.490871 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:50:03.500122 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:03.500098 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:50:03.608861 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:03.608761 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:50:03.613452 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:03.613430 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:50:03.625430 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:03.625407 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:50:03.635544 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:03.635523 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:50:03.675395 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:03.675368 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:50:03.684062 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:03.684041 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:50:03.692653 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:03.692635 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:50:03.707189 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:03.707170 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:50:03.716592 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:03.716569 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:50:03.725592 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:03.725564 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:50:03.737332 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:03.737301 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:50:03.748236 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:03.748209 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:50:03.764548 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:03.764520 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:50:03.773898 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:03.773880 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:50:05.581351 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:05.581322 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:50:05.591730 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:05.591703 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:50:05.600732 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:05.600707 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:50:05.638914 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:05.638883 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:50:05.647799 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:05.647772 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:50:05.662580 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:05.662546 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:50:05.671104 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:05.671068 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:50:05.685707 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:05.685677 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:50:05.694035 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:05.694014 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:50:05.749223 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:05.749185 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:50:05.758381 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:05.758360 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:50:05.850219 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:05.850127 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:50:05.855570 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:05.855541 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:50:05.868156 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:05.868129 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:50:05.876922 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:05.876900 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:50:05.912025 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:05.911994 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:50:05.922155 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:05.922098 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:50:05.932118 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:05.932090 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:50:05.944402 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:05.944378 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:50:05.954713 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:05.954683 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:50:05.963970 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:05.963934 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:50:05.976131 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:05.976090 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:50:05.985208 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:05.985181 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:50:05.998402 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:05.998374 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:50:06.006945 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:06.006916 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:50:07.808678 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:07.808646 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:50:07.818113 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:07.818092 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:50:07.827063 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:07.827044 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:50:07.862881 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:07.862848 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:50:07.871912 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:07.871890 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:50:07.885113 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:07.885092 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:50:07.907170 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:07.907138 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:50:07.925049 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:07.925031 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:50:07.936219 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:07.936193 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:50:07.992026 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:07.992002 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:50:08.000940 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:08.000917 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:50:08.099701 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:08.099612 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:50:08.104715 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:08.104693 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:50:08.116292 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:08.116268 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:50:08.125223 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:08.125192 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:50:08.159720 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:08.159695 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:50:08.168795 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:08.168765 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:50:08.179101 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:08.179073 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:50:08.191450 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:08.191424 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:50:08.201202 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:08.201179 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:50:08.210258 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:08.210218 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:50:08.221875 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:08.221854 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:50:08.230607 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:08.230583 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:50:08.242787 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:08.242766 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:50:08.251793 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:08.251773 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:50:10.080677 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:10.080643 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:50:10.089294 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:10.089272 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:50:10.097615 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:10.097595 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:50:10.132792 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:10.132761 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:50:10.139815 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:10.139788 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:50:10.153591 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:10.153565 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:50:10.160438 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:10.160420 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:50:10.173120 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:10.173100 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:50:10.179953 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:10.179925 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:50:10.233728 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:10.233690 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:50:10.242189 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:10.242165 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:50:10.326795 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:10.326763 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:50:10.331449 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:10.331378 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:50:10.342140 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:10.342116 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:50:10.350596 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:10.350575 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:50:10.383719 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:10.383679 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:50:10.392462 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:10.392433 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:50:10.400626 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:10.400601 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:50:10.432507 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:10.432459 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:50:10.441263 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:10.441214 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:50:10.449843 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:10.449818 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:50:10.460590 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:10.460565 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:50:10.469040 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:10.469005 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:50:10.481033 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:10.481006 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:50:10.488319 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:10.488297 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:50:12.264393 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:12.264364 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:50:12.277056 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:12.277029 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:50:12.290619 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:12.290592 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:50:12.336592 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:12.336564 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:50:12.345934 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:12.345911 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:50:12.360283 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:12.360237 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:50:12.372356 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:12.372319 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:50:12.387594 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:12.387562 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:50:12.397508 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:12.397482 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:50:12.470367 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:12.470334 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:50:12.482614 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:12.482592 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:50:12.610381 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:12.610351 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:50:12.615605 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:12.615585 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:50:12.629660 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:12.629633 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:50:12.645045 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:12.645014 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:50:12.694357 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:12.694323 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:50:12.703870 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:12.703844 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:50:12.725084 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:12.725047 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:50:12.740777 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:12.740748 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:50:12.751538 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:12.751506 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:50:12.766298 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:12.766260 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:50:12.787233 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:12.787201 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:50:12.800372 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:12.800349 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:50:12.814953 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:12.814928 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:50:12.827983 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:12.827959 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:50:14.681113 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:14.681086 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:50:14.691130 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:14.691104 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:50:14.699916 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:14.699895 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:50:14.734668 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:14.734641 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:50:14.741740 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:14.741717 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:50:14.756403 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:14.756374 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:50:14.763674 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:14.763648 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:50:14.777151 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:14.777127 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:50:14.784942 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:14.784922 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:50:14.838957 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:14.838926 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:50:14.847476 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:14.847452 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:50:14.935087 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:14.935003 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:50:14.943130 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:14.943107 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:50:14.955333 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:14.955311 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:50:14.964413 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:14.964394 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:50:15.002673 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:15.002643 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:50:15.011437 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:15.011413 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:50:15.019657 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:15.019637 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:50:15.032024 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:15.032000 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:50:15.040809 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:15.040790 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:50:15.049482 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:15.049463 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:50:15.060505 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:15.060485 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:50:15.069739 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:15.069713 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:50:15.081807 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:15.081769 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:50:15.089461 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:15.089441 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:50:16.918265 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:16.918173 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:50:16.927674 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:16.927647 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:50:16.936419 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:16.936396 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:50:16.970002 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:16.969976 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:50:16.977017 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:16.976978 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:50:16.990808 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:16.990775 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:50:16.999452 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:16.999431 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:50:17.016114 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:17.016089 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:50:17.023711 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:17.023688 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:50:17.079917 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:17.079888 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:50:17.089235 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:17.089212 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:50:17.179081 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:17.178994 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:50:17.184807 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:17.184788 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:50:17.196575 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:17.196558 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:50:17.206262 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:17.206222 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:50:17.243472 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:17.243438 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:50:17.253280 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:17.253230 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:50:17.262691 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:17.262646 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:50:17.274801 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:17.274772 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:50:17.284164 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:17.284137 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:50:17.292946 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:17.292927 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:50:17.309508 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:17.309472 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:50:17.318664 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:17.318638 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:50:17.331481 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:17.331460 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:50:17.339933 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:17.339915 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:50:19.097669 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:19.097639 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:50:19.106991 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:19.106968 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:50:19.115451 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:19.115427 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:50:19.151866 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:19.151838 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:50:19.159925 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:19.159896 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:50:19.173492 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:19.173467 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:50:19.180578 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:19.180541 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:50:19.193286 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:19.193233 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:50:19.200591 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:19.200565 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:50:19.254912 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:19.254880 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:50:19.264927 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:19.264903 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:50:19.359226 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:19.359140 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:50:19.363774 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:19.363745 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:50:19.375098 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:19.375077 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:50:19.383468 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:19.383448 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:50:19.416718 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:19.416689 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:50:19.425876 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:19.425850 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:50:19.434396 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:19.434361 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:50:19.448014 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:19.447990 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:50:19.456678 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:19.456653 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:50:19.465504 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:19.465477 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:50:19.476980 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:19.476955 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:50:19.485728 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:19.485706 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:50:19.497524 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:19.497506 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:50:19.505059 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:19.505037 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:50:21.269581 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:21.269548 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:50:21.278762 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:21.278742 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:50:21.287992 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:21.287969 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:50:21.323540 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:21.323513 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:50:21.331136 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:21.331107 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:50:21.347236 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:21.347205 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:50:21.355363 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:21.355342 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:50:21.368961 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:21.368926 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:50:21.376055 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:21.376011 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:50:21.441594 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:21.441567 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:50:21.453050 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:21.452990 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:50:21.544981 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:21.544950 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:50:21.549898 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:21.549873 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:50:21.563318 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:21.563290 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:50:21.575313 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:21.575286 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:50:21.612095 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:21.612061 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:50:21.622677 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:21.622643 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:50:21.632391 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:21.632365 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:50:21.645676 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:21.645651 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:50:21.655885 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:21.655855 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:50:21.665539 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:21.665515 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:50:21.677698 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:21.677678 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:50:21.686258 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:21.686202 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:50:21.698980 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:21.698957 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:50:21.706958 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:21.706935 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:50:23.493673 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:23.493641 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:50:23.504516 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:23.504488 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:50:23.513712 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:23.513684 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:50:23.549385 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:23.549354 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:50:23.556487 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:23.556462 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:50:23.572489 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:23.572458 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:50:23.579633 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:23.579574 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:50:23.596361 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:23.596334 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:50:23.602660 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:23.602614 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:50:23.653985 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:23.653948 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:50:23.662464 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:23.662436 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:50:23.750145 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:23.750051 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:50:23.755997 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:23.755976 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:50:23.768169 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:23.768143 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:50:23.776930 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:23.776905 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:50:23.812199 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:23.812168 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:50:23.820953 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:23.820928 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:50:23.829546 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:23.829523 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:50:23.841946 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:23.841923 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:50:23.853168 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:23.853145 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:50:23.862205 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:23.862187 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:50:23.873748 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:23.873710 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:50:23.882069 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:23.882052 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:50:23.894418 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:23.894378 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:50:23.903456 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:23.903429 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:50:25.700989 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:25.700959 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:50:25.710348 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:25.710321 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:50:25.720000 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:25.719976 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:50:25.759316 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:25.759286 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:50:25.766809 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:25.766780 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:50:25.779846 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:25.779823 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:50:25.787401 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:25.787379 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:50:25.802507 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:25.802478 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:50:25.809164 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:25.809139 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:50:25.865838 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:25.865803 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:50:25.874516 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:25.874491 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:50:25.958654 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:25.958569 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:50:25.963271 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:25.963227 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:50:25.974997 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:25.974976 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:50:25.983851 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:25.983830 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:50:26.020123 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:26.020092 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:50:26.029732 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:26.029692 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:50:26.038898 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:26.038867 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:50:26.050464 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:26.050434 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:50:26.059617 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:26.059593 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:50:26.068682 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:26.068651 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:50:26.080350 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:26.080315 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:50:26.091850 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:26.091830 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:50:26.104154 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:26.104122 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:50:26.114615 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:26.114585 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:50:27.894011 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:27.893985 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:50:27.904389 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:27.904367 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:50:27.912919 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:27.912899 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:50:27.951878 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:27.951846 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:50:27.959715 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:27.959691 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:50:27.983064 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:27.983036 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:50:27.990458 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:27.990434 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:50:28.004230 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:28.004199 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:50:28.011037 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:28.011012 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:50:28.064380 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:28.064344 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:50:28.073305 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:28.073264 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:50:28.159031 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:28.158947 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:50:28.165222 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:28.165197 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:50:28.176296 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:28.176271 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:50:28.184948 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:28.184923 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:50:28.225905 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:28.225877 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:50:28.235987 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:28.235962 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:50:28.247677 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:28.247647 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:50:28.260042 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:28.260012 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:50:28.269668 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:28.269650 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:50:28.278486 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:28.278462 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:50:28.290233 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:28.290210 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:50:28.302200 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:28.302177 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:50:28.313996 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:28.313971 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:50:28.321885 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:28.321862 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:50:30.082144 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:30.082113 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:50:30.091720 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:30.091696 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:50:30.103381 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:30.103356 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:50:30.139594 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:30.139561 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:50:30.147500 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:30.147463 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:50:30.160801 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:30.160777 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:50:30.170876 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:30.170850 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:50:30.186215 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:30.186191 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:50:30.195286 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:30.195263 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:50:30.249306 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:30.249278 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:50:30.258390 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:30.258366 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:50:30.344459 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:30.344378 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:50:30.349065 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:30.349044 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:50:30.360660 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:30.360632 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:50:30.368952 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:30.368925 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:50:30.403663 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:30.403634 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:50:30.412512 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:30.412487 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:50:30.421184 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:30.421159 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:50:30.432982 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:30.432950 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:50:30.441914 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:30.441889 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:50:30.450604 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:30.450573 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:50:30.461938 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:30.461911 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:50:30.470068 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:30.470046 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:50:30.482336 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:30.482311 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:50:30.489968 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:30.489946 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:50:32.244310 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:32.244278 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:50:32.253038 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:32.253010 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:50:32.260599 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:32.260566 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:50:32.293337 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:32.293299 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:50:32.306352 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:32.306321 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:50:32.319314 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:32.319282 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:50:32.326132 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:32.326109 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:50:32.339528 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:32.339492 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:50:32.346754 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:32.346725 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:50:32.406452 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:32.406424 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:50:32.416380 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:32.416354 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:50:32.512792 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:32.512704 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:50:32.518226 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:32.518195 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:50:32.530619 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:32.530568 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:50:32.539705 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:32.539680 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:50:32.575908 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:32.575880 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:50:32.584768 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:32.584741 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:50:32.592871 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:32.592848 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:50:32.607357 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:32.607338 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:50:32.615838 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:32.615813 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:50:32.624441 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:32.624424 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:50:32.635903 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:32.635878 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:50:32.643964 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:32.643945 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:50:32.655926 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:32.655906 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:50:32.663762 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:32.663739 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:50:34.412988 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:34.412956 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:50:34.421874 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:34.421849 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:50:34.430515 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:34.430493 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:50:34.464468 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:34.464432 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:50:34.474430 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:34.474406 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:50:34.487846 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:34.487809 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:50:34.495340 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:34.495313 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:50:34.509404 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:34.509363 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:50:34.516233 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:34.516213 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:50:34.569358 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:34.569325 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:50:34.578490 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:34.578465 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:50:34.671075 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:34.670988 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:50:34.675960 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:34.675938 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:50:34.687838 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:34.687806 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:50:34.697135 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:34.697106 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:50:34.736824 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:34.736790 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:50:34.744468 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:34.744441 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:50:34.753623 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:34.753605 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:50:34.765690 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:34.765667 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:50:34.774794 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:34.774773 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:50:34.784444 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:34.784419 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:50:34.796389 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:34.796371 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:50:34.805308 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:34.805285 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:50:34.817889 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:34.817865 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:50:34.826259 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:34.826223 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:50:36.614210 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:36.614177 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:50:36.623082 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:36.623058 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:50:36.632083 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:36.632052 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:50:36.670863 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:36.670837 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:50:36.677882 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:36.677855 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:50:36.691465 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:36.691443 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:50:36.697969 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:36.697944 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:50:36.713820 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:36.713794 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:50:36.720507 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:36.720486 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:50:36.770694 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:36.770669 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:50:36.779133 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:36.779111 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:50:36.869257 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:36.869149 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:50:36.873823 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:36.873787 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:50:36.885212 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:36.885191 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:50:36.894027 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:36.894000 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:50:36.928490 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:36.928452 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:50:36.937164 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:36.937140 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:50:36.945935 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:36.945916 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:50:36.958376 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:36.958354 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:50:36.967112 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:36.967088 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:50:36.975878 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:36.975852 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:50:36.987475 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:36.987450 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:50:36.999149 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:36.999126 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:50:37.011918 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:37.011899 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:50:37.019639 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:37.019622 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:50:38.799081 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:38.799049 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:50:38.808141 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:38.808121 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:50:38.816645 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:38.816626 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:50:38.850552 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:38.850528 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:50:38.857431 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:38.857415 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:50:38.870231 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:38.870205 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:50:38.877845 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:38.877738 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:50:38.892806 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:38.892785 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:50:38.899559 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:38.899535 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:50:38.955538 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:38.955508 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:50:38.964466 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:38.964436 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:50:39.054308 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:39.054174 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:50:39.059508 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:39.059483 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:50:39.071302 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:39.071279 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:50:39.080270 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:39.080237 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:50:39.117097 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:39.117075 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:50:39.125803 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:39.125768 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:50:39.134470 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:39.134451 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:50:39.145923 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:39.145906 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:50:39.155046 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:39.155027 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:50:39.163544 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:39.163528 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:50:39.175373 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:39.175355 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:50:39.184166 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:39.184146 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:50:39.195956 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:39.195933 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:50:39.204117 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:39.204099 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:50:40.957900 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:40.957866 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:50:40.968492 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:40.968462 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:50:40.978157 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:40.978126 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:50:41.014919 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:41.014889 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:50:41.021801 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:41.021774 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:50:41.035984 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:41.035959 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:50:41.044784 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:41.044759 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:50:41.058657 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:41.058633 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:50:41.065900 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:41.065875 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:50:41.129918 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:41.129887 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:50:41.139737 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:41.139715 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:50:41.230143 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:41.230047 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:50:41.235132 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:41.235108 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:50:41.246555 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:41.246518 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:50:41.258595 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:41.258574 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:50:41.294479 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:41.294459 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:50:41.303353 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:41.303335 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:50:41.311993 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:41.311967 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:50:41.323745 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:41.323724 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:50:41.332502 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:41.332483 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:50:41.341626 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:41.341610 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:50:41.353705 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:41.353685 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:50:41.362953 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:41.362936 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:50:41.375608 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:41.375584 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:50:41.383668 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:41.383643 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:50:43.166668 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:43.166642 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:50:43.174651 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:43.174627 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:50:43.182779 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:43.182757 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:50:43.218619 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:43.218592 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:50:43.225296 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:43.225273 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:50:43.237577 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:43.237552 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:50:43.244532 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:43.244504 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:50:43.257811 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:43.257792 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:50:43.264077 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:43.264050 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:50:43.313529 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:43.313500 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:50:43.321882 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:43.321852 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:50:43.413555 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:43.413526 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:50:43.418497 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:43.418426 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:50:43.429562 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:43.429540 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:50:43.437599 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:43.437571 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:50:43.471115 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:43.471085 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:50:43.480332 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:43.480305 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:50:43.490907 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:43.490884 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:50:43.502228 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:43.502208 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:50:43.510650 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:43.510630 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:50:43.519340 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:43.519320 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:50:43.531291 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:43.531270 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:50:43.539050 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:43.539033 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:50:43.551104 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:43.551086 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:50:43.558847 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:43.558827 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:50:45.331655 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:45.331621 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:50:45.341145 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:45.341118 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:50:45.349895 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:45.349870 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:50:45.386330 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:45.386301 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:50:45.393635 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:45.393614 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:50:45.408683 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:45.408664 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:50:45.415612 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:45.415589 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:50:45.429175 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:45.429154 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:50:45.436218 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:45.436196 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:50:45.487382 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:45.487359 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:50:45.496165 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:45.496142 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:50:45.583473 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:45.583381 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:50:45.588095 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:45.588068 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:50:45.599089 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:45.599065 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:50:45.607519 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:45.607491 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:50:45.642511 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:45.642473 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:50:45.651296 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:45.651240 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:50:45.659531 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:45.659504 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:50:45.671682 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:45.671657 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:50:45.679834 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:45.679809 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:50:45.688453 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:45.688435 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:50:45.699889 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:45.699861 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:50:45.708406 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:45.708381 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:50:45.720505 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:45.720486 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:50:45.727986 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:45.727965 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:50:47.498599 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:47.498551 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:50:47.508646 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:47.508613 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:50:47.517424 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:47.517404 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:50:47.551856 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:47.551828 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:50:47.559165 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:47.559145 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:50:47.573692 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:47.573670 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:50:47.580672 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:47.580655 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:50:47.593992 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:47.593967 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:50:47.600495 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:47.600468 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:50:47.652591 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:47.652559 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:50:47.661419 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:47.661397 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:50:47.747236 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:47.747203 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:50:47.752689 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:47.752613 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:50:47.764967 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:47.764947 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:50:47.773553 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:47.773532 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:50:47.808258 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:47.808214 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:50:47.817665 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:47.817635 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:50:47.826871 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:47.826849 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:50:47.842428 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:47.842410 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:50:47.851692 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:47.851672 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:50:47.860673 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:47.860657 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:50:47.872187 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:47.872163 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:50:47.882176 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:47.882149 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:50:47.895054 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:47.895034 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:50:47.902697 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:47.902677 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:50:49.707955 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:49.707925 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:50:49.716563 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:49.716539 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:50:49.725120 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:49.725098 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:50:49.759727 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:49.759697 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:50:49.766647 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:49.766625 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:50:49.779179 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:49.779150 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:50:49.785800 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:49.785775 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:50:49.802098 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:49.802074 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:50:49.811051 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:49.811029 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:50:49.861579 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:49.861544 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:50:49.870070 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:49.870042 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:50:49.959879 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:49.959797 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:50:49.964626 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:49.964606 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:50:49.975751 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:49.975726 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:50:49.984010 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:49.983990 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:50:50.018764 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:50.018735 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:50:50.027300 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:50.027274 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:50:50.035989 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:50.035961 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:50:50.047603 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:50.047577 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:50:50.056021 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:50.055988 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:50:50.064449 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:50.064414 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:50:50.076033 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:50.076005 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:50:50.084116 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:50.084094 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:50:50.097103 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:50.097075 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:50:50.104374 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:50.104353 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:50:51.880538 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:51.880510 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:50:51.892890 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:51.892857 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:50:51.904449 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:51.904429 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:50:51.950824 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:51.950794 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:50:51.964973 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:51.964943 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:50:51.985500 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:51.985471 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:50:51.992763 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:51.992739 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:50:52.007239 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:52.007213 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:50:52.020997 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:52.020972 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:50:52.094390 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:52.094361 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:50:52.106987 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:52.106952 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:50:52.214227 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:52.214130 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:50:52.218875 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:52.218851 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:50:52.234416 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:52.234388 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:50:52.253536 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:52.253514 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:50:52.297951 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:52.297926 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:50:52.310831 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:52.310808 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:50:52.322474 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:52.322446 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:50:52.336581 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:52.336560 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:50:52.346579 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:52.346561 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:50:52.358735 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:52.358711 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:50:52.373581 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:52.373559 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:50:52.386730 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:52.386704 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:50:52.403548 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:52.403528 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:50:52.413660 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:52.413631 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:50:54.165949 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:54.165919 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:50:54.174262 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:54.174223 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:50:54.184159 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:54.184140 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:50:54.218747 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:54.218716 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:50:54.225148 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:54.225125 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:50:54.239084 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:54.239052 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:50:54.245775 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:54.245751 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:50:54.258729 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:54.258707 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:50:54.265293 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:54.265259 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:50:54.317668 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:54.317640 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:50:54.325366 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:54.325325 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:50:54.411062 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:54.411033 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:50:54.416628 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:54.416556 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:50:54.427690 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:54.427665 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:50:54.436105 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:54.436077 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:50:54.471882 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:54.471851 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:50:54.480373 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:54.480343 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:50:54.488416 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:54.488396 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:50:54.499754 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:54.499730 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:50:54.507773 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:54.507757 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:50:54.515564 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:54.515545 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:50:54.525742 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:54.525718 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:50:54.533489 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:54.533472 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:50:54.544575 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:54.544552 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:50:54.552962 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:54.552944 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:50:56.305908 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:56.305877 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:50:56.315184 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:56.315157 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:50:56.324321 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:56.324290 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:50:56.359114 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:56.359077 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:50:56.366366 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:56.366344 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:50:56.380862 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:56.380826 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:50:56.389236 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:56.389200 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:50:56.402827 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:56.402798 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:50:56.410279 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:56.410236 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:50:56.462862 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:56.462823 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:50:56.471142 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:56.471112 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:50:56.558711 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:56.558627 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:50:56.563415 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:56.563389 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:50:56.574610 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:56.574590 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:50:56.582978 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:56.582953 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:50:56.617014 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:56.616989 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:50:56.627088 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:56.627063 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:50:56.635461 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:56.635435 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:50:56.655933 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:56.655895 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:50:56.664682 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:56.664652 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:50:56.673102 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:56.673068 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:50:56.684157 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:56.684125 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:50:56.692335 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:56.692313 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:50:56.704654 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:56.704629 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:50:56.712231 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:56.712212 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:50:58.462665 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:58.462635 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:50:58.471836 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:58.471815 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:50:58.480509 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:58.480492 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:50:58.514779 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:58.514750 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:50:58.522646 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:58.522618 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:50:58.538608 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:58.538561 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:50:58.544656 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:58.544635 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:50:58.557974 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:58.557954 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:50:58.564326 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:58.564302 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:50:58.617006 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:58.616985 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:50:58.626149 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:58.626127 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:50:58.713548 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:58.713443 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:50:58.717899 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:58.717854 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:50:58.729774 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:58.729751 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:50:58.738079 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:58.738060 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:50:58.770942 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:58.770924 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:50:58.779956 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:58.779929 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:50:58.788291 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:58.788273 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:50:58.799814 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:58.799790 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:50:58.808300 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:58.808278 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:50:58.817098 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:58.817080 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:50:58.828730 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:58.828711 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:50:58.837203 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:58.837187 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:50:58.849156 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:58.849135 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:50:58.858122 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:50:58.858103 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:51:00.643334 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:00.643277 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:51:00.652363 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:00.652342 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:51:00.660139 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:00.660119 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:51:00.691750 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:00.691725 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:51:00.697802 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:00.697769 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:51:00.709944 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:00.709919 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:51:00.716353 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:00.716325 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:51:00.729099 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:00.729066 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:51:00.735106 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:00.735088 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:51:00.783089 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:00.783065 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:51:00.790541 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:00.790518 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:51:00.873173 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:00.873143 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:51:00.877375 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:00.877352 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:51:00.887686 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:00.887670 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:51:00.895306 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:00.895254 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:51:00.927847 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:00.927822 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:51:00.935622 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:00.935605 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:51:00.943587 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:00.943570 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:51:00.954316 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:00.954293 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:51:00.962446 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:00.962416 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:51:00.970220 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:00.970205 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:51:00.980585 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:00.980570 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:51:00.992414 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:00.988792 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:51:01.002937 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:01.002900 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:51:01.010035 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:01.010010 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:51:02.720922 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:02.720897 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:51:02.730724 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:02.730699 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:51:02.740203 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:02.740182 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:51:02.776109 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:02.776083 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:51:02.783690 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:02.783666 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:51:02.797003 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:02.796968 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:51:02.807716 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:02.807688 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:51:02.825003 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:02.824964 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:51:02.832040 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:02.832018 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:51:02.883925 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:02.883895 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:51:02.892772 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:02.892745 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:51:02.978603 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:02.978525 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:51:02.984267 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:02.984223 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:51:02.995995 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:02.995969 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:51:03.005348 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:03.005315 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:51:03.043407 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:03.043374 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:51:03.052542 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:03.052508 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:51:03.061232 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:03.061209 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:51:03.074443 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:03.074416 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:51:03.083216 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:03.083191 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:51:03.091926 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:03.091906 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:51:03.103340 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:03.103317 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:51:03.112498 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:03.112478 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:51:03.124969 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:03.124949 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:51:03.132702 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:03.132684 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:51:04.895512 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:04.895461 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:51:04.905391 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:04.905371 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:51:04.913855 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:04.913839 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:51:04.946780 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:04.946757 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:51:04.953951 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:04.953925 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:51:04.966904 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:04.966885 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:51:04.973843 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:04.973819 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:51:04.987053 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:04.987031 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:51:04.994170 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:04.994153 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:51:05.045412 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:05.045390 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:51:05.053864 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:05.053843 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:51:05.141581 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:05.141550 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:51:05.146369 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:05.146302 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:51:05.157106 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:05.157084 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:51:05.165212 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:05.165192 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:51:05.198396 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:05.198376 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:51:05.207139 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:05.207111 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:51:05.215526 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:05.215496 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:51:05.227581 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:05.227552 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:51:05.235533 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:05.235515 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:51:05.244104 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:05.244084 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:51:05.255077 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:05.255053 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:51:05.263903 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:05.263886 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:51:05.276211 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:05.276195 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:51:05.283587 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:05.283572 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:51:07.030409 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:07.030382 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:51:07.049595 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:07.049559 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:51:07.058692 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:07.058668 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:51:07.096597 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:07.096560 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:51:07.104170 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:07.104151 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:51:07.118716 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:07.118696 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:51:07.126006 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:07.125982 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:51:07.139180 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:07.139158 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:51:07.145793 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:07.145771 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:51:07.198437 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:07.198414 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:51:07.206879 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:07.206861 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:51:07.293280 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:07.293236 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:51:07.298178 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:07.298157 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:51:07.309394 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:07.309376 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:51:07.318126 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:07.318110 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:51:07.357628 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:07.357602 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:51:07.367713 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:07.367691 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:51:07.376887 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:07.376871 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:51:07.388627 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:07.388609 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:51:07.397783 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:07.397763 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:51:07.407186 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:07.407166 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:51:07.418966 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:07.418948 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:51:07.427616 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:07.427588 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:51:07.442703 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:07.442685 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:51:07.451282 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:07.451265 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:51:09.232118 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:09.232091 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:51:09.241584 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:09.241559 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:51:09.250631 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:09.250607 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:51:09.287174 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:09.287143 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:51:09.294359 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:09.294330 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:51:09.310282 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:09.310235 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:51:09.317414 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:09.317367 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:51:09.330367 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:09.330344 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:51:09.337762 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:09.337724 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:51:09.387333 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:09.387297 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:51:09.395387 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:09.395365 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:51:09.485022 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:09.484930 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:51:09.490447 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:09.490419 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:51:09.503717 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:09.503674 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:51:09.512217 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:09.512197 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:51:09.546788 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:09.546765 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:51:09.555289 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:09.555268 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:51:09.563469 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:09.563448 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:51:09.574916 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:09.574897 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:51:09.583790 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:09.583769 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:51:09.592354 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:09.592338 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:51:09.603741 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:09.603721 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:51:09.612998 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:09.612971 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:51:09.625239 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:09.625217 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:51:09.633045 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:09.633027 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:51:11.456677 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:11.456647 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:51:11.466859 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:11.466831 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:51:11.475855 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:11.475833 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:51:11.512216 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:11.512192 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:51:11.520019 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:11.519997 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:51:11.533758 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:11.533733 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:51:11.541986 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:11.541966 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:51:11.556823 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:11.556805 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:51:11.564997 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:11.564981 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:51:11.620988 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:11.620961 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:51:11.630295 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:11.630272 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:51:11.721910 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:11.721826 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:51:11.727102 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:11.727081 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:51:11.739135 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:11.739117 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:51:11.747917 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:11.747894 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:51:11.785018 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:11.784994 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:51:11.793650 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:11.793629 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:51:11.803960 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:11.803945 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:51:11.816284 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:11.816238 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:51:11.824980 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:11.824962 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:51:11.833926 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:11.833895 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:51:11.846294 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:11.846273 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:51:11.855206 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:11.855190 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:51:11.868011 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:11.867991 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:51:11.876191 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:11.876168 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:51:13.700015 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:13.699985 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:51:13.711719 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:13.711699 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:51:13.722953 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:13.722928 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:51:13.763899 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:13.763865 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:51:13.774390 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:13.774367 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:51:13.789274 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:13.789234 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:51:13.797451 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:13.797424 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:51:13.811873 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:13.811846 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:51:13.819442 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:13.819415 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:51:13.877780 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:13.877747 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:51:13.891576 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:13.891547 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:51:14.003434 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:14.003347 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:51:14.010603 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:14.010582 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:51:14.026598 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:14.026577 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:51:14.038203 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:14.038181 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:51:14.081141 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:14.081113 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:51:14.093903 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:14.093878 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:51:14.109375 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:14.109351 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:51:14.124447 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:14.124418 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:51:14.136194 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:14.136172 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:51:14.156100 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:14.156077 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:51:14.182335 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:14.182313 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:51:14.205092 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:14.205071 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:51:14.223547 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:14.223525 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:51:14.237968 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:14.237944 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:51:16.136214 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:16.136183 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:51:16.145509 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:16.145479 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:51:16.154296 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:16.154277 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:51:16.188664 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:16.188621 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:51:16.198480 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:16.198455 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:51:16.213429 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:16.213401 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:51:16.220384 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:16.220360 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:51:16.235124 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:16.235097 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:51:16.245176 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:16.245156 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:51:16.299517 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:16.299488 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:51:16.308902 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:16.308875 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:51:16.402889 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:16.402797 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:51:16.408801 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:16.408778 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:51:16.421807 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:16.421785 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:51:16.431913 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:16.431890 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:51:16.469506 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:16.469476 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:51:16.478894 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:16.478872 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:51:16.487744 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:16.487723 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:51:16.500240 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:16.500213 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:51:16.509891 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:16.509857 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:51:16.519008 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:16.518991 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:51:16.531656 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:16.531637 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:51:16.540537 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:16.540514 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:51:16.555166 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:16.555143 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:51:16.563020 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:16.563004 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:51:18.430092 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:18.430068 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:51:18.439421 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:18.439404 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:51:18.448355 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:18.448338 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:51:18.483649 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:18.483627 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:51:18.490987 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:18.490964 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:51:18.504631 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:18.504598 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:51:18.511992 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:18.511973 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:51:18.525863 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:18.525846 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:51:18.533275 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:18.533239 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:51:18.591556 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:18.591519 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:51:18.601645 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:18.601626 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:51:18.693759 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:18.693650 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:51:18.698967 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:18.698942 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:51:18.711938 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:18.711913 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:51:18.720714 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:18.720691 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:51:18.759296 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:18.759267 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:51:18.769565 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:18.769540 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:51:18.779981 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:18.779952 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:51:18.794581 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:18.794552 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:51:18.803775 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:18.803749 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:51:18.812754 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:18.812735 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:51:18.824542 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:18.824507 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:51:18.833948 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:18.833932 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:51:18.848142 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:18.848110 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:51:18.856984 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:18.856928 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:51:20.741383 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:20.741356 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:51:20.750361 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:20.750332 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:51:20.758688 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:20.758644 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:51:20.795538 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:20.795509 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:51:20.804509 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:20.804488 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:51:20.817377 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:20.817354 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:51:20.824225 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:20.824202 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:51:20.838083 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:20.838061 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:51:20.844894 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:20.844872 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:51:20.897820 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:20.897784 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:51:20.907898 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:20.907874 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:51:20.993947 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:20.993864 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:51:20.998795 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:20.998771 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:51:21.013232 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:21.013201 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:51:21.022062 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:21.022038 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:51:21.058266 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:21.058220 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:51:21.067330 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:21.067310 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:51:21.075631 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:21.075611 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:51:21.087685 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:21.087659 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:51:21.097361 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:21.097338 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:51:21.107114 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:21.107092 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:51:21.118680 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:21.118659 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:51:21.126792 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:21.126769 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:51:21.139229 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:21.139202 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:51:21.146786 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:21.146768 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:51:22.958646 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:22.958619 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:51:22.967910 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:22.967886 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:51:22.976657 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:22.976626 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:51:23.016099 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:23.016077 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:51:23.023785 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:23.023761 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:51:23.037299 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:23.037277 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:51:23.045184 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:23.045165 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:51:23.058977 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:23.058956 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:51:23.066122 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:23.066099 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:51:23.120913 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:23.120884 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:51:23.129881 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:23.129855 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:51:23.229069 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:23.228977 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:51:23.234200 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:23.234176 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:51:23.246994 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:23.246971 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:51:23.256468 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:23.256450 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:51:23.292596 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:23.292565 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:51:23.302465 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:23.302441 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:51:23.311447 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:23.311419 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:51:23.323860 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:23.323824 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:51:23.332553 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:23.332530 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:51:23.341728 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:23.341707 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:51:23.355143 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:23.355097 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:51:23.365628 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:23.365600 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:51:23.378653 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:23.378621 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:51:23.386853 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:23.386829 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:51:25.221221 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:25.221191 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:51:25.230992 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:25.230969 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:51:25.239347 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:25.239321 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:51:25.273732 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:25.273703 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:51:25.282580 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:25.282555 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:51:25.295252 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:25.295226 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:51:25.302597 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:25.302573 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:51:25.333235 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:25.333205 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:51:25.340396 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:25.340372 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:51:25.392032 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:25.391997 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:51:25.400859 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:25.400839 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:51:25.492007 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:25.491918 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:51:25.501953 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:25.501923 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:51:25.514021 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:25.513989 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:51:25.529008 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:25.528987 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:51:25.563992 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:25.563962 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:51:25.573227 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:25.573190 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:51:25.582806 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:25.582778 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:51:25.595106 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:25.595082 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:51:25.604151 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:25.604128 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:51:25.613212 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:25.613193 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:51:25.624554 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:25.624536 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:51:25.632900 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:25.632880 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:51:25.645135 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:25.645100 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:51:25.652761 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:25.652737 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:51:27.440536 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:27.440498 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/kserve-container/0.log" Apr 16 17:51:27.451797 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:27.451752 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/agent/0.log" Apr 16 17:51:27.460402 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:27.460374 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-predictor-7fc456f8b8-qzxjk_59298cc7-7cfd-49cd-b675-a5784a4851b6/storage-initializer/0.log" Apr 16 17:51:27.493291 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:27.493258 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/kserve-container/0.log" Apr 16 17:51:27.499701 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:27.499677 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-predictor-7dddcb4bd4-z562j_2183d52a-e000-4a8f-9d4b-3d289a306e23/storage-initializer/0.log" Apr 16 17:51:27.513036 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:27.513013 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/kserve-container/0.log" Apr 16 17:51:27.519962 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:27.519939 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-runtime-predictor-86b49c4466-65rhw_41e6c8cc-bfa4-441f-9a7d-189270eafb7a/storage-initializer/0.log" Apr 16 17:51:27.533065 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:27.533030 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/kserve-container/0.log" Apr 16 17:51:27.539264 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:27.539226 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-paddle-v2-kserve-predictor-7c6bd76f7b-fp452_fa6e2817-5d34-4c5e-9436-ab23a1de66a7/storage-initializer/0.log" Apr 16 17:51:27.591331 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:27.591300 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/kserve-container/0.log" Apr 16 17:51:27.599912 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:27.599881 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-pmml-v2-kserve-predictor-75b87ff64c-p9vtl_6ef0fefe-cad8-48c9-b622-e02732895f5d/storage-initializer/0.log" Apr 16 17:51:27.685872 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:27.685837 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/kserve-container/0.log" Apr 16 17:51:27.690967 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:27.690895 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-fm9rb_1117ed58-9128-467f-93b0-8c8777987c99/storage-initializer/0.log" Apr 16 17:51:27.702179 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:27.702149 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/kserve-container/0.log" Apr 16 17:51:27.710658 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:27.710640 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-predictive-xgboost-predictor-794b54b9b4-xw6q7_969ea101-644f-4d51-9566-7e8cf20aa33f/storage-initializer/0.log" Apr 16 17:51:27.745518 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:27.745483 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/kserve-container/0.log" Apr 16 17:51:27.754277 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:27.754231 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/agent/0.log" Apr 16 17:51:27.762738 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:27.762720 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-848df975bd-s4p84_df201950-65db-4d8e-a187-b29ae219ea5d/storage-initializer/0.log" Apr 16 17:51:27.775962 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:27.775934 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/kserve-container/0.log" Apr 16 17:51:27.784098 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:27.784078 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/agent/0.log" Apr 16 17:51:27.792448 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:27.792429 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-predictor-6d5844b6c5-2pp6r_e190ead5-ee2d-4785-9efa-b6f854dfee7f/storage-initializer/0.log" Apr 16 17:51:27.807643 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:27.807623 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/kserve-container/0.log" Apr 16 17:51:27.818469 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:27.818449 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-66fb7c44cc-gzszm_907fcecd-6d8b-4ac3-9eb5-92055fe5f000/storage-initializer/0.log" Apr 16 17:51:27.831816 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:27.831795 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/kserve-container/0.log" Apr 16 17:51:27.841762 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:27.841742 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf744c55f-54m8x_04655ab7-174e-4cf3-b5cf-52106e407e9c/storage-initializer/0.log" Apr 16 17:51:33.964087 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:33.964051 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pgvd2_98d7d9af-1c21-4f39-b605-21a11d06f2ae/global-pull-secret-syncer/0.log" Apr 16 17:51:34.050933 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:34.050900 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kwz5r_f6b12331-2022-4c05-ac42-753274b499dc/konnectivity-agent/0.log" Apr 16 17:51:34.186837 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:34.186809 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-37.ec2.internal_8b1f1c157df36eece23646e8d91601bf/haproxy/0.log" Apr 16 17:51:37.521484 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:37.521443 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e1c13e2d-7bc4-4764-b62a-01c231a9b003/alertmanager/0.log" Apr 16 17:51:37.538413 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:37.538385 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e1c13e2d-7bc4-4764-b62a-01c231a9b003/config-reloader/0.log" Apr 16 17:51:37.557996 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:37.557971 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e1c13e2d-7bc4-4764-b62a-01c231a9b003/kube-rbac-proxy-web/0.log" Apr 16 17:51:37.578647 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:37.578621 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e1c13e2d-7bc4-4764-b62a-01c231a9b003/kube-rbac-proxy/0.log" Apr 16 17:51:37.599665 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:37.599629 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e1c13e2d-7bc4-4764-b62a-01c231a9b003/kube-rbac-proxy-metric/0.log" Apr 16 17:51:37.618410 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:37.618380 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e1c13e2d-7bc4-4764-b62a-01c231a9b003/prom-label-proxy/0.log" Apr 16 17:51:37.635873 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:37.635847 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e1c13e2d-7bc4-4764-b62a-01c231a9b003/init-config-reloader/0.log" Apr 16 17:51:37.695043 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:37.695006 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6667474d89-lqmw4_12d57405-05cf-462d-a747-06c91ca773cb/cluster-monitoring-operator/0.log" Apr 16 17:51:37.717490 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:37.717467 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7479c89684-8kzt5_1a20b040-7ded-4aba-a389-84ea64de0827/kube-state-metrics/0.log" Apr 16 17:51:37.735475 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:37.735451 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7479c89684-8kzt5_1a20b040-7ded-4aba-a389-84ea64de0827/kube-rbac-proxy-main/0.log" Apr 16 17:51:37.755770 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:37.755730 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7479c89684-8kzt5_1a20b040-7ded-4aba-a389-84ea64de0827/kube-rbac-proxy-self/0.log" Apr 16 17:51:37.804705 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:37.804658 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5876b4bbc7-gmxs2_d620b8bc-d805-441c-864d-5f23f7621826/monitoring-plugin/0.log" Apr 16 17:51:37.833145 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:37.833125 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-59tvd_26f9c3a6-a68f-4294-ad49-f50574e36b2d/node-exporter/0.log" Apr 16 17:51:37.849780 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:37.849759 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-59tvd_26f9c3a6-a68f-4294-ad49-f50574e36b2d/kube-rbac-proxy/0.log" Apr 16 17:51:37.866477 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:37.866451 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-59tvd_26f9c3a6-a68f-4294-ad49-f50574e36b2d/init-textfile/0.log" Apr 16 17:51:38.108632 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:38.108548 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_05797269-685d-4822-90bf-01c7aa513b40/prometheus/0.log" Apr 16 17:51:38.123179 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:38.123145 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_05797269-685d-4822-90bf-01c7aa513b40/config-reloader/0.log" Apr 16 17:51:38.145014 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:38.144991 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_05797269-685d-4822-90bf-01c7aa513b40/thanos-sidecar/0.log" Apr 16 17:51:38.163568 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:38.163541 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_05797269-685d-4822-90bf-01c7aa513b40/kube-rbac-proxy-web/0.log" Apr 16 17:51:38.185088 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:38.185068 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_05797269-685d-4822-90bf-01c7aa513b40/kube-rbac-proxy/0.log" Apr 16 17:51:38.207018 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:38.206993 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_05797269-685d-4822-90bf-01c7aa513b40/kube-rbac-proxy-thanos/0.log" Apr 16 17:51:38.236691 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:38.236669 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_05797269-685d-4822-90bf-01c7aa513b40/init-config-reloader/0.log" Apr 16 17:51:38.265182 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:38.265157 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-78f957474d-8qhj4_a627f9f0-08a8-49a0-a0f6-34229f6c3330/prometheus-operator/0.log" Apr 16 17:51:38.278447 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:38.278419 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-78f957474d-8qhj4_a627f9f0-08a8-49a0-a0f6-34229f6c3330/kube-rbac-proxy/0.log" Apr 16 17:51:38.302156 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:38.302130 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-9cb97cd87-rwt5j_5c8c501e-0685-46ea-a440-9417fdc93588/prometheus-operator-admission-webhook/0.log" Apr 16 17:51:38.335675 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:38.335648 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-674cd685d5-c72ss_a5a5bf5d-ebc9-47c7-a825-754a41a34908/telemeter-client/0.log" Apr 16 17:51:38.354378 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:38.354341 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-674cd685d5-c72ss_a5a5bf5d-ebc9-47c7-a825-754a41a34908/reload/0.log" Apr 16 17:51:38.374471 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:38.374372 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-674cd685d5-c72ss_a5a5bf5d-ebc9-47c7-a825-754a41a34908/kube-rbac-proxy/0.log" Apr 16 17:51:38.399739 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:38.399712 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78df5754bd-ngrn7_05869c45-9aab-4f0a-93bd-06c4bba46382/thanos-query/0.log" Apr 16 17:51:38.418748 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:38.418720 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78df5754bd-ngrn7_05869c45-9aab-4f0a-93bd-06c4bba46382/kube-rbac-proxy-web/0.log" Apr 16 17:51:38.437680 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:38.437658 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78df5754bd-ngrn7_05869c45-9aab-4f0a-93bd-06c4bba46382/kube-rbac-proxy/0.log" Apr 16 17:51:38.457704 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:38.457684 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78df5754bd-ngrn7_05869c45-9aab-4f0a-93bd-06c4bba46382/prom-label-proxy/0.log" Apr 16 17:51:38.474967 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:38.474939 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78df5754bd-ngrn7_05869c45-9aab-4f0a-93bd-06c4bba46382/kube-rbac-proxy-rules/0.log" Apr 16 17:51:38.492567 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:38.492542 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78df5754bd-ngrn7_05869c45-9aab-4f0a-93bd-06c4bba46382/kube-rbac-proxy-metrics/0.log" Apr 16 17:51:39.667508 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:39.667449 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5cb6cf4cb4-rvpkl_faaff24f-341b-422f-833f-902c51df7815/networking-console-plugin/0.log" Apr 16 17:51:41.212334 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:41.212295 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dtnls/perf-node-gather-daemonset-p69dr"] Apr 16 17:51:41.216713 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:41.216693 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dtnls/perf-node-gather-daemonset-p69dr"] Apr 16 17:51:41.216835 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:41.216791 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dtnls/perf-node-gather-daemonset-p69dr" Apr 16 17:51:41.218626 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:41.218607 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dtnls\"/\"kube-root-ca.crt\"" Apr 16 17:51:41.219123 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:41.219089 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dtnls\"/\"default-dockercfg-zt7d5\"" Apr 16 17:51:41.219190 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:41.219135 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dtnls\"/\"openshift-service-ca.crt\"" Apr 16 17:51:41.305483 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:41.305443 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6b71f42d-9d31-478c-b7b6-ae284d9bb424-podres\") pod \"perf-node-gather-daemonset-p69dr\" (UID: \"6b71f42d-9d31-478c-b7b6-ae284d9bb424\") " pod="openshift-must-gather-dtnls/perf-node-gather-daemonset-p69dr" Apr 16 17:51:41.305669 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:41.305491 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq9tg\" (UniqueName: \"kubernetes.io/projected/6b71f42d-9d31-478c-b7b6-ae284d9bb424-kube-api-access-zq9tg\") pod \"perf-node-gather-daemonset-p69dr\" (UID: \"6b71f42d-9d31-478c-b7b6-ae284d9bb424\") " pod="openshift-must-gather-dtnls/perf-node-gather-daemonset-p69dr" Apr 16 17:51:41.305669 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:41.305586 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6b71f42d-9d31-478c-b7b6-ae284d9bb424-proc\") pod \"perf-node-gather-daemonset-p69dr\" (UID: \"6b71f42d-9d31-478c-b7b6-ae284d9bb424\") " pod="openshift-must-gather-dtnls/perf-node-gather-daemonset-p69dr" Apr 16 17:51:41.305669 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:41.305645 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b71f42d-9d31-478c-b7b6-ae284d9bb424-sys\") pod \"perf-node-gather-daemonset-p69dr\" (UID: \"6b71f42d-9d31-478c-b7b6-ae284d9bb424\") " pod="openshift-must-gather-dtnls/perf-node-gather-daemonset-p69dr" Apr 16 17:51:41.305669 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:41.305667 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b71f42d-9d31-478c-b7b6-ae284d9bb424-lib-modules\") pod \"perf-node-gather-daemonset-p69dr\" (UID: \"6b71f42d-9d31-478c-b7b6-ae284d9bb424\") " pod="openshift-must-gather-dtnls/perf-node-gather-daemonset-p69dr" Apr 16 17:51:41.406277 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:41.406216 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6b71f42d-9d31-478c-b7b6-ae284d9bb424-podres\") pod \"perf-node-gather-daemonset-p69dr\" (UID: \"6b71f42d-9d31-478c-b7b6-ae284d9bb424\") " pod="openshift-must-gather-dtnls/perf-node-gather-daemonset-p69dr" Apr 16 17:51:41.406470 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:41.406284 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zq9tg\" (UniqueName: \"kubernetes.io/projected/6b71f42d-9d31-478c-b7b6-ae284d9bb424-kube-api-access-zq9tg\") pod \"perf-node-gather-daemonset-p69dr\" (UID: \"6b71f42d-9d31-478c-b7b6-ae284d9bb424\") " pod="openshift-must-gather-dtnls/perf-node-gather-daemonset-p69dr" Apr 16 17:51:41.406470 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:41.406323 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6b71f42d-9d31-478c-b7b6-ae284d9bb424-proc\") pod \"perf-node-gather-daemonset-p69dr\" (UID: \"6b71f42d-9d31-478c-b7b6-ae284d9bb424\") " pod="openshift-must-gather-dtnls/perf-node-gather-daemonset-p69dr" Apr 16 17:51:41.406470 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:41.406369 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b71f42d-9d31-478c-b7b6-ae284d9bb424-sys\") pod \"perf-node-gather-daemonset-p69dr\" (UID: \"6b71f42d-9d31-478c-b7b6-ae284d9bb424\") " pod="openshift-must-gather-dtnls/perf-node-gather-daemonset-p69dr" Apr 16 17:51:41.406470 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:41.406389 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b71f42d-9d31-478c-b7b6-ae284d9bb424-lib-modules\") pod \"perf-node-gather-daemonset-p69dr\" (UID: \"6b71f42d-9d31-478c-b7b6-ae284d9bb424\") " pod="openshift-must-gather-dtnls/perf-node-gather-daemonset-p69dr" Apr 16 17:51:41.406470 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:41.406403 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6b71f42d-9d31-478c-b7b6-ae284d9bb424-podres\") pod \"perf-node-gather-daemonset-p69dr\" (UID: \"6b71f42d-9d31-478c-b7b6-ae284d9bb424\") " pod="openshift-must-gather-dtnls/perf-node-gather-daemonset-p69dr" Apr 16 17:51:41.406703 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:41.406496 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b71f42d-9d31-478c-b7b6-ae284d9bb424-sys\") pod \"perf-node-gather-daemonset-p69dr\" (UID: \"6b71f42d-9d31-478c-b7b6-ae284d9bb424\") " pod="openshift-must-gather-dtnls/perf-node-gather-daemonset-p69dr" Apr 16 17:51:41.406703 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:41.406547 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6b71f42d-9d31-478c-b7b6-ae284d9bb424-proc\") pod \"perf-node-gather-daemonset-p69dr\" (UID: \"6b71f42d-9d31-478c-b7b6-ae284d9bb424\") " pod="openshift-must-gather-dtnls/perf-node-gather-daemonset-p69dr" Apr 16 17:51:41.406703 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:41.406609 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b71f42d-9d31-478c-b7b6-ae284d9bb424-lib-modules\") pod \"perf-node-gather-daemonset-p69dr\" (UID: \"6b71f42d-9d31-478c-b7b6-ae284d9bb424\") " pod="openshift-must-gather-dtnls/perf-node-gather-daemonset-p69dr" Apr 16 17:51:41.413531 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:41.413510 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq9tg\" (UniqueName: \"kubernetes.io/projected/6b71f42d-9d31-478c-b7b6-ae284d9bb424-kube-api-access-zq9tg\") pod \"perf-node-gather-daemonset-p69dr\" (UID: \"6b71f42d-9d31-478c-b7b6-ae284d9bb424\") " pod="openshift-must-gather-dtnls/perf-node-gather-daemonset-p69dr" Apr 16 17:51:41.529908 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:41.529830 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dtnls\"/\"default-dockercfg-zt7d5\"" Apr 16 17:51:41.538485 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:41.538456 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dtnls/perf-node-gather-daemonset-p69dr" Apr 16 17:51:41.613650 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:41.613622 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rzx7h_b4e2d35c-1627-4f9c-bea6-985f08319902/dns/0.log" Apr 16 17:51:41.635099 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:41.635072 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rzx7h_b4e2d35c-1627-4f9c-bea6-985f08319902/kube-rbac-proxy/0.log" Apr 16 17:51:41.667650 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:41.667627 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dtnls/perf-node-gather-daemonset-p69dr"] Apr 16 17:51:41.670115 ip-10-0-142-37 kubenswrapper[2576]: W0416 17:51:41.670087 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6b71f42d_9d31_478c_b7b6_ae284d9bb424.slice/crio-b22ed2a6eec8f47d434d652f542c2de7a655ac03e80b2087792c4e0e20a44681 WatchSource:0}: Error finding container b22ed2a6eec8f47d434d652f542c2de7a655ac03e80b2087792c4e0e20a44681: Status 404 returned error can't find the container with id b22ed2a6eec8f47d434d652f542c2de7a655ac03e80b2087792c4e0e20a44681 Apr 16 17:51:41.671974 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:41.671957 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Apr 16 17:51:41.699259 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:41.699222 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-42cdh_99da12c7-93e3-4848-afc5-36f2cbf5645d/dns-node-resolver/0.log" Apr 16 17:51:41.710648 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:41.710617 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dtnls/perf-node-gather-daemonset-p69dr" event={"ID":"6b71f42d-9d31-478c-b7b6-ae284d9bb424","Type":"ContainerStarted","Data":"b22ed2a6eec8f47d434d652f542c2de7a655ac03e80b2087792c4e0e20a44681"} Apr 16 17:51:42.176695 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:42.176658 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q6fvl_fce43675-3aee-453f-b5ad-a28fba84c810/node-ca/0.log" Apr 16 17:51:42.715482 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:42.715450 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dtnls/perf-node-gather-daemonset-p69dr" event={"ID":"6b71f42d-9d31-478c-b7b6-ae284d9bb424","Type":"ContainerStarted","Data":"65c6d000421de776aa0a50d39e46facefab470f41f5c57a5d89d50d6e2d2123a"} Apr 16 17:51:42.715868 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:42.715565 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dtnls/perf-node-gather-daemonset-p69dr" Apr 16 17:51:42.731625 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:42.731584 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-dtnls/perf-node-gather-daemonset-p69dr" podStartSLOduration=1.731567419 podStartE2EDuration="1.731567419s" podCreationTimestamp="2026-04-16 17:51:41 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 17:51:42.73090326 +0000 UTC m=+3841.792052793" watchObservedRunningTime="2026-04-16 17:51:42.731567419 +0000 UTC m=+3841.792716947" Apr 16 17:51:42.900691 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:42.900661 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-85bf56ff4-nrtll_45604778-a27a-4c44-b245-0208bfb4154a/router/0.log" Apr 16 17:51:43.249568 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:43.249538 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xfnx6_efee44e6-2e2b-47f0-b356-6e77267f19e0/serve-healthcheck-canary/0.log" Apr 16 17:51:43.766535 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:43.766508 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bgr9k_b7a95004-7ff6-4be3-bb13-bb10aa6d5840/kube-rbac-proxy/0.log" Apr 16 17:51:43.784103 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:43.784081 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bgr9k_b7a95004-7ff6-4be3-bb13-bb10aa6d5840/exporter/0.log" Apr 16 17:51:43.802421 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:43.802398 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bgr9k_b7a95004-7ff6-4be3-bb13-bb10aa6d5840/extractor/0.log" Apr 16 17:51:46.116119 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:46.116022 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-fxl9h_182c48d6-e302-4868-b398-a0447e32a794/s3-init/0.log" Apr 16 17:51:46.134661 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:46.134630 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-vqqz2_14b0808e-16e3-4b10-993d-c7c6d9fd96d2/s3-tls-init-custom/0.log" Apr 16 17:51:46.152710 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:46.152681 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-nwhqq_42b09675-9fe9-480a-8470-a70374427d0d/s3-tls-init-serving/0.log" Apr 16 17:51:48.730100 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:48.730073 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dtnls/perf-node-gather-daemonset-p69dr" Apr 16 17:51:50.177672 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:50.177575 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-756bb7d76f-dzm7p_4bb5aacd-20aa-4fb6-ae32-eee3df133067/kube-storage-version-migrator-operator/1.log" Apr 16 17:51:50.179555 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:50.179523 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-756bb7d76f-dzm7p_4bb5aacd-20aa-4fb6-ae32-eee3df133067/kube-storage-version-migrator-operator/0.log" Apr 16 17:51:51.217549 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:51.217519 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8zhpq_20d62408-3657-49b5-a83b-39331108c8d9/kube-multus-additional-cni-plugins/0.log" Apr 16 17:51:51.236789 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:51.236764 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8zhpq_20d62408-3657-49b5-a83b-39331108c8d9/egress-router-binary-copy/0.log" Apr 16 17:51:51.254216 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:51.254185 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8zhpq_20d62408-3657-49b5-a83b-39331108c8d9/cni-plugins/0.log" Apr 16 17:51:51.270084 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:51.270063 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8zhpq_20d62408-3657-49b5-a83b-39331108c8d9/bond-cni-plugin/0.log" Apr 16 17:51:51.286496 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:51.286443 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8zhpq_20d62408-3657-49b5-a83b-39331108c8d9/routeoverride-cni/0.log" Apr 16 17:51:51.301961 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:51.301938 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8zhpq_20d62408-3657-49b5-a83b-39331108c8d9/whereabouts-cni-bincopy/0.log" Apr 16 17:51:51.318062 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:51.318041 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8zhpq_20d62408-3657-49b5-a83b-39331108c8d9/whereabouts-cni/0.log" Apr 16 17:51:51.648622 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:51.648576 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-ndl5w_8e078538-0666-4e1e-8b80-2f09583fbd00/kube-multus/0.log" Apr 16 17:51:51.665414 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:51.665388 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hk5gr_412718e2-2075-4c7b-a6a9-3b94ed57564c/network-metrics-daemon/0.log" Apr 16 17:51:51.679677 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:51.679648 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hk5gr_412718e2-2075-4c7b-a6a9-3b94ed57564c/kube-rbac-proxy/0.log" Apr 16 17:51:52.743033 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:52.742996 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7c8c_c2fec5cf-333c-4342-bd77-642bb2581568/ovn-controller/0.log" Apr 16 17:51:52.794773 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:52.794742 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7c8c_c2fec5cf-333c-4342-bd77-642bb2581568/ovn-acl-logging/0.log" Apr 16 17:51:52.813385 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:52.813337 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7c8c_c2fec5cf-333c-4342-bd77-642bb2581568/kube-rbac-proxy-node/0.log" Apr 16 17:51:52.832587 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:52.832553 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7c8c_c2fec5cf-333c-4342-bd77-642bb2581568/kube-rbac-proxy-ovn-metrics/0.log" Apr 16 17:51:52.845878 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:52.845852 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7c8c_c2fec5cf-333c-4342-bd77-642bb2581568/northd/0.log" Apr 16 17:51:52.863103 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:52.863018 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7c8c_c2fec5cf-333c-4342-bd77-642bb2581568/nbdb/0.log" Apr 16 17:51:52.881095 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:52.881065 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7c8c_c2fec5cf-333c-4342-bd77-642bb2581568/sbdb/0.log" Apr 16 17:51:53.061778 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:53.061747 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7c8c_c2fec5cf-333c-4342-bd77-642bb2581568/ovnkube-controller/0.log" Apr 16 17:51:54.212467 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:54.212433 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7b678d77c7-pk46g_120a4def-c7cf-426f-8c72-f746c56c4674/check-endpoints/0.log" Apr 16 17:51:54.231621 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:54.231593 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-5kz89_e730d89c-85b3-4a44-883b-d65f347261ec/network-check-target-container/0.log" Apr 16 17:51:55.120980 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:55.120944 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-9vmmq_2fef7f9f-e604-47d1-877a-3dfcbbdebba0/iptables-alerter/0.log" Apr 16 17:51:55.801522 ip-10-0-142-37 kubenswrapper[2576]: I0416 17:51:55.801494 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-vh7lw_145fe4b0-c23a-49dc-abe3-212e53cff730/tuned/0.log"