Jul 21 17:26:12.122435 ip-10-0-132-249 systemd[1]: Starting Kubernetes Kubelet...
Jul 21 17:26:12.498337 ip-10-0-132-249 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 21 17:26:12.498337 ip-10-0-132-249 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 21 17:26:12.498337 ip-10-0-132-249 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 21 17:26:12.498337 ip-10-0-132-249 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 21 17:26:12.498337 ip-10-0-132-249 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 21 17:26:12.499944 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.499858    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 21 17:26:12.505827 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505812    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 21 17:26:12.505827 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505827    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 21 17:26:12.505891 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505833    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 21 17:26:12.505891 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505836    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 21 17:26:12.505891 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505839    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 21 17:26:12.505891 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505843    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 21 17:26:12.505891 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505846    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 21 17:26:12.505891 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505849    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 21 17:26:12.505891 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505852    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 21 17:26:12.505891 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505854    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 21 17:26:12.505891 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505857    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 21 17:26:12.505891 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505860    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 21 17:26:12.505891 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505862    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 21 17:26:12.505891 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505865    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 21 17:26:12.505891 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505867    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 21 17:26:12.505891 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505870    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 21 17:26:12.505891 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505873    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 21 17:26:12.505891 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505875    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 21 17:26:12.505891 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505883    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 21 17:26:12.505891 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505886    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 21 17:26:12.505891 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505888    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 21 17:26:12.505891 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505891    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 21 17:26:12.506396 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505893    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 21 17:26:12.506396 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505896    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 21 17:26:12.506396 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505899    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 21 17:26:12.506396 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505901    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 21 17:26:12.506396 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505904    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 21 17:26:12.506396 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505907    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 21 17:26:12.506396 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505909    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 21 17:26:12.506396 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505911    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 21 17:26:12.506396 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505914    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 21 17:26:12.506396 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505916    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 21 17:26:12.506396 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505918    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 21 17:26:12.506396 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505921    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 21 17:26:12.506396 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505923    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 21 17:26:12.506396 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505925    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 21 17:26:12.506396 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505928    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 21 17:26:12.506396 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505930    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 21 17:26:12.506396 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505933    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 21 17:26:12.506396 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505935    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 21 17:26:12.506396 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505937    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 21 17:26:12.506396 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505940    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 21 17:26:12.506873 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505942    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 21 17:26:12.506873 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505945    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 21 17:26:12.506873 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505948    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 21 17:26:12.506873 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505951    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 21 17:26:12.506873 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505953    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 21 17:26:12.506873 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505956    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 21 17:26:12.506873 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505958    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 21 17:26:12.506873 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505961    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 21 17:26:12.506873 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505963    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 21 17:26:12.506873 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505965    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 21 17:26:12.506873 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505968    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 21 17:26:12.506873 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505972    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 21 17:26:12.506873 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505974    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 21 17:26:12.506873 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505977    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 21 17:26:12.506873 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505979    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 21 17:26:12.506873 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505982    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 21 17:26:12.506873 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505984    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 21 17:26:12.506873 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505987    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 21 17:26:12.506873 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505989    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 21 17:26:12.506873 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505992    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 21 17:26:12.507390 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505994    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 21 17:26:12.507390 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505996    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 21 17:26:12.507390 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.505999    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 21 17:26:12.507390 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506002    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 21 17:26:12.507390 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506005    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 21 17:26:12.507390 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506007    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 21 17:26:12.507390 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506010    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 21 17:26:12.507390 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506012    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 21 17:26:12.507390 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506014    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 21 17:26:12.507390 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506017    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 21 17:26:12.507390 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506019    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 21 17:26:12.507390 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506021    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 21 17:26:12.507390 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506024    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 21 17:26:12.507390 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506028    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 21 17:26:12.507390 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506032    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 21 17:26:12.507390 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506036    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 21 17:26:12.507390 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506039    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 21 17:26:12.507390 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506042    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 21 17:26:12.507390 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506051    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 21 17:26:12.507842 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506054    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 21 17:26:12.507842 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506057    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 21 17:26:12.507842 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506060    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 21 17:26:12.507842 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506062    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 21 17:26:12.507842 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506066    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 21 17:26:12.507842 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506068    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 21 17:26:12.507842 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506071    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 21 17:26:12.507842 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506074    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 21 17:26:12.507842 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506076    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 21 17:26:12.507842 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506080    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 21 17:26:12.507842 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506083    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 21 17:26:12.507842 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506086    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 21 17:26:12.507842 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506089    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 21 17:26:12.507842 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506094    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 21 17:26:12.507842 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506096    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 21 17:26:12.507842 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506099    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 21 17:26:12.507842 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506102    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 21 17:26:12.507842 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506104    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 21 17:26:12.507842 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506799    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 21 17:26:12.508334 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506805    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 21 17:26:12.508334 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506808    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 21 17:26:12.508334 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506811    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 21 17:26:12.508334 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506814    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 21 17:26:12.508334 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506817    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 21 17:26:12.508334 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506819    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 21 17:26:12.508334 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506822    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 21 17:26:12.508334 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506824    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 21 17:26:12.508334 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506827    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 21 17:26:12.508334 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506830    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 21 17:26:12.508334 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506833    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 21 17:26:12.508334 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506835    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 21 17:26:12.508334 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506837    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 21 17:26:12.508334 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506840    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 21 17:26:12.508334 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506843    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 21 17:26:12.508334 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506845    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 21 17:26:12.508334 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506847    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 21 17:26:12.508334 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506850    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 21 17:26:12.508334 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506853    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 21 17:26:12.508792 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506856    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 21 17:26:12.508792 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506859    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 21 17:26:12.508792 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506863    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 21 17:26:12.508792 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506866    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 21 17:26:12.508792 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506869    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 21 17:26:12.508792 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506872    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 21 17:26:12.508792 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506875    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 21 17:26:12.508792 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506878    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 21 17:26:12.508792 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506881    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 21 17:26:12.508792 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506884    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 21 17:26:12.508792 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506886    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 21 17:26:12.508792 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506889    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 21 17:26:12.508792 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506892    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 21 17:26:12.508792 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506895    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 21 17:26:12.508792 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506897    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 21 17:26:12.508792 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506899    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 21 17:26:12.508792 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506902    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 21 17:26:12.508792 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506905    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 21 17:26:12.508792 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506908    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 21 17:26:12.509312 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506910    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 21 17:26:12.509312 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506912    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 21 17:26:12.509312 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506916    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 21 17:26:12.509312 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506918    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 21 17:26:12.509312 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506920    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 21 17:26:12.509312 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506923    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 21 17:26:12.509312 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506925    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 21 17:26:12.509312 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506928    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 21 17:26:12.509312 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506930    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 21 17:26:12.509312 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506933    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 21 17:26:12.509312 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506935    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 21 17:26:12.509312 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506938    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 21 17:26:12.509312 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506941    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 21 17:26:12.509312 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506944    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 21 17:26:12.509312 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506946    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 21 17:26:12.509312 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506949    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 21 17:26:12.509312 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506952    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 21 17:26:12.509312 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506954    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 21 17:26:12.509312 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506956    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 21 17:26:12.509763 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506959    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 21 17:26:12.509763 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506961    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 21 17:26:12.509763 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506963    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 21 17:26:12.509763 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506966    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 21 17:26:12.509763 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506968    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 21 17:26:12.509763 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506971    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 21 17:26:12.509763 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506975    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 21 17:26:12.509763 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506977    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 21 17:26:12.509763 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506979    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 21 17:26:12.509763 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506982    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 21 17:26:12.509763 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506984    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 21 17:26:12.509763 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506986    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 21 17:26:12.509763 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506988    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 21 17:26:12.509763 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506991    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 21 17:26:12.509763 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506993    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 21 17:26:12.509763 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506995    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 21 17:26:12.509763 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.506998    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 21 17:26:12.509763 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507000    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 21 17:26:12.509763 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507002    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 21 17:26:12.509763 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507005    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 21 17:26:12.509763 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507007    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 21 17:26:12.510371 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507009    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 21 17:26:12.510371 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507014    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 21 17:26:12.510371 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507017    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 21 17:26:12.510371 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507023    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 21 17:26:12.510371 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507026    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 21 17:26:12.510371 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507029    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 21 17:26:12.510371 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507032    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 21 17:26:12.510371 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507034    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 21 17:26:12.510371 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507037    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 21 17:26:12.510371 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507040    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 21 17:26:12.510371 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507043    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 21 17:26:12.510371 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507045    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 21 17:26:12.510371 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507048    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 21 17:26:12.510371 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507050    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 21 17:26:12.510371 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507053    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 21 17:26:12.510371 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507055    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 21 17:26:12.510371 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507057    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 21 17:26:12.510371 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507060    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 21 17:26:12.510371 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507062    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 21 17:26:12.510371 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507066    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 21 17:26:12.510856 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507462    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 21 17:26:12.510856 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507468    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 21 17:26:12.510856 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507471    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 21 17:26:12.510856 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507474    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 21 17:26:12.510856 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507476    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 21 17:26:12.510856 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507479    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 21 17:26:12.510856 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507482    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 21 17:26:12.510856 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507484    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 21 17:26:12.510856 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507487    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 21 17:26:12.510856 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507489    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 21 17:26:12.510856 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507492    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 21 17:26:12.510856 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507494    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 21 17:26:12.510856 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507496    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 21 17:26:12.510856 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507500    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 21 17:26:12.510856 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507502    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 21 17:26:12.510856 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507509    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 21 17:26:12.510856 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507512    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 21 17:26:12.510856 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507514    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 21 17:26:12.510856 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507516    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 21 17:26:12.510856 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507519    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 21 17:26:12.511352 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507521    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 21 17:26:12.511352 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507524    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 21 17:26:12.511352 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507526    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 21 17:26:12.511352 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507529    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 21 17:26:12.511352 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507531    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 21 17:26:12.511352 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507533    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 21 17:26:12.511352 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507536    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 21 17:26:12.511352 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507538    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 21 17:26:12.511352 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507541    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 21 17:26:12.511352 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507544    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 21 17:26:12.511352 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507546    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 21 17:26:12.511352 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507549    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 21 17:26:12.511352 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507551    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 21 17:26:12.511352 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507553    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 21 17:26:12.511352 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507556    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 21 17:26:12.511352 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507558    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 21 17:26:12.511352 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507560    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 21 17:26:12.511352 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507563    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 21 17:26:12.511352 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507565    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 21 17:26:12.511352 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507567    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 21 17:26:12.511823 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507570    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 21 17:26:12.511823 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507572    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 21 17:26:12.511823 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507575    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 21 17:26:12.511823 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507577    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 21 17:26:12.511823 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507580    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 21 17:26:12.511823 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507582    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 21 17:26:12.511823 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507585    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 21 17:26:12.511823 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507589    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 21 17:26:12.511823 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507591    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 21 17:26:12.511823 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507594    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 21 17:26:12.511823 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507596    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 21 17:26:12.511823 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507599    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 21 17:26:12.511823 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507601    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 21 17:26:12.511823 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507604    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 21 17:26:12.511823 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507606    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 21 17:26:12.511823 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507609    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 21 17:26:12.511823 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507611    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 21 17:26:12.511823 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507615    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 21 17:26:12.511823 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507618    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 21 17:26:12.511823 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507621    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 21 17:26:12.512316 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507624    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 21 17:26:12.512316 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507627    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 21 17:26:12.512316 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507631    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 21 17:26:12.512316 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507634    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 21 17:26:12.512316 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507637    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 21 17:26:12.512316 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507640    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 21 17:26:12.512316 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507643    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 21 17:26:12.512316 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507645    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 21 17:26:12.512316 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507648    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 21 17:26:12.512316 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507650    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 21 17:26:12.512316 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507653    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 21 17:26:12.512316 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507655    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 21 17:26:12.512316 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507658    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 21 17:26:12.512316 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507660    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 21 17:26:12.512316 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507663    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 21 17:26:12.512316 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507665    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 21 17:26:12.512316 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507668    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 21 17:26:12.512316 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507672    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 21 17:26:12.512316 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507674    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 21 17:26:12.512765 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507679    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 21 17:26:12.512765 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507682    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 21 17:26:12.512765 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507685    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 21 17:26:12.512765 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507687    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 21 17:26:12.512765 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507690    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 21 17:26:12.512765 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507692    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 21 17:26:12.512765 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507695    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 21 17:26:12.512765 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507697    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 21 17:26:12.512765 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507700    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 21 17:26:12.512765 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507702    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 21 17:26:12.512765 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507705    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 21 17:26:12.512765 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507707    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 21 17:26:12.512765 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507709    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 21 17:26:12.512765 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507711    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 21 17:26:12.512765 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507714    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 21 17:26:12.512765 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507716    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 21 17:26:12.512765 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507719    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 21 17:26:12.512765 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507721    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 21 17:26:12.512765 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507724    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 21 17:26:12.512765 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.507726    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 21 17:26:12.513265 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507794    2563 flags.go:64] FLAG: --address="0.0.0.0"
Jul 21 17:26:12.513265 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507801    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 21 17:26:12.513265 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507808    2563 flags.go:64] FLAG: --anonymous-auth="true"
Jul 21 17:26:12.513265 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507812    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 21 17:26:12.513265 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507816    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 21 17:26:12.513265 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507820    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 21 17:26:12.513265 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507823    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 21 17:26:12.513265 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507827    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 21 17:26:12.513265 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507830    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 21 17:26:12.513265 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507833    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 21 17:26:12.513265 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507836    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 21 17:26:12.513265 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507839    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 21 17:26:12.513265 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507845    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 21 17:26:12.513265 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507849    2563 flags.go:64] FLAG: --cgroup-root=""
Jul 21 17:26:12.513265 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507852    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 21 17:26:12.513265 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507855    2563 flags.go:64] FLAG: --client-ca-file=""
Jul 21 17:26:12.513265 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507858    2563 flags.go:64] FLAG: --cloud-provider="external"
Jul 21 17:26:12.513265 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507861    2563 flags.go:64] FLAG: --cluster-dns="[]"
Jul 21 17:26:12.513265 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507865    2563 flags.go:64] FLAG: --cluster-domain=""
Jul 21 17:26:12.513265 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507867    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 21 17:26:12.513265 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507870    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 21 17:26:12.513265 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507873    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 21 17:26:12.513265 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507876    2563 flags.go:64] FLAG: --container-log-max-files="5"
Jul 21 17:26:12.513265 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507880    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 21 17:26:12.513851 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507883    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 21 17:26:12.513851 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507886    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 21 17:26:12.513851 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507889    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 21 17:26:12.513851 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507892    2563 flags.go:64] FLAG: --contention-profiling="false"
Jul 21 17:26:12.513851 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507895    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 21 17:26:12.513851 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507898    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 21 17:26:12.513851 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507901    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 21 17:26:12.513851 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507903    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 21 17:26:12.513851 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507907    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 21 17:26:12.513851 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507910    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 21 17:26:12.513851 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507913    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 21 17:26:12.513851 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507915    2563 flags.go:64] FLAG: --enable-load-reader="false"
Jul 21 17:26:12.513851 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507918    2563 flags.go:64] FLAG: --enable-server="true"
Jul 21 17:26:12.513851 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507921    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 21 17:26:12.513851 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507926    2563 flags.go:64] FLAG: --event-burst="100"
Jul 21 17:26:12.513851 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507929    2563 flags.go:64] FLAG: --event-qps="50"
Jul 21 17:26:12.513851 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507932    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 21 17:26:12.513851 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507934    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 21 17:26:12.513851 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507937    2563 flags.go:64] FLAG: --eviction-hard=""
Jul 21 17:26:12.513851 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507941    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 21 17:26:12.513851 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507946    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 21 17:26:12.513851 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507949    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 21 17:26:12.513851 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507953    2563 flags.go:64] FLAG: --eviction-soft=""
Jul 21 17:26:12.513851 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507955    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 21 17:26:12.513851 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507958    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 21 17:26:12.514599 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507961    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 21 17:26:12.514599 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507964    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 21 17:26:12.514599 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507967    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 21 17:26:12.514599 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507969    2563 flags.go:64] FLAG: --fail-swap-on="true"
Jul 21 17:26:12.514599 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507972    2563 flags.go:64] FLAG: --feature-gates=""
Jul 21 17:26:12.514599 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507976    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 21 17:26:12.514599 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507978    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 21 17:26:12.514599 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507981    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 21 17:26:12.514599 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507984    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 21 17:26:12.514599 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507987    2563 flags.go:64] FLAG: --healthz-port="10248"
Jul 21 17:26:12.514599 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507990    2563 flags.go:64] FLAG: --help="false"
Jul 21 17:26:12.514599 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507992    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-132-249.ec2.internal"
Jul 21 17:26:12.514599 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507995    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 21 17:26:12.514599 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.507998    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 21 17:26:12.514599 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508001    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 21 17:26:12.514599 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508004    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 21 17:26:12.514599 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508007    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 21 17:26:12.514599 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508010    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 21 17:26:12.514599 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508012    2563 flags.go:64] FLAG: --image-service-endpoint=""
Jul 21 17:26:12.514599 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508015    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 21 17:26:12.514599 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508017    2563 flags.go:64] FLAG: --kube-api-burst="100"
Jul 21 17:26:12.514599 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508020    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 21 17:26:12.514599 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508023    2563 flags.go:64] FLAG: --kube-api-qps="50"
Jul 21 17:26:12.514599 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508026    2563 flags.go:64] FLAG: --kube-reserved=""
Jul 21 17:26:12.515654 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508028    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 21 17:26:12.515654 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508031    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 21 17:26:12.515654 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508034    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 21 17:26:12.515654 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508038    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 21 17:26:12.515654 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508041    2563 flags.go:64] FLAG: --lock-file=""
Jul 21 17:26:12.515654 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508044    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 21 17:26:12.515654 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508047    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 21 17:26:12.515654 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508050    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 21 17:26:12.515654 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508055    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 21 17:26:12.515654 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508058    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 21 17:26:12.515654 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508060    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 21 17:26:12.515654 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508063    2563 flags.go:64] FLAG: --logging-format="text"
Jul 21 17:26:12.515654 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508066    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 21 17:26:12.515654 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508069    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 21 17:26:12.515654 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508072    2563 flags.go:64] FLAG: --manifest-url=""
Jul 21 17:26:12.515654 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508074    2563 flags.go:64] FLAG: --manifest-url-header=""
Jul 21 17:26:12.515654 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508083    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 21 17:26:12.515654 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508085    2563 flags.go:64] FLAG: --max-open-files="1000000"
Jul 21 17:26:12.515654 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508089    2563 flags.go:64] FLAG: --max-pods="110"
Jul 21 17:26:12.515654 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508092    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 21 17:26:12.515654 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508095    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 21 17:26:12.515654 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508098    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 21 17:26:12.515654 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508101    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 21 17:26:12.515654 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508103    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 21 17:26:12.515654 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508106    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 21 17:26:12.516720 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508109    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 21 17:26:12.516720 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508116    2563 flags.go:64] FLAG: --node-status-max-images="50"
Jul 21 17:26:12.516720 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508118    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 21 17:26:12.516720 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508121    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 21 17:26:12.516720 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508124    2563 flags.go:64] FLAG: --pod-cidr=""
Jul 21 17:26:12.516720 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508126    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 21 17:26:12.516720 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508148    2563 flags.go:64] FLAG: --pod-manifest-path=""
Jul 21 17:26:12.516720 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508151    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 21 17:26:12.516720 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508153    2563 flags.go:64] FLAG: --pods-per-core="0"
Jul 21 17:26:12.516720 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508156    2563 flags.go:64] FLAG: --port="10250"
Jul 21 17:26:12.516720 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508160    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 21 17:26:12.516720 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508164    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00a15965abfbd5f6b"
Jul 21 17:26:12.516720 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508167    2563 flags.go:64] FLAG: --qos-reserved=""
Jul 21 17:26:12.516720 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508170    2563 flags.go:64] FLAG: --read-only-port="10255"
Jul 21 17:26:12.516720 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508173    2563 flags.go:64] FLAG: --register-node="true"
Jul 21 17:26:12.516720 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508176    2563 flags.go:64] FLAG: --register-with-taints=""
Jul 21 17:26:12.516720 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508180    2563 flags.go:64] FLAG: --registry-burst="10"
Jul 21 17:26:12.516720 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508183    2563 flags.go:64] FLAG: --registry-qps="5"
Jul 21 17:26:12.516720 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508185    2563 flags.go:64] FLAG: --reserved-cpus=""
Jul 21 17:26:12.516720 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508188    2563 flags.go:64] FLAG: --reserved-memory=""
Jul 21 17:26:12.516720 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508191    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 21 17:26:12.516720 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508194    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 21 17:26:12.516720 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508197    2563 flags.go:64] FLAG: --rotate-certificates="false"
Jul 21 17:26:12.516720 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508199    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 21 17:26:12.516720 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508202    2563 flags.go:64] FLAG: --runonce="false"
Jul 21 17:26:12.517530 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508205    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 21 17:26:12.517530 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508208    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 21 17:26:12.517530 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508210    2563 flags.go:64] FLAG: --seccomp-default="false"
Jul 21 17:26:12.517530 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508213    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 21 17:26:12.517530 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508216    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 21 17:26:12.517530 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508218    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 21 17:26:12.517530 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508221    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 21 17:26:12.517530 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508224    2563 flags.go:64] FLAG: --storage-driver-password="root"
Jul 21 17:26:12.517530 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508227    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 21 17:26:12.517530 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508230    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 21 17:26:12.517530 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508232    2563 flags.go:64] FLAG: --storage-driver-user="root"
Jul 21 17:26:12.517530 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508235    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 21 17:26:12.517530 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508238    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 21 17:26:12.517530 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508240    2563 flags.go:64] FLAG: --system-cgroups=""
Jul 21 17:26:12.517530 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508243    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 21 17:26:12.517530 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508248    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 21 17:26:12.517530 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508250    2563 flags.go:64] FLAG: --tls-cert-file=""
Jul 21 17:26:12.517530 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508255    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 21 17:26:12.517530 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508260    2563 flags.go:64] FLAG: --tls-min-version=""
Jul 21 17:26:12.517530 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508263    2563 flags.go:64] FLAG: --tls-private-key-file=""
Jul 21 17:26:12.517530 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508265    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 21 17:26:12.517530 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508268    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 21 17:26:12.517530 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508271    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 21 17:26:12.517530 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508274    2563 flags.go:64] FLAG: --v="2"
Jul 21 17:26:12.517530 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508278    2563 flags.go:64] FLAG: --version="false"
Jul 21 17:26:12.518149 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508282    2563 flags.go:64] FLAG: --vmodule=""
Jul 21 17:26:12.518149 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508285    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 21 17:26:12.518149 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.508288    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 21 17:26:12.518149 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508397    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 21 17:26:12.518149 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508400    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 21 17:26:12.518149 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508403    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 21 17:26:12.518149 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508405    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 21 17:26:12.518149 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508408    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 21 17:26:12.518149 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508410    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 21 17:26:12.518149 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508413    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 21 17:26:12.518149 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508415    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 21 17:26:12.518149 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508418    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 21 17:26:12.518149 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508420    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 21 17:26:12.518149 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508423    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 21 17:26:12.518149 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508425    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 21 17:26:12.518149 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508427    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 21 17:26:12.518149 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508430    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 21 17:26:12.518149 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508432    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 21 17:26:12.518149 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508434    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 21 17:26:12.518149 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508437    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 21 17:26:12.518651 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508440    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 21 17:26:12.518651 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508442    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 21 17:26:12.518651 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508445    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 21 17:26:12.518651 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508447    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 21 17:26:12.518651 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508451    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 21 17:26:12.518651 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508455    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 21 17:26:12.518651 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508457    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 21 17:26:12.518651 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508460    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 21 17:26:12.518651 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508462    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 21 17:26:12.518651 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508465    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 21 17:26:12.518651 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508468    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 21 17:26:12.518651 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508472    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 21 17:26:12.518651 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508475    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 21 17:26:12.518651 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508478    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 21 17:26:12.518651 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508481    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 21 17:26:12.518651 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508484    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 21 17:26:12.518651 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508487    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 21 17:26:12.518651 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508490    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 21 17:26:12.518651 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508492    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 21 17:26:12.518651 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508495    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 21 17:26:12.519178 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508498    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 21 17:26:12.519178 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508501    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 21 17:26:12.519178 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508503    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 21 17:26:12.519178 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508505    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 21 17:26:12.519178 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508508    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 21 17:26:12.519178 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508510    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 21 17:26:12.519178 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508513    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 21 17:26:12.519178 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508515    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 21 17:26:12.519178 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508518    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 21 17:26:12.519178 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508520    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 21 17:26:12.519178 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508523    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 21 17:26:12.519178 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508525    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 21 17:26:12.519178 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508528    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 21 17:26:12.519178 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508530    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 21 17:26:12.519178 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508532    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 21 17:26:12.519178 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508535    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 21 17:26:12.519178 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508539    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 21 17:26:12.519178 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508544    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 21 17:26:12.519178 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508547    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 21 17:26:12.519178 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508550    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 21 17:26:12.519656 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508552    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 21 17:26:12.519656 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508556    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 21 17:26:12.519656 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508558    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 21 17:26:12.519656 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508561    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 21 17:26:12.519656 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508564    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 21 17:26:12.519656 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508567    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 21 17:26:12.519656 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508569    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 21 17:26:12.519656 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508572    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 21 17:26:12.519656 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508574    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 21 17:26:12.519656 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508576    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 21 17:26:12.519656 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508579    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 21 17:26:12.519656 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508581    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 21 17:26:12.519656 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508584    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 21 17:26:12.519656 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508586    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 21 17:26:12.519656 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508588    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 21 17:26:12.519656 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508591    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 21 17:26:12.519656 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508594    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 21 17:26:12.519656 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508596    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 21 17:26:12.519656 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508598    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 21 17:26:12.520125 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508601    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 21 17:26:12.520125 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508603    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 21 17:26:12.520125 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508606    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 21 17:26:12.520125 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508608    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 21 17:26:12.520125 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508611    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 21 17:26:12.520125 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508613    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 21 17:26:12.520125 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508616    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 21 17:26:12.520125 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508618    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 21 17:26:12.520125 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508620    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 21 17:26:12.520125 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508624    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 21 17:26:12.520125 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508627    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 21 17:26:12.520125 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508631    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 21 17:26:12.520125 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508634    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 21 17:26:12.520125 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508637    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 21 17:26:12.520125 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508640    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 21 17:26:12.520125 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508642    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 21 17:26:12.520125 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508645    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 21 17:26:12.520125 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508648    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 21 17:26:12.520125 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508650    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 21 17:26:12.520606 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508653    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 21 17:26:12.520606 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508655    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 21 17:26:12.520606 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508657    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 21 17:26:12.520606 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.508660    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 21 17:26:12.520606 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.509254    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 21 17:26:12.520606 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.517567    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 21 17:26:12.520606 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.517583    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 21 17:26:12.520606 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.518378    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 21 17:26:12.520606 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.518385    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 21 17:26:12.520606 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518445    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 21 17:26:12.520606 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518450    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 21 17:26:12.520606 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518453    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 21 17:26:12.520606 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518457    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 21 17:26:12.520606 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518460    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 21 17:26:12.520606 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518463    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 21 17:26:12.520987 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518466    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 21 17:26:12.520987 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518469    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 21 17:26:12.520987 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518473    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 21 17:26:12.520987 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518477    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 21 17:26:12.520987 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518479    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 21 17:26:12.520987 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518482    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 21 17:26:12.520987 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518485    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 21 17:26:12.520987 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518487    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 21 17:26:12.520987 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518490    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 21 17:26:12.520987 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518493    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 21 17:26:12.520987 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518495    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 21 17:26:12.520987 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518498    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 21 17:26:12.520987 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518501    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 21 17:26:12.520987 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518503    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 21 17:26:12.520987 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518506    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 21 17:26:12.520987 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518508    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 21 17:26:12.520987 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518511    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 21 17:26:12.520987 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518514    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 21 17:26:12.520987 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518516    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 21 17:26:12.520987 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518519    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 21 17:26:12.521611 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518521    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 21 17:26:12.521611 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518524    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 21 17:26:12.521611 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518527    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 21 17:26:12.521611 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518530    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 21 17:26:12.521611 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518532    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 21 17:26:12.521611 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518535    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 21 17:26:12.521611 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518538    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 21 17:26:12.521611 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518540    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 21 17:26:12.521611 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518542    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 21 17:26:12.521611 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518545    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 21 17:26:12.521611 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518548    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 21 17:26:12.521611 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518550    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 21 17:26:12.521611 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518553    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 21 17:26:12.521611 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518555    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 21 17:26:12.521611 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518557    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 21 17:26:12.521611 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518560    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 21 17:26:12.521611 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518562    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 21 17:26:12.521611 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518565    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 21 17:26:12.521611 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518568    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 21 17:26:12.521611 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518572    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 21 17:26:12.522109 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518575    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 21 17:26:12.522109 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518578    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 21 17:26:12.522109 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518582    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 21 17:26:12.522109 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518584    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 21 17:26:12.522109 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518588    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 21 17:26:12.522109 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518590    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 21 17:26:12.522109 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518593    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 21 17:26:12.522109 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518595    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 21 17:26:12.522109 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518598    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 21 17:26:12.522109 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518601    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 21 17:26:12.522109 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518604    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 21 17:26:12.522109 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518607    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 21 17:26:12.522109 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518610    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 21 17:26:12.522109 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518614    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 21 17:26:12.522109 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518616    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 21 17:26:12.522109 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518620    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 21 17:26:12.522109 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518623    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 21 17:26:12.522109 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518625    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 21 17:26:12.522109 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518628    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 21 17:26:12.522589 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518631    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 21 17:26:12.522589 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518634    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 21 17:26:12.522589 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518637    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 21 17:26:12.522589 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518639    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 21 17:26:12.522589 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518641    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 21 17:26:12.522589 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518644    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 21 17:26:12.522589 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518646    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 21 17:26:12.522589 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518649    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 21 17:26:12.522589 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518651    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 21 17:26:12.522589 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518654    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 21 17:26:12.522589 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518656    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 21 17:26:12.522589 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518659    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 21 17:26:12.522589 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518662    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 21 17:26:12.522589 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518665    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 21 17:26:12.522589 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518668    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 21 17:26:12.522589 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518671    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 21 17:26:12.522589 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518674    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 21 17:26:12.522589 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518676    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 21 17:26:12.522589 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518679    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 21 17:26:12.522589 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518682    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 21 17:26:12.523119 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518684    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 21 17:26:12.523119 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518687    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 21 17:26:12.523119 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518689    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 21 17:26:12.523119 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518691    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 21 17:26:12.523119 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518694    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 21 17:26:12.523119 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518696    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 21 17:26:12.523119 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518698    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 21 17:26:12.523119 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518700    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 21 17:26:12.523119 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518703    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 21 17:26:12.523119 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518706    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 21 17:26:12.523119 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518708    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 21 17:26:12.523119 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518710    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 21 17:26:12.523119 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518713    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 21 17:26:12.523119 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518715    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 21 17:26:12.523119 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.518721    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 21 17:26:12.523520 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518836    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 21 17:26:12.523520 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518841    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 21 17:26:12.523520 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518844    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 21 17:26:12.523520 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518847    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 21 17:26:12.523520 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518849    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 21 17:26:12.523520 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518852    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 21 17:26:12.523520 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518855    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 21 17:26:12.523520 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518857    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 21 17:26:12.523520 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518860    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 21 17:26:12.523520 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518862    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 21 17:26:12.523520 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518865    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 21 17:26:12.523520 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518867    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 21 17:26:12.523520 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518870    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 21 17:26:12.523520 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518873    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 21 17:26:12.523520 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518875    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 21 17:26:12.523520 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518878    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 21 17:26:12.523520 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518880    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 21 17:26:12.523520 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518883    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 21 17:26:12.523520 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518886    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 21 17:26:12.523520 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518888    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 21 17:26:12.523993 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518891    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 21 17:26:12.523993 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518895    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 21 17:26:12.523993 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518899    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 21 17:26:12.523993 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518902    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 21 17:26:12.523993 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518904    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 21 17:26:12.523993 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518907    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 21 17:26:12.523993 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518909    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 21 17:26:12.523993 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518911    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 21 17:26:12.523993 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518914    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 21 17:26:12.523993 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518916    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 21 17:26:12.523993 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518919    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 21 17:26:12.523993 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518921    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 21 17:26:12.523993 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518924    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 21 17:26:12.523993 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518926    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 21 17:26:12.523993 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518928    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 21 17:26:12.523993 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518931    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 21 17:26:12.523993 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518933    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 21 17:26:12.523993 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518936    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 21 17:26:12.523993 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518938    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 21 17:26:12.524481 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518940    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 21 17:26:12.524481 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518943    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 21 17:26:12.524481 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518945    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 21 17:26:12.524481 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518947    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 21 17:26:12.524481 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518950    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 21 17:26:12.524481 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518952    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 21 17:26:12.524481 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518955    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 21 17:26:12.524481 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518958    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 21 17:26:12.524481 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518961    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 21 17:26:12.524481 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518964    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 21 17:26:12.524481 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518966    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 21 17:26:12.524481 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518968    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 21 17:26:12.524481 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518971    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 21 17:26:12.524481 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518973    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 21 17:26:12.524481 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518975    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 21 17:26:12.524481 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518978    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 21 17:26:12.524481 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518980    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 21 17:26:12.524481 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518983    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 21 17:26:12.524481 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518985    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 21 17:26:12.524943 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518988    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 21 17:26:12.524943 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518990    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 21 17:26:12.524943 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518992    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 21 17:26:12.524943 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518996    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 21 17:26:12.524943 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.518999    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 21 17:26:12.524943 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.519002    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 21 17:26:12.524943 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.519005    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 21 17:26:12.524943 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.519007    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 21 17:26:12.524943 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.519010    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 21 17:26:12.524943 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.519012    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 21 17:26:12.524943 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.519015    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 21 17:26:12.524943 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.519017    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 21 17:26:12.524943 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.519019    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 21 17:26:12.524943 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.519022    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 21 17:26:12.524943 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.519024    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 21 17:26:12.524943 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.519026    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 21 17:26:12.524943 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.519029    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 21 17:26:12.524943 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.519031    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 21 17:26:12.524943 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.519035    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 21 17:26:12.525441 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.519038    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 21 17:26:12.525441 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.519041    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 21 17:26:12.525441 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.519044    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 21 17:26:12.525441 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.519047    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 21 17:26:12.525441 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.519050    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 21 17:26:12.525441 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.519052    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 21 17:26:12.525441 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.519054    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 21 17:26:12.525441 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.519057    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 21 17:26:12.525441 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.519059    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 21 17:26:12.525441 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.519061    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 21 17:26:12.525441 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.519065    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 21 17:26:12.525441 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.519067    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 21 17:26:12.525441 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.519069    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 21 17:26:12.525441 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.519072    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 21 17:26:12.525441 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.519074    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 21 17:26:12.525441 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.519077    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 21 17:26:12.525441 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.519079    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 21 17:26:12.525441 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.519082    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 21 17:26:12.525441 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.519084    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 21 17:26:12.525441 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.519086    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 21 17:26:12.525926 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.519089    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 21 17:26:12.525926 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:12.519091    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 21 17:26:12.525926 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.519096    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 21 17:26:12.525926 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.519225    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 21 17:26:12.525926 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.521990    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 21 17:26:12.525926 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.523451    2563 server.go:1019] "Starting client certificate rotation"
Jul 21 17:26:12.525926 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.523555    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 21 17:26:12.525926 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.523594    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 21 17:26:12.525926 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.525744    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 21 17:26:12.528850 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.528829    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 21 17:26:12.539339 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.539318    2563 log.go:25] "Validated CRI v1 runtime API"
Jul 21 17:26:12.542600 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.542583    2563 log.go:25] "Validated CRI v1 image API"
Jul 21 17:26:12.543444 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.543428    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 21 17:26:12.545840 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.545817    2563 fs.go:135] Filesystem UUIDs: map[5ba214e8-faa6-46b8-b35a-a9708c10de92:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 abb439e4-7dbc-47b2-a72c-beef954cea74:/dev/nvme0n1p4]
Jul 21 17:26:12.545897 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.545841    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 21 17:26:12.552677 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.552557    2563 manager.go:216] Machine: {Timestamp:2026-07-21 17:26:12.551121393 +0000 UTC m=+0.313260989 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3133148 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec274bcafe7ed2fcebe2ef0273b8290d SystemUUID:ec274bca-fe7e-d2fc-ebe2-ef0273b8290d BootID:7dc789d3-fa71-4d83-bae1-07010f813493 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:79:db:9e:92:47 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:79:db:9e:92:47 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c6:04:a3:04:f9:83 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 21 17:26:12.552677 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.552670    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 21 17:26:12.552834 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.552818    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 21 17:26:12.552932 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.552918    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 21 17:26:12.556638 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.556616    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 21 17:26:12.556780 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.556642    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-249.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 21 17:26:12.557385 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.557373    2563 topology_manager.go:138] "Creating topology manager with none policy"
Jul 21 17:26:12.557425 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.557387    2563 container_manager_linux.go:309] "Creating device plugin manager"
Jul 21 17:26:12.557425 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.557395    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 21 17:26:12.558198 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.558187    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 21 17:26:12.559491 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.559480    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 21 17:26:12.562364 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.562353    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 21 17:26:12.562665 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.562654    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 21 17:26:12.562702 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.562684    2563 kubelet.go:486] "Attempting to sync node with API server"
Jul 21 17:26:12.562702 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.562699    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 21 17:26:12.562766 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.562758    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 21 17:26:12.562797 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.562770    2563 kubelet.go:398] "Adding apiserver pod source"
Jul 21 17:26:12.562797 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.562779    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 21 17:26:12.563997 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.563979    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 21 17:26:12.564095 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.564009    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 21 17:26:12.567259 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.567242    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 21 17:26:12.568866 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.568850    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 21 17:26:12.568924 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.568895    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 21 17:26:12.569859 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.569847    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 21 17:26:12.569902 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.569865    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 21 17:26:12.569902 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.569872    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 21 17:26:12.569902 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.569879    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 21 17:26:12.569902 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.569892    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 21 17:26:12.569902 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.569900    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 21 17:26:12.570033 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.569909    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 21 17:26:12.570033 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.569915    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 21 17:26:12.570033 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.569922    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 21 17:26:12.570033 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.569928    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 21 17:26:12.570033 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.569936    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 21 17:26:12.570033 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.569942    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 21 17:26:12.571357 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.571347    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 21 17:26:12.571391 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.571361    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 21 17:26:12.572554 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.572538    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cfc85"
Jul 21 17:26:12.573434 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.573414    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-249.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 21 17:26:12.573494 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:12.573473    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 21 17:26:12.573764 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:12.573745    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-249.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 21 17:26:12.574911 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.574900    2563 server.go:1268] "Started kubelet"
Jul 21 17:26:12.574992 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.574971    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 21 17:26:12.575045 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.574974    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 21 17:26:12.575045 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.575039    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 21 17:26:12.575919 ip-10-0-132-249 systemd[1]: Started Kubernetes Kubelet.
Jul 21 17:26:12.576992 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.576977    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 21 17:26:12.577303 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.577291    2563 server.go:310] "Adding debug handlers to kubelet server"
Jul 21 17:26:12.580822 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:12.580805    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 21 17:26:12.581088 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.581070    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cfc85"
Jul 21 17:26:12.582372 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.582357    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 21 17:26:12.582704 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:12.582443    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-249.ec2.internal\" not found"
Jul 21 17:26:12.582704 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.582635    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 21 17:26:12.582874 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.582860    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 21 17:26:12.583145 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.583113    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 21 17:26:12.583211 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.583149    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 21 17:26:12.583262 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.583231    2563 factory.go:55] Registering systemd factory
Jul 21 17:26:12.583262 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.583246    2563 factory.go:223] Registration of the systemd container factory successfully
Jul 21 17:26:12.583362 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.583305    2563 reconstruct.go:97] "Volume reconstruction finished"
Jul 21 17:26:12.583362 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.583315    2563 reconciler.go:29] "Reconciler: start to sync state"
Jul 21 17:26:12.583661 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.583639    2563 factory.go:153] Registering CRI-O factory
Jul 21 17:26:12.583661 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.583656    2563 factory.go:223] Registration of the crio container factory successfully
Jul 21 17:26:12.583781 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.583706    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 21 17:26:12.583781 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.583731    2563 factory.go:103] Registering Raw factory
Jul 21 17:26:12.583781 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.583748    2563 manager.go:1196] Started watching for new ooms in manager
Jul 21 17:26:12.584097 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.584082    2563 manager.go:349] Starting recovery of all containers
Jul 21 17:26:12.584961 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:12.583743    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-249.ec2.internal.18c45e0f6607d004  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-249.ec2.internal,UID:ip-10-0-132-249.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-249.ec2.internal,},FirstTimestamp:2026-07-21 17:26:12.57486746 +0000 UTC m=+0.337007055,LastTimestamp:2026-07-21 17:26:12.57486746 +0000 UTC m=+0.337007055,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-249.ec2.internal,}"
Jul 21 17:26:12.589646 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.589457    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 21 17:26:12.595067 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.595011    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 21 17:26:12.596334 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.596284    2563 manager.go:354] Recovery completed
Jul 21 17:26:12.597323 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:12.597300    2563 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-132-249.ec2.internal\" not found" node="ip-10-0-132-249.ec2.internal"
Jul 21 17:26:12.598633 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:12.598587    2563 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 21 17:26:12.601677 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.601663    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 21 17:26:12.603967 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.603952    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-249.ec2.internal" event="NodeHasSufficientMemory"
Jul 21 17:26:12.604015 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.603982    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-249.ec2.internal" event="NodeHasNoDiskPressure"
Jul 21 17:26:12.604015 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.603993    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-249.ec2.internal" event="NodeHasSufficientPID"
Jul 21 17:26:12.604544 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.604531    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 21 17:26:12.604544 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.604538    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 21 17:26:12.604544 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.604543    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 21 17:26:12.604646 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.604561    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 21 17:26:12.605292 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.605277    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-249.ec2.internal" event="NodeHasSufficientMemory"
Jul 21 17:26:12.605348 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.605304    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-249.ec2.internal" event="NodeHasNoDiskPressure"
Jul 21 17:26:12.605348 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.605314    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-249.ec2.internal" event="NodeHasSufficientPID"
Jul 21 17:26:12.605758 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.605747    2563 policy_none.go:49] "None policy: Start"
Jul 21 17:26:12.605789 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.605762    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 21 17:26:12.605789 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.605773    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 21 17:26:12.606744 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.606733    2563 policy_none.go:47] "Start"
Jul 21 17:26:12.646747 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.646730    2563 manager.go:337] "Starting Device Plugin manager"
Jul 21 17:26:12.646862 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:12.646764    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 21 17:26:12.646862 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.646774    2563 server.go:87] "Starting device plugin registration server"
Jul 21 17:26:12.647041 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.647029    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 21 17:26:12.647078 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.647044    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 21 17:26:12.647308 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.647166    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 21 17:26:12.647308 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.647249    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 21 17:26:12.647308 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.647261    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 21 17:26:12.647822 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:12.647805    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 21 17:26:12.647878 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:12.647849    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-249.ec2.internal\" not found"
Jul 21 17:26:12.747643 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.747602    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 21 17:26:12.748844 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.748799    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-249.ec2.internal" event="NodeHasSufficientMemory"
Jul 21 17:26:12.748844 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.748836    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-249.ec2.internal" event="NodeHasNoDiskPressure"
Jul 21 17:26:12.748969 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.748851    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-249.ec2.internal" event="NodeHasSufficientPID"
Jul 21 17:26:12.748969 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.748880    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-249.ec2.internal"
Jul 21 17:26:12.757238 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.757117    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-249.ec2.internal"
Jul 21 17:26:12.757351 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:12.757261    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-249.ec2.internal\": node \"ip-10-0-132-249.ec2.internal\" not found"
Jul 21 17:26:12.758474 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.758457    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 21 17:26:12.758515 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.758494    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 21 17:26:12.758515 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.758514    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 21 17:26:12.758574 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:12.758550    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 21 17:26:12.760640 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.760620    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 21 17:26:12.792190 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:12.792155    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-249.ec2.internal\" not found"
Jul 21 17:26:12.859835 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.859218    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-249.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-249.ec2.internal"]
Jul 21 17:26:12.859926 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.859879    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 21 17:26:12.861176 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.861157    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-249.ec2.internal" event="NodeHasSufficientMemory"
Jul 21 17:26:12.861281 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.861187    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-249.ec2.internal" event="NodeHasNoDiskPressure"
Jul 21 17:26:12.861281 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.861199    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-249.ec2.internal" event="NodeHasSufficientPID"
Jul 21 17:26:12.862555 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.862544    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 21 17:26:12.862660 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.862644    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-249.ec2.internal"
Jul 21 17:26:12.862712 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.862676    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 21 17:26:12.863313 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.863296    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-249.ec2.internal" event="NodeHasSufficientMemory"
Jul 21 17:26:12.863374 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.863323    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-249.ec2.internal" event="NodeHasNoDiskPressure"
Jul 21 17:26:12.863374 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.863336    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-249.ec2.internal" event="NodeHasSufficientPID"
Jul 21 17:26:12.863374 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.863342    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-249.ec2.internal" event="NodeHasSufficientMemory"
Jul 21 17:26:12.863374 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.863365    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-249.ec2.internal" event="NodeHasNoDiskPressure"
Jul 21 17:26:12.863374 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.863374    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-249.ec2.internal" event="NodeHasSufficientPID"
Jul 21 17:26:12.864435 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.864423    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-249.ec2.internal"
Jul 21 17:26:12.864486 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.864448    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 21 17:26:12.865165 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.865148    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-249.ec2.internal" event="NodeHasSufficientMemory"
Jul 21 17:26:12.865239 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.865177    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-249.ec2.internal" event="NodeHasNoDiskPressure"
Jul 21 17:26:12.865239 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.865192    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-249.ec2.internal" event="NodeHasSufficientPID"
Jul 21 17:26:12.884938 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.884912    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fb93a1b33c0d6ced6dfb0f4505a58dc6-config\") pod \"kube-apiserver-proxy-ip-10-0-132-249.ec2.internal\" (UID: \"fb93a1b33c0d6ced6dfb0f4505a58dc6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-249.ec2.internal"
Jul 21 17:26:12.885043 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.884940    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/db56e96cb19cb0d0fae9df6533d0a2ea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-249.ec2.internal\" (UID: \"db56e96cb19cb0d0fae9df6533d0a2ea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-249.ec2.internal"
Jul 21 17:26:12.885043 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.884961    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db56e96cb19cb0d0fae9df6533d0a2ea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-249.ec2.internal\" (UID: \"db56e96cb19cb0d0fae9df6533d0a2ea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-249.ec2.internal"
Jul 21 17:26:12.890822 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:12.890806    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-249.ec2.internal\" not found" node="ip-10-0-132-249.ec2.internal"
Jul 21 17:26:12.892787 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:12.892751    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-249.ec2.internal\" not found"
Jul 21 17:26:12.895184 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:12.895168    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-249.ec2.internal\" not found" node="ip-10-0-132-249.ec2.internal"
Jul 21 17:26:12.985344 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.985303    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fb93a1b33c0d6ced6dfb0f4505a58dc6-config\") pod \"kube-apiserver-proxy-ip-10-0-132-249.ec2.internal\" (UID: \"fb93a1b33c0d6ced6dfb0f4505a58dc6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-249.ec2.internal"
Jul 21 17:26:12.985344 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.985346    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/db56e96cb19cb0d0fae9df6533d0a2ea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-249.ec2.internal\" (UID: \"db56e96cb19cb0d0fae9df6533d0a2ea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-249.ec2.internal"
Jul 21 17:26:12.985503 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.985363    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db56e96cb19cb0d0fae9df6533d0a2ea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-249.ec2.internal\" (UID: \"db56e96cb19cb0d0fae9df6533d0a2ea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-249.ec2.internal"
Jul 21 17:26:12.985503 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.985389    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db56e96cb19cb0d0fae9df6533d0a2ea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-249.ec2.internal\" (UID: \"db56e96cb19cb0d0fae9df6533d0a2ea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-249.ec2.internal"
Jul 21 17:26:12.985503 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.985412    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fb93a1b33c0d6ced6dfb0f4505a58dc6-config\") pod \"kube-apiserver-proxy-ip-10-0-132-249.ec2.internal\" (UID: \"fb93a1b33c0d6ced6dfb0f4505a58dc6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-249.ec2.internal"
Jul 21 17:26:12.985503 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:12.985415    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/db56e96cb19cb0d0fae9df6533d0a2ea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-249.ec2.internal\" (UID: \"db56e96cb19cb0d0fae9df6533d0a2ea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-249.ec2.internal"
Jul 21 17:26:12.993123 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:12.993092    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-249.ec2.internal\" not found"
Jul 21 17:26:13.094016 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:13.093928    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-249.ec2.internal\" not found"
Jul 21 17:26:13.193556 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:13.193521    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-249.ec2.internal"
Jul 21 17:26:13.194582 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:13.194561    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-249.ec2.internal\" not found"
Jul 21 17:26:13.198338 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:13.198324    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-249.ec2.internal"
Jul 21 17:26:13.295443 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:13.295412    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-249.ec2.internal\" not found"
Jul 21 17:26:13.395988 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:13.395919    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-249.ec2.internal\" not found"
Jul 21 17:26:13.496438 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:13.496407    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-249.ec2.internal\" not found"
Jul 21 17:26:13.522598 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:13.522559    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 21 17:26:13.523242 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:13.522735    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 21 17:26:13.523242 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:13.522747    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 21 17:26:13.568464 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:13.568431    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 21 17:26:13.583464 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:13.583428    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-249.ec2.internal"
Jul 21 17:26:13.583666 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:13.583446    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 21 17:26:13.586327 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:13.586115    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-20 17:21:13 +0000 UTC" deadline="2028-04-23 14:17:51.964443102 +0000 UTC"
Jul 21 17:26:13.586327 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:13.586157    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15404h51m38.378290086s"
Jul 21 17:26:13.594254 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:13.594232    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 21 17:26:13.596110 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:13.596094    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-249.ec2.internal"
Jul 21 17:26:13.596653 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:13.596636    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 21 17:26:13.602856 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:13.602831    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 21 17:26:13.619591 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:13.619557    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hwkzv"
Jul 21 17:26:13.626303 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:13.626279    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hwkzv"
Jul 21 17:26:13.685430 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:13.685395    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb56e96cb19cb0d0fae9df6533d0a2ea.slice/crio-f04f265ebf143dd2740811a78abd974b73ea457e82cb7c19c00bfd2d12258785 WatchSource:0}: Error finding container f04f265ebf143dd2740811a78abd974b73ea457e82cb7c19c00bfd2d12258785: Status 404 returned error can't find the container with id f04f265ebf143dd2740811a78abd974b73ea457e82cb7c19c00bfd2d12258785
Jul 21 17:26:13.686172 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:13.686154    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb93a1b33c0d6ced6dfb0f4505a58dc6.slice/crio-97473f79903b456152f787fda4320a46284108712a4d86010c1ba7b6e234a03b WatchSource:0}: Error finding container 97473f79903b456152f787fda4320a46284108712a4d86010c1ba7b6e234a03b: Status 404 returned error can't find the container with id 97473f79903b456152f787fda4320a46284108712a4d86010c1ba7b6e234a03b
Jul 21 17:26:13.690650 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:13.690637    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 21 17:26:13.766101 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:13.766042    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-249.ec2.internal" event={"ID":"fb93a1b33c0d6ced6dfb0f4505a58dc6","Type":"ContainerStarted","Data":"97473f79903b456152f787fda4320a46284108712a4d86010c1ba7b6e234a03b"}
Jul 21 17:26:13.767300 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:13.767272    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-249.ec2.internal" event={"ID":"db56e96cb19cb0d0fae9df6533d0a2ea","Type":"ContainerStarted","Data":"f04f265ebf143dd2740811a78abd974b73ea457e82cb7c19c00bfd2d12258785"}
Jul 21 17:26:13.959651 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:13.959557    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 21 17:26:14.564484 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.564448    2563 apiserver.go:52] "Watching apiserver"
Jul 21 17:26:14.577076 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.577051    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 21 17:26:14.578791 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.578442    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-249.ec2.internal","openshift-dns/node-resolver-gxckp","openshift-multus/multus-additional-cni-plugins-5872t","openshift-multus/network-metrics-daemon-sxmzk","openshift-network-diagnostics/network-check-target-d8gnt","openshift-ovn-kubernetes/ovnkube-node-dq7d6","kube-system/global-pull-secret-syncer-g76xh","kube-system/konnectivity-agent-rg2jl","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wng2j","openshift-cluster-node-tuning-operator/tuned-fwcw8","openshift-image-registry/node-ca-mgp2n","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-249.ec2.internal","openshift-multus/multus-zp2df","openshift-network-operator/iptables-alerter-zqxmf"]
Jul 21 17:26:14.580348 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.580081    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:26:14.580348 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:14.580186    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g76xh" podUID="fa6ca187-d1f3-4d33-bd20-f70784dbbe02"
Jul 21 17:26:14.582228 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.582201    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.583790 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.583767    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5872t"
Jul 21 17:26:14.585223 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.585202    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmzk"
Jul 21 17:26:14.585319 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:14.585277    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmzk" podUID="82279305-9475-429f-a495-143fc616fb34"
Jul 21 17:26:14.585699 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.585676    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 21 17:26:14.585817 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.585738    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 21 17:26:14.586135 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.586012    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5mkbx\""
Jul 21 17:26:14.586135 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.586028    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 21 17:26:14.586135 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.586082    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 21 17:26:14.587106 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.586677    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 21 17:26:14.587106 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.586764    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8gnt"
Jul 21 17:26:14.587106 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:14.586834    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8gnt" podUID="b92707b0-e59d-4123-82e2-658e4361cb6e"
Jul 21 17:26:14.587106 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.587000    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 21 17:26:14.587106 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.587007    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hf5rc\""
Jul 21 17:26:14.589055 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.589035    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.591188 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.591166    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wng2j"
Jul 21 17:26:14.591426 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.591407    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 21 17:26:14.591730 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.591714    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nln7t\""
Jul 21 17:26:14.591886 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.591868    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 21 17:26:14.592603 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.592581    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zqxmf"
Jul 21 17:26:14.593044 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.592921    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 21 17:26:14.593760 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.593401    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 21 17:26:14.593760 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.593405    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 21 17:26:14.593760 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.593737    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa6ca187-d1f3-4d33-bd20-f70784dbbe02-original-pull-secret\") pod \"global-pull-secret-syncer-g76xh\" (UID: \"fa6ca187-d1f3-4d33-bd20-f70784dbbe02\") " pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:26:14.593760 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.593758    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2998a9d6-86dc-4ab8-961d-7f873ce559c6-system-cni-dir\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.593994 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.593774    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj86j\" (UniqueName: \"kubernetes.io/projected/2998a9d6-86dc-4ab8-961d-7f873ce559c6-kube-api-access-rj86j\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.593994 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.593797    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35f84d29-3ea1-4c65-b000-d910c8012948-system-cni-dir\") pod \"multus-additional-cni-plugins-5872t\" (UID: \"35f84d29-3ea1-4c65-b000-d910c8012948\") " pod="openshift-multus/multus-additional-cni-plugins-5872t"
Jul 21 17:26:14.593994 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.593842    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2998a9d6-86dc-4ab8-961d-7f873ce559c6-host-run-netns\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.593994 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.593867    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2998a9d6-86dc-4ab8-961d-7f873ce559c6-host-var-lib-cni-bin\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.593994 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.593908    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2998a9d6-86dc-4ab8-961d-7f873ce559c6-host-var-lib-cni-multus\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.593994 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.593927    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2998a9d6-86dc-4ab8-961d-7f873ce559c6-host-var-lib-kubelet\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.593994 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.593981    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/35f84d29-3ea1-4c65-b000-d910c8012948-os-release\") pod \"multus-additional-cni-plugins-5872t\" (UID: \"35f84d29-3ea1-4c65-b000-d910c8012948\") " pod="openshift-multus/multus-additional-cni-plugins-5872t"
Jul 21 17:26:14.594370 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.594021    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fa6ca187-d1f3-4d33-bd20-f70784dbbe02-kubelet-config\") pod \"global-pull-secret-syncer-g76xh\" (UID: \"fa6ca187-d1f3-4d33-bd20-f70784dbbe02\") " pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:26:14.594370 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.594047    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2998a9d6-86dc-4ab8-961d-7f873ce559c6-cnibin\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.594370 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.594072    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2998a9d6-86dc-4ab8-961d-7f873ce559c6-multus-socket-dir-parent\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.594370 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.594098    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2998a9d6-86dc-4ab8-961d-7f873ce559c6-host-run-k8s-cni-cncf-io\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.594370 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.594163    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg9gh\" (UniqueName: \"kubernetes.io/projected/35f84d29-3ea1-4c65-b000-d910c8012948-kube-api-access-wg9gh\") pod \"multus-additional-cni-plugins-5872t\" (UID: \"35f84d29-3ea1-4c65-b000-d910c8012948\") " pod="openshift-multus/multus-additional-cni-plugins-5872t"
Jul 21 17:26:14.594370 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.594176    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 21 17:26:14.594370 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.594192    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2998a9d6-86dc-4ab8-961d-7f873ce559c6-multus-cni-dir\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.594370 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.594220    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2998a9d6-86dc-4ab8-961d-7f873ce559c6-os-release\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.594370 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.594248    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2998a9d6-86dc-4ab8-961d-7f873ce559c6-multus-daemon-config\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.594370 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.594312    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35f84d29-3ea1-4c65-b000-d910c8012948-cni-binary-copy\") pod \"multus-additional-cni-plugins-5872t\" (UID: \"35f84d29-3ea1-4c65-b000-d910c8012948\") " pod="openshift-multus/multus-additional-cni-plugins-5872t"
Jul 21 17:26:14.594370 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.594337    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2998a9d6-86dc-4ab8-961d-7f873ce559c6-hostroot\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.594370 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.594364    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35f84d29-3ea1-4c65-b000-d910c8012948-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5872t\" (UID: \"35f84d29-3ea1-4c65-b000-d910c8012948\") " pod="openshift-multus/multus-additional-cni-plugins-5872t"
Jul 21 17:26:14.594963 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.594391    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/35f84d29-3ea1-4c65-b000-d910c8012948-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5872t\" (UID: \"35f84d29-3ea1-4c65-b000-d910c8012948\") " pod="openshift-multus/multus-additional-cni-plugins-5872t"
Jul 21 17:26:14.594963 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.594435    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2998a9d6-86dc-4ab8-961d-7f873ce559c6-multus-conf-dir\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.594963 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.594441    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 21 17:26:14.594963 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.594458    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2998a9d6-86dc-4ab8-961d-7f873ce559c6-etc-kubernetes\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.594963 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.594492    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fa6ca187-d1f3-4d33-bd20-f70784dbbe02-dbus\") pod \"global-pull-secret-syncer-g76xh\" (UID: \"fa6ca187-d1f3-4d33-bd20-f70784dbbe02\") " pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:26:14.594963 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.594516    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2998a9d6-86dc-4ab8-961d-7f873ce559c6-cni-binary-copy\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.594963 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.594540    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2998a9d6-86dc-4ab8-961d-7f873ce559c6-host-run-multus-certs\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.594963 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.594562    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35f84d29-3ea1-4c65-b000-d910c8012948-cnibin\") pod \"multus-additional-cni-plugins-5872t\" (UID: \"35f84d29-3ea1-4c65-b000-d910c8012948\") " pod="openshift-multus/multus-additional-cni-plugins-5872t"
Jul 21 17:26:14.594963 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.594585    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/35f84d29-3ea1-4c65-b000-d910c8012948-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5872t\" (UID: \"35f84d29-3ea1-4c65-b000-d910c8012948\") " pod="openshift-multus/multus-additional-cni-plugins-5872t"
Jul 21 17:26:14.594963 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.594738    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xkvnl\""
Jul 21 17:26:14.595462 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.595053    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gxckp"
Jul 21 17:26:14.595527 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.595491    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 21 17:26:14.595595 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.595533    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 21 17:26:14.595595 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.595577    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 21 17:26:14.595692 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.595635    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9zr4g\""
Jul 21 17:26:14.598234 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.597085    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 21 17:26:14.598234 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.597456    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.600853 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.598827    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 21 17:26:14.600853 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.599023    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 21 17:26:14.600853 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.599124    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-scbxh\""
Jul 21 17:26:14.600853 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.599325    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 21 17:26:14.600853 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.600619    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 21 17:26:14.601144 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.600948    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 21 17:26:14.601310 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.601291    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-g5xl6\""
Jul 21 17:26:14.603279 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.602317    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mgp2n"
Jul 21 17:26:14.604471 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.604448    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rg2jl"
Jul 21 17:26:14.607143 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.607122    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8fs9n\""
Jul 21 17:26:14.607265 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.607144    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 21 17:26:14.607265 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.607182    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-crsgh\""
Jul 21 17:26:14.607265 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.607213    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 21 17:26:14.607422 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.607267    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 21 17:26:14.607422 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.607343    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 21 17:26:14.607559 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.607540    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 21 17:26:14.628398 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.628310    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 17:21:14 +0000 UTC" deadline="2028-01-17 22:41:03.617323645 +0000 UTC"
Jul 21 17:26:14.628398 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.628399    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13085h14m48.988929618s"
Jul 21 17:26:14.650923 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.650898    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 21 17:26:14.666333 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.666308    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 21 17:26:14.684547 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.684520    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 21 17:26:14.695737 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.695703    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2998a9d6-86dc-4ab8-961d-7f873ce559c6-multus-conf-dir\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.695871 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.695744    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2998a9d6-86dc-4ab8-961d-7f873ce559c6-etc-kubernetes\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.695871 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.695772    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8e07071d-5eed-4b15-844b-8dd2c6bf1eaa-etc-tuned\") pod \"tuned-fwcw8\" (UID: \"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa\") " pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.695871 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.695799    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-host-cni-netd\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.695871 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.695803    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2998a9d6-86dc-4ab8-961d-7f873ce559c6-multus-conf-dir\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.695871 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.695820    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkt5m\" (UniqueName: \"kubernetes.io/projected/a83f9b9f-8087-49ba-a353-ab97de0d1935-kube-api-access-tkt5m\") pod \"aws-ebs-csi-driver-node-wng2j\" (UID: \"a83f9b9f-8087-49ba-a353-ab97de0d1935\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wng2j"
Jul 21 17:26:14.695871 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.695865    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2998a9d6-86dc-4ab8-961d-7f873ce559c6-cni-binary-copy\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.696136 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.695889    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2998a9d6-86dc-4ab8-961d-7f873ce559c6-etc-kubernetes\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.696136 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.695894    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35f84d29-3ea1-4c65-b000-d910c8012948-cnibin\") pod \"multus-additional-cni-plugins-5872t\" (UID: \"35f84d29-3ea1-4c65-b000-d910c8012948\") " pod="openshift-multus/multus-additional-cni-plugins-5872t"
Jul 21 17:26:14.696136 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.695925    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/35f84d29-3ea1-4c65-b000-d910c8012948-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5872t\" (UID: \"35f84d29-3ea1-4c65-b000-d910c8012948\") " pod="openshift-multus/multus-additional-cni-plugins-5872t"
Jul 21 17:26:14.696136 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.695930    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35f84d29-3ea1-4c65-b000-d910c8012948-cnibin\") pod \"multus-additional-cni-plugins-5872t\" (UID: \"35f84d29-3ea1-4c65-b000-d910c8012948\") " pod="openshift-multus/multus-additional-cni-plugins-5872t"
Jul 21 17:26:14.696136 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.695957    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8e07071d-5eed-4b15-844b-8dd2c6bf1eaa-etc-modprobe-d\") pod \"tuned-fwcw8\" (UID: \"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa\") " pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.696136 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.695984    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e07071d-5eed-4b15-844b-8dd2c6bf1eaa-etc-kubernetes\") pod \"tuned-fwcw8\" (UID: \"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa\") " pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.696136 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.696010    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e07071d-5eed-4b15-844b-8dd2c6bf1eaa-var-lib-kubelet\") pod \"tuned-fwcw8\" (UID: \"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa\") " pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.696136 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.696036    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-run-systemd\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.696136 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.696080    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a83f9b9f-8087-49ba-a353-ab97de0d1935-etc-selinux\") pod \"aws-ebs-csi-driver-node-wng2j\" (UID: \"a83f9b9f-8087-49ba-a353-ab97de0d1935\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wng2j"
Jul 21 17:26:14.696136 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.696098    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2998a9d6-86dc-4ab8-961d-7f873ce559c6-system-cni-dir\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.696136 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.696114    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj86j\" (UniqueName: \"kubernetes.io/projected/2998a9d6-86dc-4ab8-961d-7f873ce559c6-kube-api-access-rj86j\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.696136 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.696138    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-host-cni-bin\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.696624 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.696179    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc62g\" (UniqueName: \"kubernetes.io/projected/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-kube-api-access-qc62g\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.696624 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.696203    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8e07071d-5eed-4b15-844b-8dd2c6bf1eaa-etc-sysctl-conf\") pod \"tuned-fwcw8\" (UID: \"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa\") " pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.696624 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.696228    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8e07071d-5eed-4b15-844b-8dd2c6bf1eaa-etc-systemd\") pod \"tuned-fwcw8\" (UID: \"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa\") " pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.696624 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.696251    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vwvp\" (UniqueName: \"kubernetes.io/projected/8e07071d-5eed-4b15-844b-8dd2c6bf1eaa-kube-api-access-4vwvp\") pod \"tuned-fwcw8\" (UID: \"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa\") " pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.696624 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.696291    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5f7ddf4c-27b9-4b30-8d4a-483cfa1fb74a-serviceca\") pod \"node-ca-mgp2n\" (UID: \"5f7ddf4c-27b9-4b30-8d4a-483cfa1fb74a\") " pod="openshift-image-registry/node-ca-mgp2n"
Jul 21 17:26:14.696624 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.696322    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwmzj\" (UniqueName: \"kubernetes.io/projected/5f7ddf4c-27b9-4b30-8d4a-483cfa1fb74a-kube-api-access-kwmzj\") pod \"node-ca-mgp2n\" (UID: \"5f7ddf4c-27b9-4b30-8d4a-483cfa1fb74a\") " pod="openshift-image-registry/node-ca-mgp2n"
Jul 21 17:26:14.696624 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.696346    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-var-lib-openvswitch\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.696624 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.696369    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a83f9b9f-8087-49ba-a353-ab97de0d1935-socket-dir\") pod \"aws-ebs-csi-driver-node-wng2j\" (UID: \"a83f9b9f-8087-49ba-a353-ab97de0d1935\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wng2j"
Jul 21 17:26:14.696624 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.696394    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2998a9d6-86dc-4ab8-961d-7f873ce559c6-host-var-lib-kubelet\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.696624 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.696418    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82279305-9475-429f-a495-143fc616fb34-metrics-certs\") pod \"network-metrics-daemon-sxmzk\" (UID: \"82279305-9475-429f-a495-143fc616fb34\") " pod="openshift-multus/network-metrics-daemon-sxmzk"
Jul 21 17:26:14.696624 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.696473    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-log-socket\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.696624 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.696500    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-host-run-ovn-kubernetes\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.696624 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.696204    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2998a9d6-86dc-4ab8-961d-7f873ce559c6-system-cni-dir\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.696624 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.696525    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-ovn-node-metrics-cert\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.696624 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.696548    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2998a9d6-86dc-4ab8-961d-7f873ce559c6-cni-binary-copy\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.696624 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.696575    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2998a9d6-86dc-4ab8-961d-7f873ce559c6-host-var-lib-kubelet\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.696624 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.696548    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/35f84d29-3ea1-4c65-b000-d910c8012948-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5872t\" (UID: \"35f84d29-3ea1-4c65-b000-d910c8012948\") " pod="openshift-multus/multus-additional-cni-plugins-5872t"
Jul 21 17:26:14.697357 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.696552    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2998a9d6-86dc-4ab8-961d-7f873ce559c6-host-run-k8s-cni-cncf-io\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.697357 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.696603    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2998a9d6-86dc-4ab8-961d-7f873ce559c6-host-run-k8s-cni-cncf-io\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.697357 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.696628    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg9gh\" (UniqueName: \"kubernetes.io/projected/35f84d29-3ea1-4c65-b000-d910c8012948-kube-api-access-wg9gh\") pod \"multus-additional-cni-plugins-5872t\" (UID: \"35f84d29-3ea1-4c65-b000-d910c8012948\") " pod="openshift-multus/multus-additional-cni-plugins-5872t"
Jul 21 17:26:14.697357 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.696651    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8e07071d-5eed-4b15-844b-8dd2c6bf1eaa-lib-modules\") pod \"tuned-fwcw8\" (UID: \"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa\") " pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.697357 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.696674    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2998a9d6-86dc-4ab8-961d-7f873ce559c6-os-release\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.697357 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.696704    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5401269-8543-4227-a538-95e2f3051c4b-host-slash\") pod \"iptables-alerter-zqxmf\" (UID: \"c5401269-8543-4227-a538-95e2f3051c4b\") " pod="openshift-network-operator/iptables-alerter-zqxmf"
Jul 21 17:26:14.697357 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.696746    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8e07071d-5eed-4b15-844b-8dd2c6bf1eaa-etc-sysconfig\") pod \"tuned-fwcw8\" (UID: \"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa\") " pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.697357 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.696769    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e07071d-5eed-4b15-844b-8dd2c6bf1eaa-sys\") pod \"tuned-fwcw8\" (UID: \"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa\") " pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.697357 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.696794    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-node-log\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.697357 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.696770    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2998a9d6-86dc-4ab8-961d-7f873ce559c6-os-release\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.697357 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.696817    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a83f9b9f-8087-49ba-a353-ab97de0d1935-sys-fs\") pod \"aws-ebs-csi-driver-node-wng2j\" (UID: \"a83f9b9f-8087-49ba-a353-ab97de0d1935\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wng2j"
Jul 21 17:26:14.697357 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.696847    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2998a9d6-86dc-4ab8-961d-7f873ce559c6-hostroot\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.697357 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.696875    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2998a9d6-86dc-4ab8-961d-7f873ce559c6-hostroot\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.697357 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.696912    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/35f84d29-3ea1-4c65-b000-d910c8012948-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5872t\" (UID: \"35f84d29-3ea1-4c65-b000-d910c8012948\") " pod="openshift-multus/multus-additional-cni-plugins-5872t"
Jul 21 17:26:14.697357 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.696941    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rmqx\" (UniqueName: \"kubernetes.io/projected/c5401269-8543-4227-a538-95e2f3051c4b-kube-api-access-2rmqx\") pod \"iptables-alerter-zqxmf\" (UID: \"c5401269-8543-4227-a538-95e2f3051c4b\") " pod="openshift-network-operator/iptables-alerter-zqxmf"
Jul 21 17:26:14.697357 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.696975    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8e07071d-5eed-4b15-844b-8dd2c6bf1eaa-etc-sysctl-d\") pod \"tuned-fwcw8\" (UID: \"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa\") " pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.697357 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.697000    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-host-slash\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.698045 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.697023    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-run-openvswitch\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.698045 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.697045    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a83f9b9f-8087-49ba-a353-ab97de0d1935-registration-dir\") pod \"aws-ebs-csi-driver-node-wng2j\" (UID: \"a83f9b9f-8087-49ba-a353-ab97de0d1935\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wng2j"
Jul 21 17:26:14.698045 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.697069    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f7ddf4c-27b9-4b30-8d4a-483cfa1fb74a-host\") pod \"node-ca-mgp2n\" (UID: \"5f7ddf4c-27b9-4b30-8d4a-483cfa1fb74a\") " pod="openshift-image-registry/node-ca-mgp2n"
Jul 21 17:26:14.698045 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.697089    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-systemd-units\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.698045 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.697113    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-etc-openvswitch\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.698045 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.697139    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fa6ca187-d1f3-4d33-bd20-f70784dbbe02-dbus\") pod \"global-pull-secret-syncer-g76xh\" (UID: \"fa6ca187-d1f3-4d33-bd20-f70784dbbe02\") " pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:26:14.698045 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.697186    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2998a9d6-86dc-4ab8-961d-7f873ce559c6-host-run-multus-certs\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.698045 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.697213    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st8r9\" (UniqueName: \"kubernetes.io/projected/189f8b1f-23e2-4447-90bd-ffa91affc6c4-kube-api-access-st8r9\") pod \"node-resolver-gxckp\" (UID: \"189f8b1f-23e2-4447-90bd-ffa91affc6c4\") " pod="openshift-dns/node-resolver-gxckp"
Jul 21 17:26:14.698045 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.697239    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-env-overrides\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.698045 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.697269    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa6ca187-d1f3-4d33-bd20-f70784dbbe02-original-pull-secret\") pod \"global-pull-secret-syncer-g76xh\" (UID: \"fa6ca187-d1f3-4d33-bd20-f70784dbbe02\") " pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:26:14.698045 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.697294    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35f84d29-3ea1-4c65-b000-d910c8012948-system-cni-dir\") pod \"multus-additional-cni-plugins-5872t\" (UID: \"35f84d29-3ea1-4c65-b000-d910c8012948\") " pod="openshift-multus/multus-additional-cni-plugins-5872t"
Jul 21 17:26:14.698045 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.697322    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5jzs\" (UniqueName: \"kubernetes.io/projected/82279305-9475-429f-a495-143fc616fb34-kube-api-access-p5jzs\") pod \"network-metrics-daemon-sxmzk\" (UID: \"82279305-9475-429f-a495-143fc616fb34\") " pod="openshift-multus/network-metrics-daemon-sxmzk"
Jul 21 17:26:14.698045 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.697349    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8e07071d-5eed-4b15-844b-8dd2c6bf1eaa-run\") pod \"tuned-fwcw8\" (UID: \"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa\") " pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.698045 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.697375    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8e07071d-5eed-4b15-844b-8dd2c6bf1eaa-host\") pod \"tuned-fwcw8\" (UID: \"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa\") " pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.698045 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.697402    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a83f9b9f-8087-49ba-a353-ab97de0d1935-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wng2j\" (UID: \"a83f9b9f-8087-49ba-a353-ab97de0d1935\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wng2j"
Jul 21 17:26:14.698045 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.697428    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1b38cc54-ebe3-48aa-b1ca-671f5f427f93-agent-certs\") pod \"konnectivity-agent-rg2jl\" (UID: \"1b38cc54-ebe3-48aa-b1ca-671f5f427f93\") " pod="kube-system/konnectivity-agent-rg2jl"
Jul 21 17:26:14.698045 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.697453    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1b38cc54-ebe3-48aa-b1ca-671f5f427f93-konnectivity-ca\") pod \"konnectivity-agent-rg2jl\" (UID: \"1b38cc54-ebe3-48aa-b1ca-671f5f427f93\") " pod="kube-system/konnectivity-agent-rg2jl"
Jul 21 17:26:14.698721 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.697472    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/35f84d29-3ea1-4c65-b000-d910c8012948-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5872t\" (UID: \"35f84d29-3ea1-4c65-b000-d910c8012948\") " pod="openshift-multus/multus-additional-cni-plugins-5872t"
Jul 21 17:26:14.698721 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.697481    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2998a9d6-86dc-4ab8-961d-7f873ce559c6-host-run-netns\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.698721 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.697509    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2998a9d6-86dc-4ab8-961d-7f873ce559c6-host-var-lib-cni-bin\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.698721 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.697573    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2998a9d6-86dc-4ab8-961d-7f873ce559c6-host-var-lib-cni-bin\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.698721 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:14.697675    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 21 17:26:14.698721 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.697722    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2998a9d6-86dc-4ab8-961d-7f873ce559c6-host-run-multus-certs\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.698721 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:14.697767    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa6ca187-d1f3-4d33-bd20-f70784dbbe02-original-pull-secret podName:fa6ca187-d1f3-4d33-bd20-f70784dbbe02 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:15.197727051 +0000 UTC m=+2.959866638 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa6ca187-d1f3-4d33-bd20-f70784dbbe02-original-pull-secret") pod "global-pull-secret-syncer-g76xh" (UID: "fa6ca187-d1f3-4d33-bd20-f70784dbbe02") : object "kube-system"/"original-pull-secret" not registered
Jul 21 17:26:14.698721 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.697794    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35f84d29-3ea1-4c65-b000-d910c8012948-system-cni-dir\") pod \"multus-additional-cni-plugins-5872t\" (UID: \"35f84d29-3ea1-4c65-b000-d910c8012948\") " pod="openshift-multus/multus-additional-cni-plugins-5872t"
Jul 21 17:26:14.698721 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.697827    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2998a9d6-86dc-4ab8-961d-7f873ce559c6-host-run-netns\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.698721 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.697676    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fa6ca187-d1f3-4d33-bd20-f70784dbbe02-dbus\") pod \"global-pull-secret-syncer-g76xh\" (UID: \"fa6ca187-d1f3-4d33-bd20-f70784dbbe02\") " pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:26:14.698721 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.697859    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2998a9d6-86dc-4ab8-961d-7f873ce559c6-host-var-lib-cni-multus\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.698721 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.697888    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n2xt\" (UniqueName: \"kubernetes.io/projected/b92707b0-e59d-4123-82e2-658e4361cb6e-kube-api-access-9n2xt\") pod \"network-check-target-d8gnt\" (UID: \"b92707b0-e59d-4123-82e2-658e4361cb6e\") " pod="openshift-network-diagnostics/network-check-target-d8gnt"
Jul 21 17:26:14.698721 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.697912    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-host-kubelet\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.698721 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.697943    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-run-ovn\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.698721 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.697968    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/35f84d29-3ea1-4c65-b000-d910c8012948-os-release\") pod \"multus-additional-cni-plugins-5872t\" (UID: \"35f84d29-3ea1-4c65-b000-d910c8012948\") " pod="openshift-multus/multus-additional-cni-plugins-5872t"
Jul 21 17:26:14.698721 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.697993    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fa6ca187-d1f3-4d33-bd20-f70784dbbe02-kubelet-config\") pod \"global-pull-secret-syncer-g76xh\" (UID: \"fa6ca187-d1f3-4d33-bd20-f70784dbbe02\") " pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:26:14.698721 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.698018    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2998a9d6-86dc-4ab8-961d-7f873ce559c6-cnibin\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.699333 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.698032    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2998a9d6-86dc-4ab8-961d-7f873ce559c6-host-var-lib-cni-multus\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.699333 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.698043    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2998a9d6-86dc-4ab8-961d-7f873ce559c6-multus-socket-dir-parent\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.699333 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.698102    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fa6ca187-d1f3-4d33-bd20-f70784dbbe02-kubelet-config\") pod \"global-pull-secret-syncer-g76xh\" (UID: \"fa6ca187-d1f3-4d33-bd20-f70784dbbe02\") " pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:26:14.699333 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.698116    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2998a9d6-86dc-4ab8-961d-7f873ce559c6-multus-socket-dir-parent\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.699333 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.698134    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e07071d-5eed-4b15-844b-8dd2c6bf1eaa-tmp\") pod \"tuned-fwcw8\" (UID: \"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa\") " pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.699333 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.698163    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-host-run-netns\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.699333 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.698174    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/35f84d29-3ea1-4c65-b000-d910c8012948-os-release\") pod \"multus-additional-cni-plugins-5872t\" (UID: \"35f84d29-3ea1-4c65-b000-d910c8012948\") " pod="openshift-multus/multus-additional-cni-plugins-5872t"
Jul 21 17:26:14.699333 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.698195    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a83f9b9f-8087-49ba-a353-ab97de0d1935-device-dir\") pod \"aws-ebs-csi-driver-node-wng2j\" (UID: \"a83f9b9f-8087-49ba-a353-ab97de0d1935\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wng2j"
Jul 21 17:26:14.699333 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.698248    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2998a9d6-86dc-4ab8-961d-7f873ce559c6-multus-cni-dir\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.699333 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.698255    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2998a9d6-86dc-4ab8-961d-7f873ce559c6-cnibin\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.699333 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.698283    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2998a9d6-86dc-4ab8-961d-7f873ce559c6-multus-daemon-config\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.699333 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.698312    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35f84d29-3ea1-4c65-b000-d910c8012948-cni-binary-copy\") pod \"multus-additional-cni-plugins-5872t\" (UID: \"35f84d29-3ea1-4c65-b000-d910c8012948\") " pod="openshift-multus/multus-additional-cni-plugins-5872t"
Jul 21 17:26:14.699333 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.698313    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2998a9d6-86dc-4ab8-961d-7f873ce559c6-multus-cni-dir\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.699333 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.698353    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c5401269-8543-4227-a538-95e2f3051c4b-iptables-alerter-script\") pod \"iptables-alerter-zqxmf\" (UID: \"c5401269-8543-4227-a538-95e2f3051c4b\") " pod="openshift-network-operator/iptables-alerter-zqxmf"
Jul 21 17:26:14.699333 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.698378    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/189f8b1f-23e2-4447-90bd-ffa91affc6c4-tmp-dir\") pod \"node-resolver-gxckp\" (UID: \"189f8b1f-23e2-4447-90bd-ffa91affc6c4\") " pod="openshift-dns/node-resolver-gxckp"
Jul 21 17:26:14.699333 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.698410    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.699333 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.698436    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-ovnkube-config\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.699939 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.698465    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35f84d29-3ea1-4c65-b000-d910c8012948-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5872t\" (UID: \"35f84d29-3ea1-4c65-b000-d910c8012948\") " pod="openshift-multus/multus-additional-cni-plugins-5872t"
Jul 21 17:26:14.699939 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.698490    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/189f8b1f-23e2-4447-90bd-ffa91affc6c4-hosts-file\") pod \"node-resolver-gxckp\" (UID: \"189f8b1f-23e2-4447-90bd-ffa91affc6c4\") " pod="openshift-dns/node-resolver-gxckp"
Jul 21 17:26:14.699939 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.698536    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-ovnkube-script-lib\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.699939 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.698798    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35f84d29-3ea1-4c65-b000-d910c8012948-cni-binary-copy\") pod \"multus-additional-cni-plugins-5872t\" (UID: \"35f84d29-3ea1-4c65-b000-d910c8012948\") " pod="openshift-multus/multus-additional-cni-plugins-5872t"
Jul 21 17:26:14.699939 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.698969    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35f84d29-3ea1-4c65-b000-d910c8012948-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5872t\" (UID: \"35f84d29-3ea1-4c65-b000-d910c8012948\") " pod="openshift-multus/multus-additional-cni-plugins-5872t"
Jul 21 17:26:14.699939 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.699037    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2998a9d6-86dc-4ab8-961d-7f873ce559c6-multus-daemon-config\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.705302 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.705265    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 21 17:26:14.709076 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.709054    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg9gh\" (UniqueName: \"kubernetes.io/projected/35f84d29-3ea1-4c65-b000-d910c8012948-kube-api-access-wg9gh\") pod \"multus-additional-cni-plugins-5872t\" (UID: \"35f84d29-3ea1-4c65-b000-d910c8012948\") " pod="openshift-multus/multus-additional-cni-plugins-5872t"
Jul 21 17:26:14.709497 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.709479    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj86j\" (UniqueName: \"kubernetes.io/projected/2998a9d6-86dc-4ab8-961d-7f873ce559c6-kube-api-access-rj86j\") pod \"multus-zp2df\" (UID: \"2998a9d6-86dc-4ab8-961d-7f873ce559c6\") " pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.799223 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.799184    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st8r9\" (UniqueName: \"kubernetes.io/projected/189f8b1f-23e2-4447-90bd-ffa91affc6c4-kube-api-access-st8r9\") pod \"node-resolver-gxckp\" (UID: \"189f8b1f-23e2-4447-90bd-ffa91affc6c4\") " pod="openshift-dns/node-resolver-gxckp"
Jul 21 17:26:14.799436 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.799240    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-env-overrides\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.799436 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.799289    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5jzs\" (UniqueName: \"kubernetes.io/projected/82279305-9475-429f-a495-143fc616fb34-kube-api-access-p5jzs\") pod \"network-metrics-daemon-sxmzk\" (UID: \"82279305-9475-429f-a495-143fc616fb34\") " pod="openshift-multus/network-metrics-daemon-sxmzk"
Jul 21 17:26:14.799436 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.799318    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8e07071d-5eed-4b15-844b-8dd2c6bf1eaa-run\") pod \"tuned-fwcw8\" (UID: \"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa\") " pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.799436 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.799350    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8e07071d-5eed-4b15-844b-8dd2c6bf1eaa-host\") pod \"tuned-fwcw8\" (UID: \"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa\") " pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.799436 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.799378    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a83f9b9f-8087-49ba-a353-ab97de0d1935-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wng2j\" (UID: \"a83f9b9f-8087-49ba-a353-ab97de0d1935\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wng2j"
Jul 21 17:26:14.799436 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.799426    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1b38cc54-ebe3-48aa-b1ca-671f5f427f93-agent-certs\") pod \"konnectivity-agent-rg2jl\" (UID: \"1b38cc54-ebe3-48aa-b1ca-671f5f427f93\") " pod="kube-system/konnectivity-agent-rg2jl"
Jul 21 17:26:14.799667 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.799440    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8e07071d-5eed-4b15-844b-8dd2c6bf1eaa-run\") pod \"tuned-fwcw8\" (UID: \"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa\") " pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.799667 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.799448    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1b38cc54-ebe3-48aa-b1ca-671f5f427f93-konnectivity-ca\") pod \"konnectivity-agent-rg2jl\" (UID: \"1b38cc54-ebe3-48aa-b1ca-671f5f427f93\") " pod="kube-system/konnectivity-agent-rg2jl"
Jul 21 17:26:14.799667 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.799508    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8e07071d-5eed-4b15-844b-8dd2c6bf1eaa-host\") pod \"tuned-fwcw8\" (UID: \"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa\") " pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.799667 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.799547    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n2xt\" (UniqueName: \"kubernetes.io/projected/b92707b0-e59d-4123-82e2-658e4361cb6e-kube-api-access-9n2xt\") pod \"network-check-target-d8gnt\" (UID: \"b92707b0-e59d-4123-82e2-658e4361cb6e\") " pod="openshift-network-diagnostics/network-check-target-d8gnt"
Jul 21 17:26:14.799667 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.799582    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-host-kubelet\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.799667 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.799610    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-run-ovn\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.799667 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.799640    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e07071d-5eed-4b15-844b-8dd2c6bf1eaa-tmp\") pod \"tuned-fwcw8\" (UID: \"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa\") " pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.799667 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.799663    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-host-run-netns\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.799991 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.799689    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a83f9b9f-8087-49ba-a353-ab97de0d1935-device-dir\") pod \"aws-ebs-csi-driver-node-wng2j\" (UID: \"a83f9b9f-8087-49ba-a353-ab97de0d1935\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wng2j"
Jul 21 17:26:14.799991 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.799717    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c5401269-8543-4227-a538-95e2f3051c4b-iptables-alerter-script\") pod \"iptables-alerter-zqxmf\" (UID: \"c5401269-8543-4227-a538-95e2f3051c4b\") " pod="openshift-network-operator/iptables-alerter-zqxmf"
Jul 21 17:26:14.799991 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.799741    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/189f8b1f-23e2-4447-90bd-ffa91affc6c4-tmp-dir\") pod \"node-resolver-gxckp\" (UID: \"189f8b1f-23e2-4447-90bd-ffa91affc6c4\") " pod="openshift-dns/node-resolver-gxckp"
Jul 21 17:26:14.799991 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.799767    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.799991 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.799791    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-ovnkube-config\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.799991 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.799817    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/189f8b1f-23e2-4447-90bd-ffa91affc6c4-hosts-file\") pod \"node-resolver-gxckp\" (UID: \"189f8b1f-23e2-4447-90bd-ffa91affc6c4\") " pod="openshift-dns/node-resolver-gxckp"
Jul 21 17:26:14.799991 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.799828    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a83f9b9f-8087-49ba-a353-ab97de0d1935-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wng2j\" (UID: \"a83f9b9f-8087-49ba-a353-ab97de0d1935\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wng2j"
Jul 21 17:26:14.799991 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.799843    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-ovnkube-script-lib\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.799991 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.799882    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8e07071d-5eed-4b15-844b-8dd2c6bf1eaa-etc-tuned\") pod \"tuned-fwcw8\" (UID: \"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa\") " pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.799991 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.799893    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.799991 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.799907    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-host-cni-netd\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.799991 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.799934    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a83f9b9f-8087-49ba-a353-ab97de0d1935-device-dir\") pod \"aws-ebs-csi-driver-node-wng2j\" (UID: \"a83f9b9f-8087-49ba-a353-ab97de0d1935\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wng2j"
Jul 21 17:26:14.799991 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.799957    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-host-cni-netd\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.800599 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.800168    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1b38cc54-ebe3-48aa-b1ca-671f5f427f93-konnectivity-ca\") pod \"konnectivity-agent-rg2jl\" (UID: \"1b38cc54-ebe3-48aa-b1ca-671f5f427f93\") " pod="kube-system/konnectivity-agent-rg2jl"
Jul 21 17:26:14.800599 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.800234    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/189f8b1f-23e2-4447-90bd-ffa91affc6c4-tmp-dir\") pod \"node-resolver-gxckp\" (UID: \"189f8b1f-23e2-4447-90bd-ffa91affc6c4\") " pod="openshift-dns/node-resolver-gxckp"
Jul 21 17:26:14.800599 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.800261    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-host-run-netns\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.800599 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.800524    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-ovnkube-config\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.800599 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.800547    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkt5m\" (UniqueName: \"kubernetes.io/projected/a83f9b9f-8087-49ba-a353-ab97de0d1935-kube-api-access-tkt5m\") pod \"aws-ebs-csi-driver-node-wng2j\" (UID: \"a83f9b9f-8087-49ba-a353-ab97de0d1935\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wng2j"
Jul 21 17:26:14.800599 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.800528    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c5401269-8543-4227-a538-95e2f3051c4b-iptables-alerter-script\") pod \"iptables-alerter-zqxmf\" (UID: \"c5401269-8543-4227-a538-95e2f3051c4b\") " pod="openshift-network-operator/iptables-alerter-zqxmf"
Jul 21 17:26:14.800599 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.799880    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-env-overrides\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.801134 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.800612    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-run-ovn\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.801134 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.800677    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/189f8b1f-23e2-4447-90bd-ffa91affc6c4-hosts-file\") pod \"node-resolver-gxckp\" (UID: \"189f8b1f-23e2-4447-90bd-ffa91affc6c4\") " pod="openshift-dns/node-resolver-gxckp"
Jul 21 17:26:14.801134 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.800712    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8e07071d-5eed-4b15-844b-8dd2c6bf1eaa-etc-modprobe-d\") pod \"tuned-fwcw8\" (UID: \"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa\") " pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.801134 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.800738    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-ovnkube-script-lib\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.801134 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.800741    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e07071d-5eed-4b15-844b-8dd2c6bf1eaa-etc-kubernetes\") pod \"tuned-fwcw8\" (UID: \"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa\") " pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.801134 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.800782    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e07071d-5eed-4b15-844b-8dd2c6bf1eaa-var-lib-kubelet\") pod \"tuned-fwcw8\" (UID: \"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa\") " pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.801134 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.800807    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-run-systemd\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.801134 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.800828    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e07071d-5eed-4b15-844b-8dd2c6bf1eaa-etc-kubernetes\") pod \"tuned-fwcw8\" (UID: \"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa\") " pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.801134 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.800832    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a83f9b9f-8087-49ba-a353-ab97de0d1935-etc-selinux\") pod \"aws-ebs-csi-driver-node-wng2j\" (UID: \"a83f9b9f-8087-49ba-a353-ab97de0d1935\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wng2j"
Jul 21 17:26:14.801134 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.800862    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-host-cni-bin\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.801134 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.800866    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8e07071d-5eed-4b15-844b-8dd2c6bf1eaa-etc-modprobe-d\") pod \"tuned-fwcw8\" (UID: \"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa\") " pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.801134 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.800879    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e07071d-5eed-4b15-844b-8dd2c6bf1eaa-var-lib-kubelet\") pod \"tuned-fwcw8\" (UID: \"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa\") " pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.801134 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.800888    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc62g\" (UniqueName: \"kubernetes.io/projected/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-kube-api-access-qc62g\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.801134 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.800903    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-host-kubelet\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.801134 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.800913    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8e07071d-5eed-4b15-844b-8dd2c6bf1eaa-etc-sysctl-conf\") pod \"tuned-fwcw8\" (UID: \"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa\") " pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.801134 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.800931    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-host-cni-bin\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.801134 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.800935    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8e07071d-5eed-4b15-844b-8dd2c6bf1eaa-etc-systemd\") pod \"tuned-fwcw8\" (UID: \"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa\") " pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.801134 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.800956    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-run-systemd\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.802057 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.800957    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vwvp\" (UniqueName: \"kubernetes.io/projected/8e07071d-5eed-4b15-844b-8dd2c6bf1eaa-kube-api-access-4vwvp\") pod \"tuned-fwcw8\" (UID: \"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa\") " pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.802057 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.800982    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5f7ddf4c-27b9-4b30-8d4a-483cfa1fb74a-serviceca\") pod \"node-ca-mgp2n\" (UID: \"5f7ddf4c-27b9-4b30-8d4a-483cfa1fb74a\") " pod="openshift-image-registry/node-ca-mgp2n"
Jul 21 17:26:14.802057 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.800989    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a83f9b9f-8087-49ba-a353-ab97de0d1935-etc-selinux\") pod \"aws-ebs-csi-driver-node-wng2j\" (UID: \"a83f9b9f-8087-49ba-a353-ab97de0d1935\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wng2j"
Jul 21 17:26:14.802057 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.801007    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwmzj\" (UniqueName: \"kubernetes.io/projected/5f7ddf4c-27b9-4b30-8d4a-483cfa1fb74a-kube-api-access-kwmzj\") pod \"node-ca-mgp2n\" (UID: \"5f7ddf4c-27b9-4b30-8d4a-483cfa1fb74a\") " pod="openshift-image-registry/node-ca-mgp2n"
Jul 21 17:26:14.802057 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.801032    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-var-lib-openvswitch\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.802057 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.801047    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8e07071d-5eed-4b15-844b-8dd2c6bf1eaa-etc-sysctl-conf\") pod \"tuned-fwcw8\" (UID: \"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa\") " pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.802057 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.801057    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a83f9b9f-8087-49ba-a353-ab97de0d1935-socket-dir\") pod \"aws-ebs-csi-driver-node-wng2j\" (UID: \"a83f9b9f-8087-49ba-a353-ab97de0d1935\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wng2j"
Jul 21 17:26:14.802057 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.801085    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82279305-9475-429f-a495-143fc616fb34-metrics-certs\") pod \"network-metrics-daemon-sxmzk\" (UID: \"82279305-9475-429f-a495-143fc616fb34\") " pod="openshift-multus/network-metrics-daemon-sxmzk"
Jul 21 17:26:14.802057 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.801108    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-log-socket\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.802057 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.801131    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-host-run-ovn-kubernetes\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.802057 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.801158    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-ovn-node-metrics-cert\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.802057 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.801168    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-var-lib-openvswitch\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.802057 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.801186    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8e07071d-5eed-4b15-844b-8dd2c6bf1eaa-lib-modules\") pod \"tuned-fwcw8\" (UID: \"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa\") " pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.802057 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.801236    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5401269-8543-4227-a538-95e2f3051c4b-host-slash\") pod \"iptables-alerter-zqxmf\" (UID: \"c5401269-8543-4227-a538-95e2f3051c4b\") " pod="openshift-network-operator/iptables-alerter-zqxmf"
Jul 21 17:26:14.802057 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.801282    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5401269-8543-4227-a538-95e2f3051c4b-host-slash\") pod \"iptables-alerter-zqxmf\" (UID: \"c5401269-8543-4227-a538-95e2f3051c4b\") " pod="openshift-network-operator/iptables-alerter-zqxmf"
Jul 21 17:26:14.802057 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.801287    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8e07071d-5eed-4b15-844b-8dd2c6bf1eaa-etc-systemd\") pod \"tuned-fwcw8\" (UID: \"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa\") " pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.802057 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.801318    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8e07071d-5eed-4b15-844b-8dd2c6bf1eaa-etc-sysconfig\") pod \"tuned-fwcw8\" (UID: \"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa\") " pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.802847 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.801342    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e07071d-5eed-4b15-844b-8dd2c6bf1eaa-sys\") pod \"tuned-fwcw8\" (UID: \"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa\") " pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.802847 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.801377    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-node-log\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.802847 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.801391    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a83f9b9f-8087-49ba-a353-ab97de0d1935-socket-dir\") pod \"aws-ebs-csi-driver-node-wng2j\" (UID: \"a83f9b9f-8087-49ba-a353-ab97de0d1935\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wng2j"
Jul 21 17:26:14.802847 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.801404    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a83f9b9f-8087-49ba-a353-ab97de0d1935-sys-fs\") pod \"aws-ebs-csi-driver-node-wng2j\" (UID: \"a83f9b9f-8087-49ba-a353-ab97de0d1935\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wng2j"
Jul 21 17:26:14.802847 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.801434    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rmqx\" (UniqueName: \"kubernetes.io/projected/c5401269-8543-4227-a538-95e2f3051c4b-kube-api-access-2rmqx\") pod \"iptables-alerter-zqxmf\" (UID: \"c5401269-8543-4227-a538-95e2f3051c4b\") " pod="openshift-network-operator/iptables-alerter-zqxmf"
Jul 21 17:26:14.802847 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.801458    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8e07071d-5eed-4b15-844b-8dd2c6bf1eaa-etc-sysctl-d\") pod \"tuned-fwcw8\" (UID: \"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa\") " pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.802847 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:14.801472    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:26:14.802847 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.801483    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-host-slash\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.802847 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.801505    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5f7ddf4c-27b9-4b30-8d4a-483cfa1fb74a-serviceca\") pod \"node-ca-mgp2n\" (UID: \"5f7ddf4c-27b9-4b30-8d4a-483cfa1fb74a\") " pod="openshift-image-registry/node-ca-mgp2n"
Jul 21 17:26:14.802847 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.801509    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-run-openvswitch\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.802847 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:14.801526    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82279305-9475-429f-a495-143fc616fb34-metrics-certs podName:82279305-9475-429f-a495-143fc616fb34 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:15.301508182 +0000 UTC m=+3.063647781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82279305-9475-429f-a495-143fc616fb34-metrics-certs") pod "network-metrics-daemon-sxmzk" (UID: "82279305-9475-429f-a495-143fc616fb34") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:26:14.802847 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.801556    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-run-openvswitch\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.802847 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.801561    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-log-socket\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.802847 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.801606    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8e07071d-5eed-4b15-844b-8dd2c6bf1eaa-etc-sysconfig\") pod \"tuned-fwcw8\" (UID: \"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa\") " pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.802847 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.801674    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8e07071d-5eed-4b15-844b-8dd2c6bf1eaa-lib-modules\") pod \"tuned-fwcw8\" (UID: \"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa\") " pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.802847 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.801820    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e07071d-5eed-4b15-844b-8dd2c6bf1eaa-sys\") pod \"tuned-fwcw8\" (UID: \"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa\") " pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.802847 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.801853    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-node-log\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.803533 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.801890    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a83f9b9f-8087-49ba-a353-ab97de0d1935-sys-fs\") pod \"aws-ebs-csi-driver-node-wng2j\" (UID: \"a83f9b9f-8087-49ba-a353-ab97de0d1935\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wng2j"
Jul 21 17:26:14.803533 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.801907    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a83f9b9f-8087-49ba-a353-ab97de0d1935-registration-dir\") pod \"aws-ebs-csi-driver-node-wng2j\" (UID: \"a83f9b9f-8087-49ba-a353-ab97de0d1935\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wng2j"
Jul 21 17:26:14.803533 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.801922    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-host-slash\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.803533 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.801937    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f7ddf4c-27b9-4b30-8d4a-483cfa1fb74a-host\") pod \"node-ca-mgp2n\" (UID: \"5f7ddf4c-27b9-4b30-8d4a-483cfa1fb74a\") " pod="openshift-image-registry/node-ca-mgp2n"
Jul 21 17:26:14.803533 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.801964    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-systemd-units\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.803533 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.801990    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-etc-openvswitch\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.803533 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.802033    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8e07071d-5eed-4b15-844b-8dd2c6bf1eaa-etc-sysctl-d\") pod \"tuned-fwcw8\" (UID: \"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa\") " pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.803533 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.802051    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-etc-openvswitch\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.803533 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.802084    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a83f9b9f-8087-49ba-a353-ab97de0d1935-registration-dir\") pod \"aws-ebs-csi-driver-node-wng2j\" (UID: \"a83f9b9f-8087-49ba-a353-ab97de0d1935\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wng2j"
Jul 21 17:26:14.803533 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.802093    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f7ddf4c-27b9-4b30-8d4a-483cfa1fb74a-host\") pod \"node-ca-mgp2n\" (UID: \"5f7ddf4c-27b9-4b30-8d4a-483cfa1fb74a\") " pod="openshift-image-registry/node-ca-mgp2n"
Jul 21 17:26:14.803533 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.802131    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-systemd-units\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.803533 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.802135    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-host-run-ovn-kubernetes\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.803533 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.802699    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e07071d-5eed-4b15-844b-8dd2c6bf1eaa-tmp\") pod \"tuned-fwcw8\" (UID: \"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa\") " pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.803533 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.802843    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1b38cc54-ebe3-48aa-b1ca-671f5f427f93-agent-certs\") pod \"konnectivity-agent-rg2jl\" (UID: \"1b38cc54-ebe3-48aa-b1ca-671f5f427f93\") " pod="kube-system/konnectivity-agent-rg2jl"
Jul 21 17:26:14.803533 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.802885    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8e07071d-5eed-4b15-844b-8dd2c6bf1eaa-etc-tuned\") pod \"tuned-fwcw8\" (UID: \"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa\") " pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.804594 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.804569    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-ovn-node-metrics-cert\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.809104 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.809079    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5jzs\" (UniqueName: \"kubernetes.io/projected/82279305-9475-429f-a495-143fc616fb34-kube-api-access-p5jzs\") pod \"network-metrics-daemon-sxmzk\" (UID: \"82279305-9475-429f-a495-143fc616fb34\") " pod="openshift-multus/network-metrics-daemon-sxmzk"
Jul 21 17:26:14.812014 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:14.811980    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 21 17:26:14.812014 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:14.812003    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 21 17:26:14.812014 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:14.812017    2563 projected.go:196] Error preparing data for projected volume kube-api-access-9n2xt for pod openshift-network-diagnostics/network-check-target-d8gnt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:26:14.812235 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:14.812080    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b92707b0-e59d-4123-82e2-658e4361cb6e-kube-api-access-9n2xt podName:b92707b0-e59d-4123-82e2-658e4361cb6e nodeName:}" failed. No retries permitted until 2026-07-21 17:26:15.31206729 +0000 UTC m=+3.074206875 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9n2xt" (UniqueName: "kubernetes.io/projected/b92707b0-e59d-4123-82e2-658e4361cb6e-kube-api-access-9n2xt") pod "network-check-target-d8gnt" (UID: "b92707b0-e59d-4123-82e2-658e4361cb6e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:26:14.812776 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.812750    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-st8r9\" (UniqueName: \"kubernetes.io/projected/189f8b1f-23e2-4447-90bd-ffa91affc6c4-kube-api-access-st8r9\") pod \"node-resolver-gxckp\" (UID: \"189f8b1f-23e2-4447-90bd-ffa91affc6c4\") " pod="openshift-dns/node-resolver-gxckp"
Jul 21 17:26:14.814921 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.814860    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vwvp\" (UniqueName: \"kubernetes.io/projected/8e07071d-5eed-4b15-844b-8dd2c6bf1eaa-kube-api-access-4vwvp\") pod \"tuned-fwcw8\" (UID: \"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa\") " pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.815144 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.815122    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkt5m\" (UniqueName: \"kubernetes.io/projected/a83f9b9f-8087-49ba-a353-ab97de0d1935-kube-api-access-tkt5m\") pod \"aws-ebs-csi-driver-node-wng2j\" (UID: \"a83f9b9f-8087-49ba-a353-ab97de0d1935\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wng2j"
Jul 21 17:26:14.815262 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.815243    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc62g\" (UniqueName: \"kubernetes.io/projected/a61ba40a-d3c2-45a9-8525-5e40c9de49b6-kube-api-access-qc62g\") pod \"ovnkube-node-dq7d6\" (UID: \"a61ba40a-d3c2-45a9-8525-5e40c9de49b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.815431 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.815416    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rmqx\" (UniqueName: \"kubernetes.io/projected/c5401269-8543-4227-a538-95e2f3051c4b-kube-api-access-2rmqx\") pod \"iptables-alerter-zqxmf\" (UID: \"c5401269-8543-4227-a538-95e2f3051c4b\") " pod="openshift-network-operator/iptables-alerter-zqxmf"
Jul 21 17:26:14.816319 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.816292    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwmzj\" (UniqueName: \"kubernetes.io/projected/5f7ddf4c-27b9-4b30-8d4a-483cfa1fb74a-kube-api-access-kwmzj\") pod \"node-ca-mgp2n\" (UID: \"5f7ddf4c-27b9-4b30-8d4a-483cfa1fb74a\") " pod="openshift-image-registry/node-ca-mgp2n"
Jul 21 17:26:14.899099 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.899062    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zp2df"
Jul 21 17:26:14.911437 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.911400    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5872t"
Jul 21 17:26:14.921406 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.921376    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:14.929257 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.929235    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wng2j"
Jul 21 17:26:14.937965 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.937936    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zqxmf"
Jul 21 17:26:14.945714 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.945692    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gxckp"
Jul 21 17:26:14.953564 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.953542    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fwcw8"
Jul 21 17:26:14.961390 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.961364    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mgp2n"
Jul 21 17:26:14.968339 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:14.968318    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rg2jl"
Jul 21 17:26:15.205517 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:15.205470    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa6ca187-d1f3-4d33-bd20-f70784dbbe02-original-pull-secret\") pod \"global-pull-secret-syncer-g76xh\" (UID: \"fa6ca187-d1f3-4d33-bd20-f70784dbbe02\") " pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:26:15.205709 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:15.205661    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 21 17:26:15.205764 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:15.205745    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa6ca187-d1f3-4d33-bd20-f70784dbbe02-original-pull-secret podName:fa6ca187-d1f3-4d33-bd20-f70784dbbe02 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:16.205729094 +0000 UTC m=+3.967868678 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa6ca187-d1f3-4d33-bd20-f70784dbbe02-original-pull-secret") pod "global-pull-secret-syncer-g76xh" (UID: "fa6ca187-d1f3-4d33-bd20-f70784dbbe02") : object "kube-system"/"original-pull-secret" not registered
Jul 21 17:26:15.306588 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:15.306551    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82279305-9475-429f-a495-143fc616fb34-metrics-certs\") pod \"network-metrics-daemon-sxmzk\" (UID: \"82279305-9475-429f-a495-143fc616fb34\") " pod="openshift-multus/network-metrics-daemon-sxmzk"
Jul 21 17:26:15.306772 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:15.306715    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:26:15.306822 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:15.306783    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82279305-9475-429f-a495-143fc616fb34-metrics-certs podName:82279305-9475-429f-a495-143fc616fb34 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:16.306768559 +0000 UTC m=+4.068908143 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82279305-9475-429f-a495-143fc616fb34-metrics-certs") pod "network-metrics-daemon-sxmzk" (UID: "82279305-9475-429f-a495-143fc616fb34") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:26:15.407599 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:15.407554    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n2xt\" (UniqueName: \"kubernetes.io/projected/b92707b0-e59d-4123-82e2-658e4361cb6e-kube-api-access-9n2xt\") pod \"network-check-target-d8gnt\" (UID: \"b92707b0-e59d-4123-82e2-658e4361cb6e\") " pod="openshift-network-diagnostics/network-check-target-d8gnt"
Jul 21 17:26:15.407796 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:15.407722    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 21 17:26:15.407796 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:15.407751    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 21 17:26:15.407796 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:15.407768    2563 projected.go:196] Error preparing data for projected volume kube-api-access-9n2xt for pod openshift-network-diagnostics/network-check-target-d8gnt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:26:15.407923 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:15.407835    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b92707b0-e59d-4123-82e2-658e4361cb6e-kube-api-access-9n2xt podName:b92707b0-e59d-4123-82e2-658e4361cb6e nodeName:}" failed. No retries permitted until 2026-07-21 17:26:16.407818732 +0000 UTC m=+4.169958317 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9n2xt" (UniqueName: "kubernetes.io/projected/b92707b0-e59d-4123-82e2-658e4361cb6e-kube-api-access-9n2xt") pod "network-check-target-d8gnt" (UID: "b92707b0-e59d-4123-82e2-658e4361cb6e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:26:15.628776 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:15.628723    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 17:21:14 +0000 UTC" deadline="2028-01-18 08:32:39.474100357 +0000 UTC"
Jul 21 17:26:15.628776 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:15.628772    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13095h6m23.845330834s"
Jul 21 17:26:15.653184 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:15.653156    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5401269_8543_4227_a538_95e2f3051c4b.slice/crio-19665810d5860462b46c04d49c1df12458d26206c2766179be769fdba6bcd66f WatchSource:0}: Error finding container 19665810d5860462b46c04d49c1df12458d26206c2766179be769fdba6bcd66f: Status 404 returned error can't find the container with id 19665810d5860462b46c04d49c1df12458d26206c2766179be769fdba6bcd66f
Jul 21 17:26:15.655512 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:15.654979    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda61ba40a_d3c2_45a9_8525_5e40c9de49b6.slice/crio-bb124adb7b1989239d7f1a1300eca786611dfb44b85ae1374854bf975bbd9bb9 WatchSource:0}: Error finding container bb124adb7b1989239d7f1a1300eca786611dfb44b85ae1374854bf975bbd9bb9: Status 404 returned error can't find the container with id bb124adb7b1989239d7f1a1300eca786611dfb44b85ae1374854bf975bbd9bb9
Jul 21 17:26:15.656672 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:15.656600    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2998a9d6_86dc_4ab8_961d_7f873ce559c6.slice/crio-27b7d27d6af20a83df2c791c1dc27bedec3c24c326f6e911561e9e7d76671bff WatchSource:0}: Error finding container 27b7d27d6af20a83df2c791c1dc27bedec3c24c326f6e911561e9e7d76671bff: Status 404 returned error can't find the container with id 27b7d27d6af20a83df2c791c1dc27bedec3c24c326f6e911561e9e7d76671bff
Jul 21 17:26:15.657733 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:15.657680    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e07071d_5eed_4b15_844b_8dd2c6bf1eaa.slice/crio-0fc1fbf6cf745dc2de581f1c40e34eb0e70a6f8601b7534f66622f6eba68b0c1 WatchSource:0}: Error finding container 0fc1fbf6cf745dc2de581f1c40e34eb0e70a6f8601b7534f66622f6eba68b0c1: Status 404 returned error can't find the container with id 0fc1fbf6cf745dc2de581f1c40e34eb0e70a6f8601b7534f66622f6eba68b0c1
Jul 21 17:26:15.659262 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:15.659178    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b38cc54_ebe3_48aa_b1ca_671f5f427f93.slice/crio-e5e5ff5451c9d4c6efc0a1b0aacc2599dedeea29b6c11bbc632ee3d378070c57 WatchSource:0}: Error finding container e5e5ff5451c9d4c6efc0a1b0aacc2599dedeea29b6c11bbc632ee3d378070c57: Status 404 returned error can't find the container with id e5e5ff5451c9d4c6efc0a1b0aacc2599dedeea29b6c11bbc632ee3d378070c57
Jul 21 17:26:15.661435 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:15.660789    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda83f9b9f_8087_49ba_a353_ab97de0d1935.slice/crio-b4c07064c6021732b120be280e383b8e3bb6cedbd9147918df0a944e0c1b991c WatchSource:0}: Error finding container b4c07064c6021732b120be280e383b8e3bb6cedbd9147918df0a944e0c1b991c: Status 404 returned error can't find the container with id b4c07064c6021732b120be280e383b8e3bb6cedbd9147918df0a944e0c1b991c
Jul 21 17:26:15.661719 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:15.661693    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35f84d29_3ea1_4c65_b000_d910c8012948.slice/crio-fc94663433c41254cb985d24e2333143ba3c5fb214835f256f865f7a028d7a4f WatchSource:0}: Error finding container fc94663433c41254cb985d24e2333143ba3c5fb214835f256f865f7a028d7a4f: Status 404 returned error can't find the container with id fc94663433c41254cb985d24e2333143ba3c5fb214835f256f865f7a028d7a4f
Jul 21 17:26:15.664399 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:15.663915    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod189f8b1f_23e2_4447_90bd_ffa91affc6c4.slice/crio-ca1dcc5f275fcb3ecab4804698be2d4190bd38f3e50f4ae74cb0391c48557dcd WatchSource:0}: Error finding container ca1dcc5f275fcb3ecab4804698be2d4190bd38f3e50f4ae74cb0391c48557dcd: Status 404 returned error can't find the container with id ca1dcc5f275fcb3ecab4804698be2d4190bd38f3e50f4ae74cb0391c48557dcd
Jul 21 17:26:15.665025 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:26:15.665002    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f7ddf4c_27b9_4b30_8d4a_483cfa1fb74a.slice/crio-13343737deeebcb31a82f3d2b9c0557183799660d8d1bc9069ece150a037a21c WatchSource:0}: Error finding container 13343737deeebcb31a82f3d2b9c0557183799660d8d1bc9069ece150a037a21c: Status 404 returned error can't find the container with id 13343737deeebcb31a82f3d2b9c0557183799660d8d1bc9069ece150a037a21c
Jul 21 17:26:15.771950 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:15.771726    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fwcw8" event={"ID":"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa","Type":"ContainerStarted","Data":"0fc1fbf6cf745dc2de581f1c40e34eb0e70a6f8601b7534f66622f6eba68b0c1"}
Jul 21 17:26:15.773083 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:15.773049    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zqxmf" event={"ID":"c5401269-8543-4227-a538-95e2f3051c4b","Type":"ContainerStarted","Data":"19665810d5860462b46c04d49c1df12458d26206c2766179be769fdba6bcd66f"}
Jul 21 17:26:15.775302 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:15.775276    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-249.ec2.internal" event={"ID":"fb93a1b33c0d6ced6dfb0f4505a58dc6","Type":"ContainerStarted","Data":"42b30e13b667e32315bfac9a6e79e618da118cb9fc9d6981df125d92d4b650a5"}
Jul 21 17:26:15.776407 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:15.776383    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gxckp" event={"ID":"189f8b1f-23e2-4447-90bd-ffa91affc6c4","Type":"ContainerStarted","Data":"ca1dcc5f275fcb3ecab4804698be2d4190bd38f3e50f4ae74cb0391c48557dcd"}
Jul 21 17:26:15.777576 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:15.777531    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rg2jl" event={"ID":"1b38cc54-ebe3-48aa-b1ca-671f5f427f93","Type":"ContainerStarted","Data":"e5e5ff5451c9d4c6efc0a1b0aacc2599dedeea29b6c11bbc632ee3d378070c57"}
Jul 21 17:26:15.778825 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:15.778792    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zp2df" event={"ID":"2998a9d6-86dc-4ab8-961d-7f873ce559c6","Type":"ContainerStarted","Data":"27b7d27d6af20a83df2c791c1dc27bedec3c24c326f6e911561e9e7d76671bff"}
Jul 21 17:26:15.780104 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:15.780079    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6" event={"ID":"a61ba40a-d3c2-45a9-8525-5e40c9de49b6","Type":"ContainerStarted","Data":"bb124adb7b1989239d7f1a1300eca786611dfb44b85ae1374854bf975bbd9bb9"}
Jul 21 17:26:15.781367 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:15.781345    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mgp2n" event={"ID":"5f7ddf4c-27b9-4b30-8d4a-483cfa1fb74a","Type":"ContainerStarted","Data":"13343737deeebcb31a82f3d2b9c0557183799660d8d1bc9069ece150a037a21c"}
Jul 21 17:26:15.782526 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:15.782504    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5872t" event={"ID":"35f84d29-3ea1-4c65-b000-d910c8012948","Type":"ContainerStarted","Data":"fc94663433c41254cb985d24e2333143ba3c5fb214835f256f865f7a028d7a4f"}
Jul 21 17:26:15.783716 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:15.783689    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wng2j" event={"ID":"a83f9b9f-8087-49ba-a353-ab97de0d1935","Type":"ContainerStarted","Data":"b4c07064c6021732b120be280e383b8e3bb6cedbd9147918df0a944e0c1b991c"}
Jul 21 17:26:15.794616 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:15.794558    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-249.ec2.internal" podStartSLOduration=1.79454272 podStartE2EDuration="1.79454272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:26:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:15.793995525 +0000 UTC m=+3.556135162" watchObservedRunningTime="2026-07-21 17:26:15.79454272 +0000 UTC m=+3.556682328"
Jul 21 17:26:16.214910 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:16.214814    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa6ca187-d1f3-4d33-bd20-f70784dbbe02-original-pull-secret\") pod \"global-pull-secret-syncer-g76xh\" (UID: \"fa6ca187-d1f3-4d33-bd20-f70784dbbe02\") " pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:26:16.215078 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:16.214999    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 21 17:26:16.215078 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:16.215071    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa6ca187-d1f3-4d33-bd20-f70784dbbe02-original-pull-secret podName:fa6ca187-d1f3-4d33-bd20-f70784dbbe02 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:18.215047895 +0000 UTC m=+5.977187492 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa6ca187-d1f3-4d33-bd20-f70784dbbe02-original-pull-secret") pod "global-pull-secret-syncer-g76xh" (UID: "fa6ca187-d1f3-4d33-bd20-f70784dbbe02") : object "kube-system"/"original-pull-secret" not registered
Jul 21 17:26:16.315259 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:16.315216    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82279305-9475-429f-a495-143fc616fb34-metrics-certs\") pod \"network-metrics-daemon-sxmzk\" (UID: \"82279305-9475-429f-a495-143fc616fb34\") " pod="openshift-multus/network-metrics-daemon-sxmzk"
Jul 21 17:26:16.315465 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:16.315441    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:26:16.315559 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:16.315512    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82279305-9475-429f-a495-143fc616fb34-metrics-certs podName:82279305-9475-429f-a495-143fc616fb34 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:18.315492826 +0000 UTC m=+6.077632415 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82279305-9475-429f-a495-143fc616fb34-metrics-certs") pod "network-metrics-daemon-sxmzk" (UID: "82279305-9475-429f-a495-143fc616fb34") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:26:16.416588 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:16.416546    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n2xt\" (UniqueName: \"kubernetes.io/projected/b92707b0-e59d-4123-82e2-658e4361cb6e-kube-api-access-9n2xt\") pod \"network-check-target-d8gnt\" (UID: \"b92707b0-e59d-4123-82e2-658e4361cb6e\") " pod="openshift-network-diagnostics/network-check-target-d8gnt"
Jul 21 17:26:16.416808 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:16.416769    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 21 17:26:16.416808 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:16.416789    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 21 17:26:16.416808 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:16.416803    2563 projected.go:196] Error preparing data for projected volume kube-api-access-9n2xt for pod openshift-network-diagnostics/network-check-target-d8gnt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:26:16.417053 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:16.416865    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b92707b0-e59d-4123-82e2-658e4361cb6e-kube-api-access-9n2xt podName:b92707b0-e59d-4123-82e2-658e4361cb6e nodeName:}" failed. No retries permitted until 2026-07-21 17:26:18.416844804 +0000 UTC m=+6.178984393 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9n2xt" (UniqueName: "kubernetes.io/projected/b92707b0-e59d-4123-82e2-658e4361cb6e-kube-api-access-9n2xt") pod "network-check-target-d8gnt" (UID: "b92707b0-e59d-4123-82e2-658e4361cb6e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:26:16.762159 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:16.762101    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:26:16.762754 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:16.762253    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g76xh" podUID="fa6ca187-d1f3-4d33-bd20-f70784dbbe02"
Jul 21 17:26:16.762754 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:16.762269    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmzk"
Jul 21 17:26:16.762754 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:16.762379    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmzk" podUID="82279305-9475-429f-a495-143fc616fb34"
Jul 21 17:26:16.762754 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:16.762408    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8gnt"
Jul 21 17:26:16.762754 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:16.762480    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8gnt" podUID="b92707b0-e59d-4123-82e2-658e4361cb6e"
Jul 21 17:26:16.795670 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:16.794396    2563 generic.go:356] "Generic (PLEG): container finished" podID="db56e96cb19cb0d0fae9df6533d0a2ea" containerID="711dd82babf96ca55f8a5d99a7288bc43e2efe60f47243c09fe3accaa81a3435" exitCode=0
Jul 21 17:26:16.795670 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:16.795434    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-249.ec2.internal" event={"ID":"db56e96cb19cb0d0fae9df6533d0a2ea","Type":"ContainerDied","Data":"711dd82babf96ca55f8a5d99a7288bc43e2efe60f47243c09fe3accaa81a3435"}
Jul 21 17:26:17.812298 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:17.812258    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-249.ec2.internal" event={"ID":"db56e96cb19cb0d0fae9df6533d0a2ea","Type":"ContainerStarted","Data":"119fefdeeeb7fb7e9d817ed43b9a4ae40a3d1a8715d78ff0791c9352ebcb14b0"}
Jul 21 17:26:18.234463 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:18.234040    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa6ca187-d1f3-4d33-bd20-f70784dbbe02-original-pull-secret\") pod \"global-pull-secret-syncer-g76xh\" (UID: \"fa6ca187-d1f3-4d33-bd20-f70784dbbe02\") " pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:26:18.234463 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:18.234258    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 21 17:26:18.234463 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:18.234324    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa6ca187-d1f3-4d33-bd20-f70784dbbe02-original-pull-secret podName:fa6ca187-d1f3-4d33-bd20-f70784dbbe02 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:22.234307606 +0000 UTC m=+9.996447189 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa6ca187-d1f3-4d33-bd20-f70784dbbe02-original-pull-secret") pod "global-pull-secret-syncer-g76xh" (UID: "fa6ca187-d1f3-4d33-bd20-f70784dbbe02") : object "kube-system"/"original-pull-secret" not registered
Jul 21 17:26:18.335837 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:18.335208    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82279305-9475-429f-a495-143fc616fb34-metrics-certs\") pod \"network-metrics-daemon-sxmzk\" (UID: \"82279305-9475-429f-a495-143fc616fb34\") " pod="openshift-multus/network-metrics-daemon-sxmzk"
Jul 21 17:26:18.335837 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:18.335406    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:26:18.335837 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:18.335474    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82279305-9475-429f-a495-143fc616fb34-metrics-certs podName:82279305-9475-429f-a495-143fc616fb34 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:22.335452895 +0000 UTC m=+10.097592481 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82279305-9475-429f-a495-143fc616fb34-metrics-certs") pod "network-metrics-daemon-sxmzk" (UID: "82279305-9475-429f-a495-143fc616fb34") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:26:18.437323 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:18.436618    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n2xt\" (UniqueName: \"kubernetes.io/projected/b92707b0-e59d-4123-82e2-658e4361cb6e-kube-api-access-9n2xt\") pod \"network-check-target-d8gnt\" (UID: \"b92707b0-e59d-4123-82e2-658e4361cb6e\") " pod="openshift-network-diagnostics/network-check-target-d8gnt"
Jul 21 17:26:18.437323 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:18.436790    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 21 17:26:18.437323 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:18.436812    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 21 17:26:18.437323 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:18.436825    2563 projected.go:196] Error preparing data for projected volume kube-api-access-9n2xt for pod openshift-network-diagnostics/network-check-target-d8gnt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:26:18.437323 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:18.436889    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b92707b0-e59d-4123-82e2-658e4361cb6e-kube-api-access-9n2xt podName:b92707b0-e59d-4123-82e2-658e4361cb6e nodeName:}" failed. No retries permitted until 2026-07-21 17:26:22.436870329 +0000 UTC m=+10.199009916 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9n2xt" (UniqueName: "kubernetes.io/projected/b92707b0-e59d-4123-82e2-658e4361cb6e-kube-api-access-9n2xt") pod "network-check-target-d8gnt" (UID: "b92707b0-e59d-4123-82e2-658e4361cb6e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:26:18.760377 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:18.760341    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:26:18.760581 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:18.760348    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmzk"
Jul 21 17:26:18.760581 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:18.760486    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g76xh" podUID="fa6ca187-d1f3-4d33-bd20-f70784dbbe02"
Jul 21 17:26:18.760581 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:18.760564    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmzk" podUID="82279305-9475-429f-a495-143fc616fb34"
Jul 21 17:26:18.760766 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:18.760633    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8gnt"
Jul 21 17:26:18.760766 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:18.760710    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8gnt" podUID="b92707b0-e59d-4123-82e2-658e4361cb6e"
Jul 21 17:26:20.759326 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:20.759298    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:26:20.759737 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:20.759436    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g76xh" podUID="fa6ca187-d1f3-4d33-bd20-f70784dbbe02"
Jul 21 17:26:20.759859 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:20.759833    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmzk"
Jul 21 17:26:20.759969 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:20.759946    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmzk" podUID="82279305-9475-429f-a495-143fc616fb34"
Jul 21 17:26:20.760165 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:20.760124    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8gnt"
Jul 21 17:26:20.760283 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:20.760260    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8gnt" podUID="b92707b0-e59d-4123-82e2-658e4361cb6e"
Jul 21 17:26:22.271895 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:22.271861    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa6ca187-d1f3-4d33-bd20-f70784dbbe02-original-pull-secret\") pod \"global-pull-secret-syncer-g76xh\" (UID: \"fa6ca187-d1f3-4d33-bd20-f70784dbbe02\") " pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:26:22.272400 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:22.272042    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 21 17:26:22.272400 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:22.272106    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa6ca187-d1f3-4d33-bd20-f70784dbbe02-original-pull-secret podName:fa6ca187-d1f3-4d33-bd20-f70784dbbe02 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:30.272086793 +0000 UTC m=+18.034226377 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa6ca187-d1f3-4d33-bd20-f70784dbbe02-original-pull-secret") pod "global-pull-secret-syncer-g76xh" (UID: "fa6ca187-d1f3-4d33-bd20-f70784dbbe02") : object "kube-system"/"original-pull-secret" not registered
Jul 21 17:26:22.373159 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:22.372904    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82279305-9475-429f-a495-143fc616fb34-metrics-certs\") pod \"network-metrics-daemon-sxmzk\" (UID: \"82279305-9475-429f-a495-143fc616fb34\") " pod="openshift-multus/network-metrics-daemon-sxmzk"
Jul 21 17:26:22.373159 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:22.373048    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:26:22.373159 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:22.373122    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82279305-9475-429f-a495-143fc616fb34-metrics-certs podName:82279305-9475-429f-a495-143fc616fb34 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:30.373101166 +0000 UTC m=+18.135240748 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82279305-9475-429f-a495-143fc616fb34-metrics-certs") pod "network-metrics-daemon-sxmzk" (UID: "82279305-9475-429f-a495-143fc616fb34") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:26:22.473715 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:22.473676    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n2xt\" (UniqueName: \"kubernetes.io/projected/b92707b0-e59d-4123-82e2-658e4361cb6e-kube-api-access-9n2xt\") pod \"network-check-target-d8gnt\" (UID: \"b92707b0-e59d-4123-82e2-658e4361cb6e\") " pod="openshift-network-diagnostics/network-check-target-d8gnt"
Jul 21 17:26:22.473903 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:22.473844    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 21 17:26:22.473903 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:22.473864    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 21 17:26:22.473903 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:22.473877    2563 projected.go:196] Error preparing data for projected volume kube-api-access-9n2xt for pod openshift-network-diagnostics/network-check-target-d8gnt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:26:22.474062 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:22.473936    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b92707b0-e59d-4123-82e2-658e4361cb6e-kube-api-access-9n2xt podName:b92707b0-e59d-4123-82e2-658e4361cb6e nodeName:}" failed. No retries permitted until 2026-07-21 17:26:30.473916812 +0000 UTC m=+18.236056398 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9n2xt" (UniqueName: "kubernetes.io/projected/b92707b0-e59d-4123-82e2-658e4361cb6e-kube-api-access-9n2xt") pod "network-check-target-d8gnt" (UID: "b92707b0-e59d-4123-82e2-658e4361cb6e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:26:22.759971 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:22.759939    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:26:22.759971 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:22.759950    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmzk"
Jul 21 17:26:22.761109 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:22.761072    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g76xh" podUID="fa6ca187-d1f3-4d33-bd20-f70784dbbe02"
Jul 21 17:26:22.761467 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:22.761427    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8gnt"
Jul 21 17:26:22.761582 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:22.761541    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmzk" podUID="82279305-9475-429f-a495-143fc616fb34"
Jul 21 17:26:22.761647 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:22.761617    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8gnt" podUID="b92707b0-e59d-4123-82e2-658e4361cb6e"
Jul 21 17:26:24.759728 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:24.759249    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmzk"
Jul 21 17:26:24.759728 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:24.759249    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:26:24.759728 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:24.759392    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmzk" podUID="82279305-9475-429f-a495-143fc616fb34"
Jul 21 17:26:24.759728 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:24.759507    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g76xh" podUID="fa6ca187-d1f3-4d33-bd20-f70784dbbe02"
Jul 21 17:26:24.759728 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:24.759569    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8gnt"
Jul 21 17:26:24.759728 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:24.759634    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8gnt" podUID="b92707b0-e59d-4123-82e2-658e4361cb6e"
Jul 21 17:26:26.761825 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:26.761791    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8gnt"
Jul 21 17:26:26.762309 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:26.761893    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:26:26.762309 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:26.761917    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmzk"
Jul 21 17:26:26.762309 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:26.761909    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8gnt" podUID="b92707b0-e59d-4123-82e2-658e4361cb6e"
Jul 21 17:26:26.762309 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:26.761979    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g76xh" podUID="fa6ca187-d1f3-4d33-bd20-f70784dbbe02"
Jul 21 17:26:26.762309 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:26.762054    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmzk" podUID="82279305-9475-429f-a495-143fc616fb34"
Jul 21 17:26:28.759353 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:28.759316    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:26:28.759765 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:28.759316    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8gnt"
Jul 21 17:26:28.759765 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:28.759463    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g76xh" podUID="fa6ca187-d1f3-4d33-bd20-f70784dbbe02"
Jul 21 17:26:28.759765 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:28.759534    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8gnt" podUID="b92707b0-e59d-4123-82e2-658e4361cb6e"
Jul 21 17:26:28.759765 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:28.759324    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmzk"
Jul 21 17:26:28.759765 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:28.759604    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmzk" podUID="82279305-9475-429f-a495-143fc616fb34"
Jul 21 17:26:30.333204 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:30.333165    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa6ca187-d1f3-4d33-bd20-f70784dbbe02-original-pull-secret\") pod \"global-pull-secret-syncer-g76xh\" (UID: \"fa6ca187-d1f3-4d33-bd20-f70784dbbe02\") " pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:26:30.333744 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:30.333360    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 21 17:26:30.333744 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:30.333447    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa6ca187-d1f3-4d33-bd20-f70784dbbe02-original-pull-secret podName:fa6ca187-d1f3-4d33-bd20-f70784dbbe02 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:46.333425235 +0000 UTC m=+34.095564821 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa6ca187-d1f3-4d33-bd20-f70784dbbe02-original-pull-secret") pod "global-pull-secret-syncer-g76xh" (UID: "fa6ca187-d1f3-4d33-bd20-f70784dbbe02") : object "kube-system"/"original-pull-secret" not registered
Jul 21 17:26:30.433512 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:30.433465    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82279305-9475-429f-a495-143fc616fb34-metrics-certs\") pod \"network-metrics-daemon-sxmzk\" (UID: \"82279305-9475-429f-a495-143fc616fb34\") " pod="openshift-multus/network-metrics-daemon-sxmzk"
Jul 21 17:26:30.433686 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:30.433642    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:26:30.433733 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:30.433720    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82279305-9475-429f-a495-143fc616fb34-metrics-certs podName:82279305-9475-429f-a495-143fc616fb34 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:46.433698213 +0000 UTC m=+34.195837795 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82279305-9475-429f-a495-143fc616fb34-metrics-certs") pod "network-metrics-daemon-sxmzk" (UID: "82279305-9475-429f-a495-143fc616fb34") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:26:30.534708 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:30.534667    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n2xt\" (UniqueName: \"kubernetes.io/projected/b92707b0-e59d-4123-82e2-658e4361cb6e-kube-api-access-9n2xt\") pod \"network-check-target-d8gnt\" (UID: \"b92707b0-e59d-4123-82e2-658e4361cb6e\") " pod="openshift-network-diagnostics/network-check-target-d8gnt"
Jul 21 17:26:30.534907 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:30.534849    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 21 17:26:30.534907 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:30.534868    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 21 17:26:30.534907 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:30.534881    2563 projected.go:196] Error preparing data for projected volume kube-api-access-9n2xt for pod openshift-network-diagnostics/network-check-target-d8gnt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:26:30.535080 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:30.534934    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b92707b0-e59d-4123-82e2-658e4361cb6e-kube-api-access-9n2xt podName:b92707b0-e59d-4123-82e2-658e4361cb6e nodeName:}" failed. No retries permitted until 2026-07-21 17:26:46.534920321 +0000 UTC m=+34.297059903 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9n2xt" (UniqueName: "kubernetes.io/projected/b92707b0-e59d-4123-82e2-658e4361cb6e-kube-api-access-9n2xt") pod "network-check-target-d8gnt" (UID: "b92707b0-e59d-4123-82e2-658e4361cb6e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:26:30.759080 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:30.759054    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmzk"
Jul 21 17:26:30.759287 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:30.759062    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:26:30.759287 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:30.759207    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmzk" podUID="82279305-9475-429f-a495-143fc616fb34"
Jul 21 17:26:30.759287 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:30.759219    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8gnt"
Jul 21 17:26:30.759476 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:30.759304    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g76xh" podUID="fa6ca187-d1f3-4d33-bd20-f70784dbbe02"
Jul 21 17:26:30.759476 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:30.759396    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8gnt" podUID="b92707b0-e59d-4123-82e2-658e4361cb6e"
Jul 21 17:26:32.760938 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:32.760471    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8gnt"
Jul 21 17:26:32.760938 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:32.760579    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmzk"
Jul 21 17:26:32.760938 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:32.760586    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8gnt" podUID="b92707b0-e59d-4123-82e2-658e4361cb6e"
Jul 21 17:26:32.760938 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:32.760713    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmzk" podUID="82279305-9475-429f-a495-143fc616fb34"
Jul 21 17:26:32.760938 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:32.760765    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:26:32.760938 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:32.760847    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g76xh" podUID="fa6ca187-d1f3-4d33-bd20-f70784dbbe02"
Jul 21 17:26:34.758913 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:34.758871    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:26:34.758913 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:34.758909    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8gnt"
Jul 21 17:26:34.759562 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:34.758876    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmzk"
Jul 21 17:26:34.759562 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:34.759017    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g76xh" podUID="fa6ca187-d1f3-4d33-bd20-f70784dbbe02"
Jul 21 17:26:34.759562 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:34.759188    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmzk" podUID="82279305-9475-429f-a495-143fc616fb34"
Jul 21 17:26:34.759562 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:34.759276    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8gnt" podUID="b92707b0-e59d-4123-82e2-658e4361cb6e"
Jul 21 17:26:35.852343 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:35.851908    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gxckp" event={"ID":"189f8b1f-23e2-4447-90bd-ffa91affc6c4","Type":"ContainerStarted","Data":"e37830a8c8d9b6ca5158f77f23b8bd4615cc8737fc6a75b7c5056016147c5761"}
Jul 21 17:26:35.856091 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:35.856061    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rg2jl" event={"ID":"1b38cc54-ebe3-48aa-b1ca-671f5f427f93","Type":"ContainerStarted","Data":"3f1290da36bc9885529cf906283ebdffb2cbd7c7a0d68a9e2c7ab3832e3e9230"}
Jul 21 17:26:35.858114 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:35.858092    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zp2df" event={"ID":"2998a9d6-86dc-4ab8-961d-7f873ce559c6","Type":"ContainerStarted","Data":"7a3dcabf26f234b4f87a0a8c969cfbe91a9c61ff133d6de0b52685a42b4b25dc"}
Jul 21 17:26:35.860956 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:35.860934    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6" event={"ID":"a61ba40a-d3c2-45a9-8525-5e40c9de49b6","Type":"ContainerStarted","Data":"e206af8967383cf5ded4f52263bd620293bc8638b87a0ae2ee70d18f43758bd0"}
Jul 21 17:26:35.861086 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:35.861004    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6" event={"ID":"a61ba40a-d3c2-45a9-8525-5e40c9de49b6","Type":"ContainerStarted","Data":"7075a1195106ff7f866638feb662a89d9c999e81552c7e65d82fa09bb9f013db"}
Jul 21 17:26:35.861086 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:35.861052    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6" event={"ID":"a61ba40a-d3c2-45a9-8525-5e40c9de49b6","Type":"ContainerStarted","Data":"3e726e1ea701aedcf909d2a1ab6abfd5ee00d62928b6e70600c49cd4d3a32ab8"}
Jul 21 17:26:35.861224 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:35.861095    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6" event={"ID":"a61ba40a-d3c2-45a9-8525-5e40c9de49b6","Type":"ContainerStarted","Data":"4d65b9995c43ceecbee0a699ae6bb8938c1ba30ff9a3ee50b6896ed046592784"}
Jul 21 17:26:35.861224 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:35.861163    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6" event={"ID":"a61ba40a-d3c2-45a9-8525-5e40c9de49b6","Type":"ContainerStarted","Data":"def4ca1624b910933a7f29bc1c888c617375ca703aff871cb97b72ba03fa8f3f"}
Jul 21 17:26:35.861224 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:35.861210    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6" event={"ID":"a61ba40a-d3c2-45a9-8525-5e40c9de49b6","Type":"ContainerStarted","Data":"a9fb9230419a529820fc33ada7b4f7380f45e5a0bb554df721796703828d3265"}
Jul 21 17:26:35.862214 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:35.862192    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mgp2n" event={"ID":"5f7ddf4c-27b9-4b30-8d4a-483cfa1fb74a","Type":"ContainerStarted","Data":"1277cd4fbf12731539e38d7c3b93c5e4f4573176833aac57bdc7239ba7eec20d"}
Jul 21 17:26:35.863393 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:35.863372    2563 generic.go:356] "Generic (PLEG): container finished" podID="35f84d29-3ea1-4c65-b000-d910c8012948" containerID="29671ee45a3ce23ab3c0b5e940d4e02f955b849ca3231efdaf7124df92f57553" exitCode=0
Jul 21 17:26:35.863491 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:35.863428    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5872t" event={"ID":"35f84d29-3ea1-4c65-b000-d910c8012948","Type":"ContainerDied","Data":"29671ee45a3ce23ab3c0b5e940d4e02f955b849ca3231efdaf7124df92f57553"}
Jul 21 17:26:35.864871 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:35.864851    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wng2j" event={"ID":"a83f9b9f-8087-49ba-a353-ab97de0d1935","Type":"ContainerStarted","Data":"24a9fb71e403894d53f51153df41d275bce1e47968475d8d3c4417f2343ea5f0"}
Jul 21 17:26:35.866149 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:35.866107    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fwcw8" event={"ID":"8e07071d-5eed-4b15-844b-8dd2c6bf1eaa","Type":"ContainerStarted","Data":"a4d6c3a1f697a778ed5b3e74e83b91b593559d704ed665a7260dcd02ddb7d31a"}
Jul 21 17:26:35.868642 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:35.868602    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-249.ec2.internal" podStartSLOduration=21.86856997 podStartE2EDuration="21.86856997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:26:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:17.840419561 +0000 UTC m=+5.602559172" watchObservedRunningTime="2026-07-21 17:26:35.86856997 +0000 UTC m=+23.630709575"
Jul 21 17:26:35.868897 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:35.868873    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gxckp" podStartSLOduration=3.5056198739999997 podStartE2EDuration="22.86886353s" totalImagesPullingTime="19.363243656s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:26:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:35.868643402 +0000 UTC m=+23.630783009" watchObservedRunningTime="2026-07-21 17:26:35.86886353 +0000 UTC m=+23.631003135"
Jul 21 17:26:35.882354 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:35.882317    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mgp2n" podStartSLOduration=3.532139035 podStartE2EDuration="22.882306458s" totalImagesPullingTime="19.350167423s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:26:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:35.881947498 +0000 UTC m=+23.644087102" watchObservedRunningTime="2026-07-21 17:26:35.882306458 +0000 UTC m=+23.644446052"
Jul 21 17:26:35.900497 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:35.900459    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-zp2df" podStartSLOduration=3.4792564710000002 podStartE2EDuration="22.900446618s" totalImagesPullingTime="19.421190147s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:26:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:35.900235473 +0000 UTC m=+23.662375078" watchObservedRunningTime="2026-07-21 17:26:35.900446618 +0000 UTC m=+23.662586221"
Jul 21 17:26:35.940428 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:35.940369    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fwcw8" podStartSLOduration=3.5805498719999997 podStartE2EDuration="22.940349317s" totalImagesPullingTime="19.359799445s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:26:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:35.940149412 +0000 UTC m=+23.702289007" watchObservedRunningTime="2026-07-21 17:26:35.940349317 +0000 UTC m=+23.702488921"
Jul 21 17:26:35.961462 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:35.961419    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rg2jl" podStartSLOduration=3.606101261 podStartE2EDuration="22.961403352s" totalImagesPullingTime="19.355302091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:26:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:35.960936193 +0000 UTC m=+23.723075798" watchObservedRunningTime="2026-07-21 17:26:35.961403352 +0000 UTC m=+23.723542956"
Jul 21 17:26:36.587594 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:36.587565    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 21 17:26:36.658933 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:36.658776    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-21T17:26:36.587588481Z","UUID":"340f9823-6e8f-40ee-9be7-0ba91e203031","Handler":null,"Name":"","Endpoint":""}
Jul 21 17:26:36.660410 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:36.660392    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 21 17:26:36.660529 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:36.660462    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 21 17:26:36.759478 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:36.759437    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:26:36.759478 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:36.759453    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmzk"
Jul 21 17:26:36.759715 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:36.759572    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g76xh" podUID="fa6ca187-d1f3-4d33-bd20-f70784dbbe02"
Jul 21 17:26:36.759715 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:36.759587    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8gnt"
Jul 21 17:26:36.759933 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:36.759715    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmzk" podUID="82279305-9475-429f-a495-143fc616fb34"
Jul 21 17:26:36.759933 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:36.759830    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8gnt" podUID="b92707b0-e59d-4123-82e2-658e4361cb6e"
Jul 21 17:26:36.870278 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:36.870235    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wng2j" event={"ID":"a83f9b9f-8087-49ba-a353-ab97de0d1935","Type":"ContainerStarted","Data":"e1a9f04bdfc667550b530cebe473f166c1687528a82824ae990aa408708a7e2b"}
Jul 21 17:26:36.871924 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:36.871877    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zqxmf" event={"ID":"c5401269-8543-4227-a538-95e2f3051c4b","Type":"ContainerStarted","Data":"5144a9ad7418f0d3a8bdab066b4712e6db3027c691b896637d0292b768c7c378"}
Jul 21 17:26:36.888709 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:36.888653    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zqxmf" podStartSLOduration=4.52649442 podStartE2EDuration="23.888633146s" totalImagesPullingTime="19.362138726s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:26:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:36.888337497 +0000 UTC m=+24.650477102" watchObservedRunningTime="2026-07-21 17:26:36.888633146 +0000 UTC m=+24.650772751"
Jul 21 17:26:37.879259 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:37.879033    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6" event={"ID":"a61ba40a-d3c2-45a9-8525-5e40c9de49b6","Type":"ContainerStarted","Data":"c6d70a3774ef74a1e41bca1a32fccfab6efc24a42398926f1cc2d074057b5b4f"}
Jul 21 17:26:37.882272 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:37.882194    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wng2j" event={"ID":"a83f9b9f-8087-49ba-a353-ab97de0d1935","Type":"ContainerStarted","Data":"a45bc6136a2719a01a7ce184c9dc266b1992c42e8e140bb3d7bfcad0de6dbaf1"}
Jul 21 17:26:38.759705 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:38.759671    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:26:38.759894 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:38.759671    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8gnt"
Jul 21 17:26:38.759894 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:38.759811    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g76xh" podUID="fa6ca187-d1f3-4d33-bd20-f70784dbbe02"
Jul 21 17:26:38.759894 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:38.759855    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmzk"
Jul 21 17:26:38.760065 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:38.760010    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8gnt" podUID="b92707b0-e59d-4123-82e2-658e4361cb6e"
Jul 21 17:26:38.760154 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:38.760118    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmzk" podUID="82279305-9475-429f-a495-143fc616fb34"
Jul 21 17:26:39.598643 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:39.598596    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rg2jl"
Jul 21 17:26:39.599284 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:39.599262    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rg2jl"
Jul 21 17:26:39.622217 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:39.622173    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wng2j" podStartSLOduration=5.064757528 podStartE2EDuration="26.622154325s" totalImagesPullingTime="21.557396797s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:26:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:37.90949206 +0000 UTC m=+25.671631666" watchObservedRunningTime="2026-07-21 17:26:39.622154325 +0000 UTC m=+27.384293923"
Jul 21 17:26:39.887291 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:39.887261    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rg2jl"
Jul 21 17:26:39.887808 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:39.887783    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rg2jl"
Jul 21 17:26:40.759827 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:40.759606    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmzk"
Jul 21 17:26:40.760605 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:40.759618    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:26:40.760605 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:40.759911    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmzk" podUID="82279305-9475-429f-a495-143fc616fb34"
Jul 21 17:26:40.760605 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:40.759618    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8gnt"
Jul 21 17:26:40.760605 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:40.759972    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g76xh" podUID="fa6ca187-d1f3-4d33-bd20-f70784dbbe02"
Jul 21 17:26:40.760605 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:40.760038    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8gnt" podUID="b92707b0-e59d-4123-82e2-658e4361cb6e"
Jul 21 17:26:40.891111 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:40.891030    2563 generic.go:356] "Generic (PLEG): container finished" podID="35f84d29-3ea1-4c65-b000-d910c8012948" containerID="38641165dc143e1f804dc013e68594ac1083652ba929dbaa05c8d3b7fca21e02" exitCode=0
Jul 21 17:26:40.891285 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:40.891152    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5872t" event={"ID":"35f84d29-3ea1-4c65-b000-d910c8012948","Type":"ContainerDied","Data":"38641165dc143e1f804dc013e68594ac1083652ba929dbaa05c8d3b7fca21e02"}
Jul 21 17:26:40.894731 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:40.894710    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6" event={"ID":"a61ba40a-d3c2-45a9-8525-5e40c9de49b6","Type":"ContainerStarted","Data":"757cdf88acdc8a315cb8afa7f1bd4ae598ecaa967e66dcb530300f3ead0d62c9"}
Jul 21 17:26:40.947115 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:40.947068    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6" podStartSLOduration=8.512528199 podStartE2EDuration="27.947053763s" totalImagesPullingTime="19.434525564s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:26:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:40.946825021 +0000 UTC m=+28.708964624" watchObservedRunningTime="2026-07-21 17:26:40.947053763 +0000 UTC m=+28.709193366"
Jul 21 17:26:41.084482 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:41.084456    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gxckp_189f8b1f-23e2-4447-90bd-ffa91affc6c4/dns-node-resolver/0.log"
Jul 21 17:26:41.857470 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:41.857449    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mgp2n_5f7ddf4c-27b9-4b30-8d4a-483cfa1fb74a/node-ca/0.log"
Jul 21 17:26:41.897903 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:41.897878    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:41.898024 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:41.897943    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:41.898024 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:41.897982    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:41.916097 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:41.916063    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:41.916655 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:41.916630    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:26:42.057667 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:42.057493    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-d8gnt"]
Jul 21 17:26:42.057812 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:42.057777    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8gnt"
Jul 21 17:26:42.057917 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:42.057891    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8gnt" podUID="b92707b0-e59d-4123-82e2-658e4361cb6e"
Jul 21 17:26:42.060049 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:42.060026    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g76xh"]
Jul 21 17:26:42.060169 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:42.060108    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:26:42.060287 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:42.060263    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g76xh" podUID="fa6ca187-d1f3-4d33-bd20-f70784dbbe02"
Jul 21 17:26:42.073413 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:42.073359    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sxmzk"]
Jul 21 17:26:42.073504 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:42.073463    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmzk"
Jul 21 17:26:42.073579 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:42.073561    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmzk" podUID="82279305-9475-429f-a495-143fc616fb34"
Jul 21 17:26:42.901314 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:42.901276    2563 generic.go:356] "Generic (PLEG): container finished" podID="35f84d29-3ea1-4c65-b000-d910c8012948" containerID="2cccc397227e846c4928409b3700a709b8f1f6f6df92bb4eb7d9e8fe75da2be5" exitCode=0
Jul 21 17:26:42.901743 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:42.901353    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5872t" event={"ID":"35f84d29-3ea1-4c65-b000-d910c8012948","Type":"ContainerDied","Data":"2cccc397227e846c4928409b3700a709b8f1f6f6df92bb4eb7d9e8fe75da2be5"}
Jul 21 17:26:43.759156 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:43.759112    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8gnt"
Jul 21 17:26:43.759156 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:43.759127    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmzk"
Jul 21 17:26:43.759430 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:43.759112    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:26:43.759430 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:43.759279    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmzk" podUID="82279305-9475-429f-a495-143fc616fb34"
Jul 21 17:26:43.759430 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:43.759321    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g76xh" podUID="fa6ca187-d1f3-4d33-bd20-f70784dbbe02"
Jul 21 17:26:43.759430 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:43.759366    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8gnt" podUID="b92707b0-e59d-4123-82e2-658e4361cb6e"
Jul 21 17:26:44.907399 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:44.907367    2563 generic.go:356] "Generic (PLEG): container finished" podID="35f84d29-3ea1-4c65-b000-d910c8012948" containerID="e82a37efa36ddfcb856a3ad43b58098815ac76102d2e30e376a6a35c89cf64d4" exitCode=0
Jul 21 17:26:44.907999 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:44.907424    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5872t" event={"ID":"35f84d29-3ea1-4c65-b000-d910c8012948","Type":"ContainerDied","Data":"e82a37efa36ddfcb856a3ad43b58098815ac76102d2e30e376a6a35c89cf64d4"}
Jul 21 17:26:45.759669 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:45.759636    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8gnt"
Jul 21 17:26:45.759822 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:45.759710    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmzk"
Jul 21 17:26:45.759822 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:45.759805    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:26:45.759906 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:45.759815    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8gnt" podUID="b92707b0-e59d-4123-82e2-658e4361cb6e"
Jul 21 17:26:45.759906 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:45.759880    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g76xh" podUID="fa6ca187-d1f3-4d33-bd20-f70784dbbe02"
Jul 21 17:26:45.760018 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:45.759964    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmzk" podUID="82279305-9475-429f-a495-143fc616fb34"
Jul 21 17:26:46.353499 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:46.353469    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa6ca187-d1f3-4d33-bd20-f70784dbbe02-original-pull-secret\") pod \"global-pull-secret-syncer-g76xh\" (UID: \"fa6ca187-d1f3-4d33-bd20-f70784dbbe02\") " pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:26:46.354065 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:46.353608    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 21 17:26:46.354065 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:46.353669    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa6ca187-d1f3-4d33-bd20-f70784dbbe02-original-pull-secret podName:fa6ca187-d1f3-4d33-bd20-f70784dbbe02 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:18.353653262 +0000 UTC m=+66.115792848 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa6ca187-d1f3-4d33-bd20-f70784dbbe02-original-pull-secret") pod "global-pull-secret-syncer-g76xh" (UID: "fa6ca187-d1f3-4d33-bd20-f70784dbbe02") : object "kube-system"/"original-pull-secret" not registered
Jul 21 17:26:46.454475 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:46.454442    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82279305-9475-429f-a495-143fc616fb34-metrics-certs\") pod \"network-metrics-daemon-sxmzk\" (UID: \"82279305-9475-429f-a495-143fc616fb34\") " pod="openshift-multus/network-metrics-daemon-sxmzk"
Jul 21 17:26:46.454649 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:46.454621    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:26:46.454709 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:46.454696    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82279305-9475-429f-a495-143fc616fb34-metrics-certs podName:82279305-9475-429f-a495-143fc616fb34 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:18.454674108 +0000 UTC m=+66.216813707 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82279305-9475-429f-a495-143fc616fb34-metrics-certs") pod "network-metrics-daemon-sxmzk" (UID: "82279305-9475-429f-a495-143fc616fb34") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:26:46.555081 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:46.555053    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n2xt\" (UniqueName: \"kubernetes.io/projected/b92707b0-e59d-4123-82e2-658e4361cb6e-kube-api-access-9n2xt\") pod \"network-check-target-d8gnt\" (UID: \"b92707b0-e59d-4123-82e2-658e4361cb6e\") " pod="openshift-network-diagnostics/network-check-target-d8gnt"
Jul 21 17:26:46.555250 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:46.555229    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 21 17:26:46.555250 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:46.555251    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 21 17:26:46.555333 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:46.555262    2563 projected.go:196] Error preparing data for projected volume kube-api-access-9n2xt for pod openshift-network-diagnostics/network-check-target-d8gnt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:26:46.555333 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:46.555316    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b92707b0-e59d-4123-82e2-658e4361cb6e-kube-api-access-9n2xt podName:b92707b0-e59d-4123-82e2-658e4361cb6e nodeName:}" failed. No retries permitted until 2026-07-21 17:27:18.555299457 +0000 UTC m=+66.317439039 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9n2xt" (UniqueName: "kubernetes.io/projected/b92707b0-e59d-4123-82e2-658e4361cb6e-kube-api-access-9n2xt") pod "network-check-target-d8gnt" (UID: "b92707b0-e59d-4123-82e2-658e4361cb6e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:26:47.759016 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:47.758971    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8gnt"
Jul 21 17:26:47.759016 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:47.759000    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:26:47.759471 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:47.758988    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmzk"
Jul 21 17:26:47.759471 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:47.759098    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8gnt" podUID="b92707b0-e59d-4123-82e2-658e4361cb6e"
Jul 21 17:26:47.759471 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:47.759204    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmzk" podUID="82279305-9475-429f-a495-143fc616fb34"
Jul 21 17:26:47.759471 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:47.759282    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g76xh" podUID="fa6ca187-d1f3-4d33-bd20-f70784dbbe02"
Jul 21 17:26:49.759677 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:49.759484    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmzk"
Jul 21 17:26:49.760043 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:49.759484    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8gnt"
Jul 21 17:26:49.760043 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:49.759757    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmzk" podUID="82279305-9475-429f-a495-143fc616fb34"
Jul 21 17:26:49.760043 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:49.759487    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:26:49.760043 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:49.759883    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g76xh" podUID="fa6ca187-d1f3-4d33-bd20-f70784dbbe02"
Jul 21 17:26:49.760043 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:49.759810    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8gnt" podUID="b92707b0-e59d-4123-82e2-658e4361cb6e"
Jul 21 17:26:49.922798 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:49.922767    2563 generic.go:356] "Generic (PLEG): container finished" podID="35f84d29-3ea1-4c65-b000-d910c8012948" containerID="6cad39ea2bc79d71f0c288e25682ae08f4976ffc7f748449d1eb5852cdbf7119" exitCode=0
Jul 21 17:26:49.922964 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:49.922826    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5872t" event={"ID":"35f84d29-3ea1-4c65-b000-d910c8012948","Type":"ContainerDied","Data":"6cad39ea2bc79d71f0c288e25682ae08f4976ffc7f748449d1eb5852cdbf7119"}
Jul 21 17:26:50.928493 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:50.928458    2563 generic.go:356] "Generic (PLEG): container finished" podID="35f84d29-3ea1-4c65-b000-d910c8012948" containerID="389174a4fcb0a1cfafd6c141eb273ef3dfc45cdb1bd9de9ee40f7a3cd9f68d3e" exitCode=0
Jul 21 17:26:50.928904 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:50.928525    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5872t" event={"ID":"35f84d29-3ea1-4c65-b000-d910c8012948","Type":"ContainerDied","Data":"389174a4fcb0a1cfafd6c141eb273ef3dfc45cdb1bd9de9ee40f7a3cd9f68d3e"}
Jul 21 17:26:51.758889 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:51.758849    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8gnt"
Jul 21 17:26:51.759086 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:51.758950    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:26:51.759086 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:51.758955    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8gnt" podUID="b92707b0-e59d-4123-82e2-658e4361cb6e"
Jul 21 17:26:51.759086 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:51.758960    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmzk"
Jul 21 17:26:51.759086 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:51.759044    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g76xh" podUID="fa6ca187-d1f3-4d33-bd20-f70784dbbe02"
Jul 21 17:26:51.759328 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:51.759127    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmzk" podUID="82279305-9475-429f-a495-143fc616fb34"
Jul 21 17:26:51.934064 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:51.934030    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5872t" event={"ID":"35f84d29-3ea1-4c65-b000-d910c8012948","Type":"ContainerStarted","Data":"03cef4eb4fefc546ef20cd99f8dcb2b6270689bae35a079d8c5017760faf6097"}
Jul 21 17:26:53.758914 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:53.758873    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmzk"
Jul 21 17:26:53.759301 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:53.758940    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8gnt"
Jul 21 17:26:53.759301 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:53.759022    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:26:53.759301 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:53.759025    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmzk" podUID="82279305-9475-429f-a495-143fc616fb34"
Jul 21 17:26:53.759301 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:53.759090    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8gnt" podUID="b92707b0-e59d-4123-82e2-658e4361cb6e"
Jul 21 17:26:53.759301 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:53.759193    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g76xh" podUID="fa6ca187-d1f3-4d33-bd20-f70784dbbe02"
Jul 21 17:26:55.758992 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:55.758957    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmzk"
Jul 21 17:26:55.758992 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:55.758978    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:26:55.759433 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:55.758978    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8gnt"
Jul 21 17:26:55.759433 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:55.759066    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmzk" podUID="82279305-9475-429f-a495-143fc616fb34"
Jul 21 17:26:55.759433 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:55.759120    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g76xh" podUID="fa6ca187-d1f3-4d33-bd20-f70784dbbe02"
Jul 21 17:26:55.759433 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:55.759200    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8gnt" podUID="b92707b0-e59d-4123-82e2-658e4361cb6e"
Jul 21 17:26:57.758973 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:57.758939    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmzk"
Jul 21 17:26:57.759439 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:57.758939    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:26:57.759439 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:57.759043    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmzk" podUID="82279305-9475-429f-a495-143fc616fb34"
Jul 21 17:26:57.759439 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:57.758939    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8gnt"
Jul 21 17:26:57.759439 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:57.759164    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g76xh" podUID="fa6ca187-d1f3-4d33-bd20-f70784dbbe02"
Jul 21 17:26:57.759439 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:57.759244    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8gnt" podUID="b92707b0-e59d-4123-82e2-658e4361cb6e"
Jul 21 17:26:59.758746 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:59.758703    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8gnt"
Jul 21 17:26:59.759318 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:59.758825    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmzk"
Jul 21 17:26:59.759318 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:59.758838    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8gnt" podUID="b92707b0-e59d-4123-82e2-658e4361cb6e"
Jul 21 17:26:59.759318 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:26:59.758855    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:26:59.759318 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:59.758948    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmzk" podUID="82279305-9475-429f-a495-143fc616fb34"
Jul 21 17:26:59.759318 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:26:59.759025    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g76xh" podUID="fa6ca187-d1f3-4d33-bd20-f70784dbbe02"
Jul 21 17:27:01.759115 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:01.759082    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmzk"
Jul 21 17:27:01.759553 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:01.759082    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8gnt"
Jul 21 17:27:01.759553 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:27:01.759199    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmzk" podUID="82279305-9475-429f-a495-143fc616fb34"
Jul 21 17:27:01.759553 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:01.759082    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:27:01.759553 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:27:01.759282    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8gnt" podUID="b92707b0-e59d-4123-82e2-658e4361cb6e"
Jul 21 17:27:01.759553 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:27:01.759344    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g76xh" podUID="fa6ca187-d1f3-4d33-bd20-f70784dbbe02"
Jul 21 17:27:03.759069 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:03.759031    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8gnt"
Jul 21 17:27:03.759069 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:03.759063    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:27:03.759069 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:03.759076    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmzk"
Jul 21 17:27:03.759551 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:27:03.759176    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8gnt" podUID="b92707b0-e59d-4123-82e2-658e4361cb6e"
Jul 21 17:27:03.759551 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:27:03.759330    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g76xh" podUID="fa6ca187-d1f3-4d33-bd20-f70784dbbe02"
Jul 21 17:27:03.759551 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:27:03.759423    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmzk" podUID="82279305-9475-429f-a495-143fc616fb34"
Jul 21 17:27:04.819261 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:04.819205    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5872t" podStartSLOduration=22.150380218 podStartE2EDuration="51.819188679s" totalImagesPullingTime="29.668808461s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:26:13 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:51.974580347 +0000 UTC m=+39.736719951" watchObservedRunningTime="2026-07-21 17:27:04.819188679 +0000 UTC m=+52.581328283"
Jul 21 17:27:05.758733 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:05.758699    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmzk"
Jul 21 17:27:05.758733 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:05.758719    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:27:05.758733 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:05.758699    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8gnt"
Jul 21 17:27:05.759028 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:27:05.758834    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g76xh" podUID="fa6ca187-d1f3-4d33-bd20-f70784dbbe02"
Jul 21 17:27:05.759028 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:27:05.758875    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8gnt" podUID="b92707b0-e59d-4123-82e2-658e4361cb6e"
Jul 21 17:27:05.759028 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:27:05.758940    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmzk" podUID="82279305-9475-429f-a495-143fc616fb34"
Jul 21 17:27:06.096628 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.096604    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-249.ec2.internal" event="NodeReady"
Jul 21 17:27:06.097013 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.096740    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 21 17:27:06.155360 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.155330    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-c9sl8"]
Jul 21 17:27:06.175553 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.175520    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dptd4"]
Jul 21 17:27:06.175702 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.175645    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c9sl8"
Jul 21 17:27:06.184307 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.184287    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 21 17:27:06.184430 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.184287    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vzv2x\""
Jul 21 17:27:06.184430 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.184339    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 21 17:27:06.196352 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.196329    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c546573-c93f-4765-bc1d-94c195e3a6da-metrics-tls\") pod \"dns-default-c9sl8\" (UID: \"0c546573-c93f-4765-bc1d-94c195e3a6da\") " pod="openshift-dns/dns-default-c9sl8"
Jul 21 17:27:06.196463 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.196363    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c4jf\" (UniqueName: \"kubernetes.io/projected/0c546573-c93f-4765-bc1d-94c195e3a6da-kube-api-access-9c4jf\") pod \"dns-default-c9sl8\" (UID: \"0c546573-c93f-4765-bc1d-94c195e3a6da\") " pod="openshift-dns/dns-default-c9sl8"
Jul 21 17:27:06.196463 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.196390    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0c546573-c93f-4765-bc1d-94c195e3a6da-config-volume\") pod \"dns-default-c9sl8\" (UID: \"0c546573-c93f-4765-bc1d-94c195e3a6da\") " pod="openshift-dns/dns-default-c9sl8"
Jul 21 17:27:06.196463 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.196437    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c546573-c93f-4765-bc1d-94c195e3a6da-tmp-dir\") pod \"dns-default-c9sl8\" (UID: \"0c546573-c93f-4765-bc1d-94c195e3a6da\") " pod="openshift-dns/dns-default-c9sl8"
Jul 21 17:27:06.202093 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.202069    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c9sl8"]
Jul 21 17:27:06.202233 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.202194    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dptd4"
Jul 21 17:27:06.202300 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.202284    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dptd4"]
Jul 21 17:27:06.208012 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.207983    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 21 17:27:06.208196 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.207999    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-v2zc6\""
Jul 21 17:27:06.208465 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.208445    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 21 17:27:06.208615 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.208597    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 21 17:27:06.208684 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.208669    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 21 17:27:06.208881 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.208864    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-x99p9"]
Jul 21 17:27:06.232625 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.232596    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x99p9"]
Jul 21 17:27:06.232770 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.232685    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x99p9"
Jul 21 17:27:06.235015 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.234997    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 21 17:27:06.235326 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.235299    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2bv6x\""
Jul 21 17:27:06.235406 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.235379    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 21 17:27:06.235645 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.235632    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 21 17:27:06.297510 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.297474    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc78g\" (UniqueName: \"kubernetes.io/projected/15d6706f-5736-4ae6-b58d-677e456478c7-kube-api-access-hc78g\") pod \"insights-runtime-extractor-dptd4\" (UID: \"15d6706f-5736-4ae6-b58d-677e456478c7\") " pod="openshift-insights/insights-runtime-extractor-dptd4"
Jul 21 17:27:06.297677 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.297530    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/15d6706f-5736-4ae6-b58d-677e456478c7-crio-socket\") pod \"insights-runtime-extractor-dptd4\" (UID: \"15d6706f-5736-4ae6-b58d-677e456478c7\") " pod="openshift-insights/insights-runtime-extractor-dptd4"
Jul 21 17:27:06.297677 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.297551    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/15d6706f-5736-4ae6-b58d-677e456478c7-data-volume\") pod \"insights-runtime-extractor-dptd4\" (UID: \"15d6706f-5736-4ae6-b58d-677e456478c7\") " pod="openshift-insights/insights-runtime-extractor-dptd4"
Jul 21 17:27:06.297677 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.297594    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c546573-c93f-4765-bc1d-94c195e3a6da-metrics-tls\") pod \"dns-default-c9sl8\" (UID: \"0c546573-c93f-4765-bc1d-94c195e3a6da\") " pod="openshift-dns/dns-default-c9sl8"
Jul 21 17:27:06.297677 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.297623    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/15d6706f-5736-4ae6-b58d-677e456478c7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dptd4\" (UID: \"15d6706f-5736-4ae6-b58d-677e456478c7\") " pod="openshift-insights/insights-runtime-extractor-dptd4"
Jul 21 17:27:06.297677 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.297650    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c4jf\" (UniqueName: \"kubernetes.io/projected/0c546573-c93f-4765-bc1d-94c195e3a6da-kube-api-access-9c4jf\") pod \"dns-default-c9sl8\" (UID: \"0c546573-c93f-4765-bc1d-94c195e3a6da\") " pod="openshift-dns/dns-default-c9sl8"
Jul 21 17:27:06.297677 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.297675    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/15d6706f-5736-4ae6-b58d-677e456478c7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dptd4\" (UID: \"15d6706f-5736-4ae6-b58d-677e456478c7\") " pod="openshift-insights/insights-runtime-extractor-dptd4"
Jul 21 17:27:06.297914 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.297706    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0c546573-c93f-4765-bc1d-94c195e3a6da-config-volume\") pod \"dns-default-c9sl8\" (UID: \"0c546573-c93f-4765-bc1d-94c195e3a6da\") " pod="openshift-dns/dns-default-c9sl8"
Jul 21 17:27:06.297914 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.297727    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c546573-c93f-4765-bc1d-94c195e3a6da-tmp-dir\") pod \"dns-default-c9sl8\" (UID: \"0c546573-c93f-4765-bc1d-94c195e3a6da\") " pod="openshift-dns/dns-default-c9sl8"
Jul 21 17:27:06.297914 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.297751    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c359a027-359d-42f9-91e4-27af042f8d9d-cert\") pod \"ingress-canary-x99p9\" (UID: \"c359a027-359d-42f9-91e4-27af042f8d9d\") " pod="openshift-ingress-canary/ingress-canary-x99p9"
Jul 21 17:27:06.297914 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.297773    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbsm5\" (UniqueName: \"kubernetes.io/projected/c359a027-359d-42f9-91e4-27af042f8d9d-kube-api-access-rbsm5\") pod \"ingress-canary-x99p9\" (UID: \"c359a027-359d-42f9-91e4-27af042f8d9d\") " pod="openshift-ingress-canary/ingress-canary-x99p9"
Jul 21 17:27:06.314487 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.314457    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c546573-c93f-4765-bc1d-94c195e3a6da-tmp-dir\") pod \"dns-default-c9sl8\" (UID: \"0c546573-c93f-4765-bc1d-94c195e3a6da\") " pod="openshift-dns/dns-default-c9sl8"
Jul 21 17:27:06.314782 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.314746    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0c546573-c93f-4765-bc1d-94c195e3a6da-config-volume\") pod \"dns-default-c9sl8\" (UID: \"0c546573-c93f-4765-bc1d-94c195e3a6da\") " pod="openshift-dns/dns-default-c9sl8"
Jul 21 17:27:06.314782 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.314759    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c546573-c93f-4765-bc1d-94c195e3a6da-metrics-tls\") pod \"dns-default-c9sl8\" (UID: \"0c546573-c93f-4765-bc1d-94c195e3a6da\") " pod="openshift-dns/dns-default-c9sl8"
Jul 21 17:27:06.316626 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.316608    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c4jf\" (UniqueName: \"kubernetes.io/projected/0c546573-c93f-4765-bc1d-94c195e3a6da-kube-api-access-9c4jf\") pod \"dns-default-c9sl8\" (UID: \"0c546573-c93f-4765-bc1d-94c195e3a6da\") " pod="openshift-dns/dns-default-c9sl8"
Jul 21 17:27:06.398800 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.398761    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/15d6706f-5736-4ae6-b58d-677e456478c7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dptd4\" (UID: \"15d6706f-5736-4ae6-b58d-677e456478c7\") " pod="openshift-insights/insights-runtime-extractor-dptd4"
Jul 21 17:27:06.398980 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.398808    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/15d6706f-5736-4ae6-b58d-677e456478c7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dptd4\" (UID: \"15d6706f-5736-4ae6-b58d-677e456478c7\") " pod="openshift-insights/insights-runtime-extractor-dptd4"
Jul 21 17:27:06.398980 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.398847    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c359a027-359d-42f9-91e4-27af042f8d9d-cert\") pod \"ingress-canary-x99p9\" (UID: \"c359a027-359d-42f9-91e4-27af042f8d9d\") " pod="openshift-ingress-canary/ingress-canary-x99p9"
Jul 21 17:27:06.398980 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.398871    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbsm5\" (UniqueName: \"kubernetes.io/projected/c359a027-359d-42f9-91e4-27af042f8d9d-kube-api-access-rbsm5\") pod \"ingress-canary-x99p9\" (UID: \"c359a027-359d-42f9-91e4-27af042f8d9d\") " pod="openshift-ingress-canary/ingress-canary-x99p9"
Jul 21 17:27:06.398980 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.398895    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc78g\" (UniqueName: \"kubernetes.io/projected/15d6706f-5736-4ae6-b58d-677e456478c7-kube-api-access-hc78g\") pod \"insights-runtime-extractor-dptd4\" (UID: \"15d6706f-5736-4ae6-b58d-677e456478c7\") " pod="openshift-insights/insights-runtime-extractor-dptd4"
Jul 21 17:27:06.398980 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.398930    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/15d6706f-5736-4ae6-b58d-677e456478c7-crio-socket\") pod \"insights-runtime-extractor-dptd4\" (UID: \"15d6706f-5736-4ae6-b58d-677e456478c7\") " pod="openshift-insights/insights-runtime-extractor-dptd4"
Jul 21 17:27:06.398980 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.398951    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/15d6706f-5736-4ae6-b58d-677e456478c7-data-volume\") pod \"insights-runtime-extractor-dptd4\" (UID: \"15d6706f-5736-4ae6-b58d-677e456478c7\") " pod="openshift-insights/insights-runtime-extractor-dptd4"
Jul 21 17:27:06.399280 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.399191    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/15d6706f-5736-4ae6-b58d-677e456478c7-crio-socket\") pod \"insights-runtime-extractor-dptd4\" (UID: \"15d6706f-5736-4ae6-b58d-677e456478c7\") " pod="openshift-insights/insights-runtime-extractor-dptd4"
Jul 21 17:27:06.399341 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.399289    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/15d6706f-5736-4ae6-b58d-677e456478c7-data-volume\") pod \"insights-runtime-extractor-dptd4\" (UID: \"15d6706f-5736-4ae6-b58d-677e456478c7\") " pod="openshift-insights/insights-runtime-extractor-dptd4"
Jul 21 17:27:06.399460 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.399438    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/15d6706f-5736-4ae6-b58d-677e456478c7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dptd4\" (UID: \"15d6706f-5736-4ae6-b58d-677e456478c7\") " pod="openshift-insights/insights-runtime-extractor-dptd4"
Jul 21 17:27:06.401321 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.401301    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/15d6706f-5736-4ae6-b58d-677e456478c7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dptd4\" (UID: \"15d6706f-5736-4ae6-b58d-677e456478c7\") " pod="openshift-insights/insights-runtime-extractor-dptd4"
Jul 21 17:27:06.401457 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.401438    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c359a027-359d-42f9-91e4-27af042f8d9d-cert\") pod \"ingress-canary-x99p9\" (UID: \"c359a027-359d-42f9-91e4-27af042f8d9d\") " pod="openshift-ingress-canary/ingress-canary-x99p9"
Jul 21 17:27:06.407428 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.407397    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc78g\" (UniqueName: \"kubernetes.io/projected/15d6706f-5736-4ae6-b58d-677e456478c7-kube-api-access-hc78g\") pod \"insights-runtime-extractor-dptd4\" (UID: \"15d6706f-5736-4ae6-b58d-677e456478c7\") " pod="openshift-insights/insights-runtime-extractor-dptd4"
Jul 21 17:27:06.407554 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.407531    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbsm5\" (UniqueName: \"kubernetes.io/projected/c359a027-359d-42f9-91e4-27af042f8d9d-kube-api-access-rbsm5\") pod \"ingress-canary-x99p9\" (UID: \"c359a027-359d-42f9-91e4-27af042f8d9d\") " pod="openshift-ingress-canary/ingress-canary-x99p9"
Jul 21 17:27:06.484083 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.484048    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c9sl8"
Jul 21 17:27:06.510869 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.510834    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dptd4"
Jul 21 17:27:06.540946 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.540906    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x99p9"
Jul 21 17:27:06.697407 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.697379    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c9sl8"]
Jul 21 17:27:06.700459 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:27:06.700424    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c546573_c93f_4765_bc1d_94c195e3a6da.slice/crio-28c2298ec929f43b6b260fbc6ed76141de195fb6955fc17bda0d04872eb9c61f WatchSource:0}: Error finding container 28c2298ec929f43b6b260fbc6ed76141de195fb6955fc17bda0d04872eb9c61f: Status 404 returned error can't find the container with id 28c2298ec929f43b6b260fbc6ed76141de195fb6955fc17bda0d04872eb9c61f
Jul 21 17:27:06.703897 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.703872    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dptd4"]
Jul 21 17:27:06.706572 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:27:06.706549    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15d6706f_5736_4ae6_b58d_677e456478c7.slice/crio-15c6806bbe7222285dda40deeedce9393c3365addf248b55a06d9a3ab5862254 WatchSource:0}: Error finding container 15c6806bbe7222285dda40deeedce9393c3365addf248b55a06d9a3ab5862254: Status 404 returned error can't find the container with id 15c6806bbe7222285dda40deeedce9393c3365addf248b55a06d9a3ab5862254
Jul 21 17:27:06.717894 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.717870    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x99p9"]
Jul 21 17:27:06.721177 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:27:06.721156    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc359a027_359d_42f9_91e4_27af042f8d9d.slice/crio-bd3443c2a99262fa4df62500ac0e9759eb85524e8a14a1884d1c518ec998933e WatchSource:0}: Error finding container bd3443c2a99262fa4df62500ac0e9759eb85524e8a14a1884d1c518ec998933e: Status 404 returned error can't find the container with id bd3443c2a99262fa4df62500ac0e9759eb85524e8a14a1884d1c518ec998933e
Jul 21 17:27:06.968190 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.968079    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x99p9" event={"ID":"c359a027-359d-42f9-91e4-27af042f8d9d","Type":"ContainerStarted","Data":"bd3443c2a99262fa4df62500ac0e9759eb85524e8a14a1884d1c518ec998933e"}
Jul 21 17:27:06.969257 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.969229    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dptd4" event={"ID":"15d6706f-5736-4ae6-b58d-677e456478c7","Type":"ContainerStarted","Data":"15c6806bbe7222285dda40deeedce9393c3365addf248b55a06d9a3ab5862254"}
Jul 21 17:27:06.970906 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:06.970889    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c9sl8" event={"ID":"0c546573-c93f-4765-bc1d-94c195e3a6da","Type":"ContainerStarted","Data":"28c2298ec929f43b6b260fbc6ed76141de195fb6955fc17bda0d04872eb9c61f"}
Jul 21 17:27:07.759474 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:07.759442    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmzk"
Jul 21 17:27:07.759917 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:07.759444    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:27:07.759917 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:07.759602    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8gnt"
Jul 21 17:27:07.763283 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:07.763205    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 21 17:27:07.763469 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:07.763448    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nmxsm\""
Jul 21 17:27:07.763574 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:07.763522    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 21 17:27:07.763660 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:07.763641    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 21 17:27:07.763732 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:07.763705    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sxr4r\""
Jul 21 17:27:07.763776 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:07.763453    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 21 17:27:09.631172 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:09.631121    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-znnms"]
Jul 21 17:27:09.657000 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:09.656969    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-znnms"]
Jul 21 17:27:09.657000 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:09.656991    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-znnms"
Jul 21 17:27:09.661040 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:09.661015    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 21 17:27:09.661203 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:09.661076    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-gtw9w\""
Jul 21 17:27:09.725287 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:09.725255    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d7cd9ffd-b8cf-4df1-a7a0-1087096d20a5-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-znnms\" (UID: \"d7cd9ffd-b8cf-4df1-a7a0-1087096d20a5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-znnms"
Jul 21 17:27:09.825762 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:09.825727    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d7cd9ffd-b8cf-4df1-a7a0-1087096d20a5-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-znnms\" (UID: \"d7cd9ffd-b8cf-4df1-a7a0-1087096d20a5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-znnms"
Jul 21 17:27:09.825920 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:27:09.825880    2563 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 21 17:27:09.825964 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:27:09.825949    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7cd9ffd-b8cf-4df1-a7a0-1087096d20a5-tls-certificates podName:d7cd9ffd-b8cf-4df1-a7a0-1087096d20a5 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:10.325932056 +0000 UTC m=+58.088071637 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/d7cd9ffd-b8cf-4df1-a7a0-1087096d20a5-tls-certificates") pod "prometheus-operator-admission-webhook-85574b5f49-znnms" (UID: "d7cd9ffd-b8cf-4df1-a7a0-1087096d20a5") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 21 17:27:10.329118 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:10.329074    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d7cd9ffd-b8cf-4df1-a7a0-1087096d20a5-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-znnms\" (UID: \"d7cd9ffd-b8cf-4df1-a7a0-1087096d20a5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-znnms"
Jul 21 17:27:10.332054 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:10.332030    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d7cd9ffd-b8cf-4df1-a7a0-1087096d20a5-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-znnms\" (UID: \"d7cd9ffd-b8cf-4df1-a7a0-1087096d20a5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-znnms"
Jul 21 17:27:10.567977 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:10.567938    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-znnms"
Jul 21 17:27:12.561991 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:12.561955    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-znnms"]
Jul 21 17:27:12.779524 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:27:12.779492    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7cd9ffd_b8cf_4df1_a7a0_1087096d20a5.slice/crio-397674cae3717c62f79db15c7acec2e59eeb77d845cff1f81727bfa765fb3380 WatchSource:0}: Error finding container 397674cae3717c62f79db15c7acec2e59eeb77d845cff1f81727bfa765fb3380: Status 404 returned error can't find the container with id 397674cae3717c62f79db15c7acec2e59eeb77d845cff1f81727bfa765fb3380
Jul 21 17:27:12.988968 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:12.988929    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c9sl8" event={"ID":"0c546573-c93f-4765-bc1d-94c195e3a6da","Type":"ContainerStarted","Data":"bb8f0cb0939ec07cc14178397a97b53af9f7ef2c8d42341f4e1326570d373bab"}
Jul 21 17:27:12.989124 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:12.989051    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-c9sl8"
Jul 21 17:27:12.989124 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:12.989082    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c9sl8" event={"ID":"0c546573-c93f-4765-bc1d-94c195e3a6da","Type":"ContainerStarted","Data":"43a2e4bd0ff07d9d5ba540046c5564fa5b1ce605047590ff338873532ca1ec13"}
Jul 21 17:27:12.990513 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:12.990491    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x99p9" event={"ID":"c359a027-359d-42f9-91e4-27af042f8d9d","Type":"ContainerStarted","Data":"dd54fc81d20aa81ded81a7d5b4d53debdc2b0bf7568259ff68fdf5787b03f9fe"}
Jul 21 17:27:12.992072 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:12.992051    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dptd4" event={"ID":"15d6706f-5736-4ae6-b58d-677e456478c7","Type":"ContainerStarted","Data":"8c731f912383d3d004d73a51276b4dcb703a89de235f5fdf732923ccb5bb6a5a"}
Jul 21 17:27:12.993318 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:12.993299    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-znnms" event={"ID":"d7cd9ffd-b8cf-4df1-a7a0-1087096d20a5","Type":"ContainerStarted","Data":"397674cae3717c62f79db15c7acec2e59eeb77d845cff1f81727bfa765fb3380"}
Jul 21 17:27:13.062004 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:13.061902    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-x99p9" podStartSLOduration=1.007746432 podStartE2EDuration="7.061887423s" totalImagesPullingTime="6.054140991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:27:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:13.061464947 +0000 UTC m=+60.823604564" watchObservedRunningTime="2026-07-21 17:27:13.061887423 +0000 UTC m=+60.824027028"
Jul 21 17:27:13.085762 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:13.085704    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-c9sl8" podStartSLOduration=1.0107323 podStartE2EDuration="7.085688377s" totalImagesPullingTime="6.074956077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:27:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:13.084882327 +0000 UTC m=+60.847021935" watchObservedRunningTime="2026-07-21 17:27:13.085688377 +0000 UTC m=+60.847827981"
Jul 21 17:27:13.916721 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:13.916697    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dq7d6"
Jul 21 17:27:13.999167 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:13.998683    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-znnms" event={"ID":"d7cd9ffd-b8cf-4df1-a7a0-1087096d20a5","Type":"ContainerStarted","Data":"abb54231ad35f2dfdd42422da6df86c700073637dcba5f2a7b86f4e8632b2d7b"}
Jul 21 17:27:13.999167 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:13.998891    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-znnms"
Jul 21 17:27:14.005499 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:14.005446    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-znnms"
Jul 21 17:27:14.025325 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:14.024149    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-znnms" podStartSLOduration=3.970231587 podStartE2EDuration="5.024114019s" totalImagesPullingTime="1.053882432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:14.023261395 +0000 UTC m=+61.785401000" watchObservedRunningTime="2026-07-21 17:27:14.024114019 +0000 UTC m=+61.786253625"
Jul 21 17:27:14.676363 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:14.676331    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-9cd5l"]
Jul 21 17:27:14.680730 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:14.680709    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-9cd5l"
Jul 21 17:27:14.683377 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:14.683353    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 21 17:27:14.683509 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:14.683390    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 21 17:27:14.684358 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:14.684326    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 21 17:27:14.684358 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:14.684339    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-477qx\""
Jul 21 17:27:14.684545 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:14.684375    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 21 17:27:14.684545 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:14.684346    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 21 17:27:14.689382 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:14.689365    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-9cd5l"]
Jul 21 17:27:14.768342 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:14.768311    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/23805b10-82e7-4dde-9ecc-4c294d17e1e8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-9cd5l\" (UID: \"23805b10-82e7-4dde-9ecc-4c294d17e1e8\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9cd5l"
Jul 21 17:27:14.768534 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:14.768461    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/23805b10-82e7-4dde-9ecc-4c294d17e1e8-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-9cd5l\" (UID: \"23805b10-82e7-4dde-9ecc-4c294d17e1e8\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9cd5l"
Jul 21 17:27:14.768534 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:14.768512    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23805b10-82e7-4dde-9ecc-4c294d17e1e8-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-9cd5l\" (UID: \"23805b10-82e7-4dde-9ecc-4c294d17e1e8\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9cd5l"
Jul 21 17:27:14.768657 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:14.768538    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qktqt\" (UniqueName: \"kubernetes.io/projected/23805b10-82e7-4dde-9ecc-4c294d17e1e8-kube-api-access-qktqt\") pod \"prometheus-operator-5945b5cfd-9cd5l\" (UID: \"23805b10-82e7-4dde-9ecc-4c294d17e1e8\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9cd5l"
Jul 21 17:27:14.869654 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:14.869610    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/23805b10-82e7-4dde-9ecc-4c294d17e1e8-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-9cd5l\" (UID: \"23805b10-82e7-4dde-9ecc-4c294d17e1e8\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9cd5l"
Jul 21 17:27:14.869879 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:14.869670    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23805b10-82e7-4dde-9ecc-4c294d17e1e8-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-9cd5l\" (UID: \"23805b10-82e7-4dde-9ecc-4c294d17e1e8\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9cd5l"
Jul 21 17:27:14.869879 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:14.869745    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qktqt\" (UniqueName: \"kubernetes.io/projected/23805b10-82e7-4dde-9ecc-4c294d17e1e8-kube-api-access-qktqt\") pod \"prometheus-operator-5945b5cfd-9cd5l\" (UID: \"23805b10-82e7-4dde-9ecc-4c294d17e1e8\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9cd5l"
Jul 21 17:27:14.869879 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:14.869779    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/23805b10-82e7-4dde-9ecc-4c294d17e1e8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-9cd5l\" (UID: \"23805b10-82e7-4dde-9ecc-4c294d17e1e8\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9cd5l"
Jul 21 17:27:14.869879 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:27:14.869795    2563 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 21 17:27:14.869879 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:27:14.869868    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23805b10-82e7-4dde-9ecc-4c294d17e1e8-prometheus-operator-tls podName:23805b10-82e7-4dde-9ecc-4c294d17e1e8 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:15.369851794 +0000 UTC m=+63.131991376 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/23805b10-82e7-4dde-9ecc-4c294d17e1e8-prometheus-operator-tls") pod "prometheus-operator-5945b5cfd-9cd5l" (UID: "23805b10-82e7-4dde-9ecc-4c294d17e1e8") : secret "prometheus-operator-tls" not found
Jul 21 17:27:14.870393 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:14.870371    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23805b10-82e7-4dde-9ecc-4c294d17e1e8-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-9cd5l\" (UID: \"23805b10-82e7-4dde-9ecc-4c294d17e1e8\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9cd5l"
Jul 21 17:27:14.872469 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:14.872448    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/23805b10-82e7-4dde-9ecc-4c294d17e1e8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-9cd5l\" (UID: \"23805b10-82e7-4dde-9ecc-4c294d17e1e8\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9cd5l"
Jul 21 17:27:14.881349 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:14.881323    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qktqt\" (UniqueName: \"kubernetes.io/projected/23805b10-82e7-4dde-9ecc-4c294d17e1e8-kube-api-access-qktqt\") pod \"prometheus-operator-5945b5cfd-9cd5l\" (UID: \"23805b10-82e7-4dde-9ecc-4c294d17e1e8\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9cd5l"
Jul 21 17:27:15.373564 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:15.373521    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/23805b10-82e7-4dde-9ecc-4c294d17e1e8-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-9cd5l\" (UID: \"23805b10-82e7-4dde-9ecc-4c294d17e1e8\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9cd5l"
Jul 21 17:27:15.376667 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:15.376638    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/23805b10-82e7-4dde-9ecc-4c294d17e1e8-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-9cd5l\" (UID: \"23805b10-82e7-4dde-9ecc-4c294d17e1e8\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9cd5l"
Jul 21 17:27:15.591645 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:15.591599    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-9cd5l"
Jul 21 17:27:16.578955 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:16.578789    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-9cd5l"]
Jul 21 17:27:16.581266 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:27:16.581233    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23805b10_82e7_4dde_9ecc_4c294d17e1e8.slice/crio-dde58acff4b2e92b0a1852db52fe978b570db94b5346e160f7ae079100493c52 WatchSource:0}: Error finding container dde58acff4b2e92b0a1852db52fe978b570db94b5346e160f7ae079100493c52: Status 404 returned error can't find the container with id dde58acff4b2e92b0a1852db52fe978b570db94b5346e160f7ae079100493c52
Jul 21 17:27:17.010860 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:17.010824    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dptd4" event={"ID":"15d6706f-5736-4ae6-b58d-677e456478c7","Type":"ContainerStarted","Data":"4c06597715b600110625a4f2e8342a5735c9aae2b0ca302a4ad2482881160fce"}
Jul 21 17:27:17.012124 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:17.012097    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-9cd5l" event={"ID":"23805b10-82e7-4dde-9ecc-4c294d17e1e8","Type":"ContainerStarted","Data":"dde58acff4b2e92b0a1852db52fe978b570db94b5346e160f7ae079100493c52"}
Jul 21 17:27:18.018404 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:18.018368    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-9cd5l" event={"ID":"23805b10-82e7-4dde-9ecc-4c294d17e1e8","Type":"ContainerStarted","Data":"741cc50db53ee48fa343de58e99bb4c6cd9372e7dcaafb7d361efe10dce73af4"}
Jul 21 17:27:18.018869 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:18.018434    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-9cd5l" event={"ID":"23805b10-82e7-4dde-9ecc-4c294d17e1e8","Type":"ContainerStarted","Data":"da9b769a951f9061ac73a8f248d9668bdfcc136f5fa766f388879c32128aed43"}
Jul 21 17:27:18.037644 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:18.037588    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-9cd5l" podStartSLOduration=2.771363697 podStartE2EDuration="4.037566799s" totalImagesPullingTime="1.266203102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:27:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:18.036545853 +0000 UTC m=+65.798685482" watchObservedRunningTime="2026-07-21 17:27:18.037566799 +0000 UTC m=+65.799706406"
Jul 21 17:27:18.396816 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:18.396779    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa6ca187-d1f3-4d33-bd20-f70784dbbe02-original-pull-secret\") pod \"global-pull-secret-syncer-g76xh\" (UID: \"fa6ca187-d1f3-4d33-bd20-f70784dbbe02\") " pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:27:18.399553 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:18.399526    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 21 17:27:18.410423 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:18.410396    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa6ca187-d1f3-4d33-bd20-f70784dbbe02-original-pull-secret\") pod \"global-pull-secret-syncer-g76xh\" (UID: \"fa6ca187-d1f3-4d33-bd20-f70784dbbe02\") " pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:27:18.497898 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:18.497858    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82279305-9475-429f-a495-143fc616fb34-metrics-certs\") pod \"network-metrics-daemon-sxmzk\" (UID: \"82279305-9475-429f-a495-143fc616fb34\") " pod="openshift-multus/network-metrics-daemon-sxmzk"
Jul 21 17:27:18.500742 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:18.500715    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 21 17:27:18.511267 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:18.511234    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82279305-9475-429f-a495-143fc616fb34-metrics-certs\") pod \"network-metrics-daemon-sxmzk\" (UID: \"82279305-9475-429f-a495-143fc616fb34\") " pod="openshift-multus/network-metrics-daemon-sxmzk"
Jul 21 17:27:18.580357 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:18.580324    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nmxsm\""
Jul 21 17:27:18.583534 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:18.583505    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g76xh"
Jul 21 17:27:18.588559 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:18.588536    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmzk"
Jul 21 17:27:18.598467 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:18.598445    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n2xt\" (UniqueName: \"kubernetes.io/projected/b92707b0-e59d-4123-82e2-658e4361cb6e-kube-api-access-9n2xt\") pod \"network-check-target-d8gnt\" (UID: \"b92707b0-e59d-4123-82e2-658e4361cb6e\") " pod="openshift-network-diagnostics/network-check-target-d8gnt"
Jul 21 17:27:18.601325 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:18.601298    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 21 17:27:18.611988 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:18.611971    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 21 17:27:18.622802 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:18.622781    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n2xt\" (UniqueName: \"kubernetes.io/projected/b92707b0-e59d-4123-82e2-658e4361cb6e-kube-api-access-9n2xt\") pod \"network-check-target-d8gnt\" (UID: \"b92707b0-e59d-4123-82e2-658e4361cb6e\") " pod="openshift-network-diagnostics/network-check-target-d8gnt"
Jul 21 17:27:18.895145 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:18.895097    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sxr4r\""
Jul 21 17:27:18.902437 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:18.902413    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8gnt"
Jul 21 17:27:20.114051 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.114027    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sxmzk"]
Jul 21 17:27:20.116713 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:27:20.116691    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82279305_9475_429f_a495_143fc616fb34.slice/crio-2f0a949c122cb107bea048135e87cca22c0c9aef0339298d31dd82a70b2a1fe6 WatchSource:0}: Error finding container 2f0a949c122cb107bea048135e87cca22c0c9aef0339298d31dd82a70b2a1fe6: Status 404 returned error can't find the container with id 2f0a949c122cb107bea048135e87cca22c0c9aef0339298d31dd82a70b2a1fe6
Jul 21 17:27:20.239537 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.239457    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-rn2q5"]
Jul 21 17:27:20.241733 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.241714    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-rn2q5"
Jul 21 17:27:20.244038 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.244021    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 21 17:27:20.244175 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.244024    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-hlll7\""
Jul 21 17:27:20.244380 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.244365    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 21 17:27:20.244993 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.244980    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 21 17:27:20.253615 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.253481    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5hwc2"]
Jul 21 17:27:20.255740 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.255725    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5hwc2"
Jul 21 17:27:20.258062 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.258041    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 21 17:27:20.258202 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.258188    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jhq4k\""
Jul 21 17:27:20.258515 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.258501    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 21 17:27:20.258930 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.258913    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 21 17:27:20.266774 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.266755    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-rn2q5"]
Jul 21 17:27:20.313495 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.313464    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7fe58db6-f487-48b9-a4cc-16d5a4844298-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-rn2q5\" (UID: \"7fe58db6-f487-48b9-a4cc-16d5a4844298\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rn2q5"
Jul 21 17:27:20.313495 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.313494    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjlj6\" (UniqueName: \"kubernetes.io/projected/7fe58db6-f487-48b9-a4cc-16d5a4844298-kube-api-access-zjlj6\") pod \"kube-state-metrics-56c5594674-rn2q5\" (UID: \"7fe58db6-f487-48b9-a4cc-16d5a4844298\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rn2q5"
Jul 21 17:27:20.313675 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.313513    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee650e41-f4c6-4ff7-bcf1-e9b74259434e-sys\") pod \"node-exporter-5hwc2\" (UID: \"ee650e41-f4c6-4ff7-bcf1-e9b74259434e\") " pod="openshift-monitoring/node-exporter-5hwc2"
Jul 21 17:27:20.313675 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.313527    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ee650e41-f4c6-4ff7-bcf1-e9b74259434e-node-exporter-wtmp\") pod \"node-exporter-5hwc2\" (UID: \"ee650e41-f4c6-4ff7-bcf1-e9b74259434e\") " pod="openshift-monitoring/node-exporter-5hwc2"
Jul 21 17:27:20.313675 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.313543    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7fe58db6-f487-48b9-a4cc-16d5a4844298-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-rn2q5\" (UID: \"7fe58db6-f487-48b9-a4cc-16d5a4844298\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rn2q5"
Jul 21 17:27:20.313675 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.313621    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ee650e41-f4c6-4ff7-bcf1-e9b74259434e-node-exporter-accelerators-collector-config\") pod \"node-exporter-5hwc2\" (UID: \"ee650e41-f4c6-4ff7-bcf1-e9b74259434e\") " pod="openshift-monitoring/node-exporter-5hwc2"
Jul 21 17:27:20.313675 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.313666    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7fe58db6-f487-48b9-a4cc-16d5a4844298-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-rn2q5\" (UID: \"7fe58db6-f487-48b9-a4cc-16d5a4844298\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rn2q5"
Jul 21 17:27:20.313823 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.313687    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee650e41-f4c6-4ff7-bcf1-e9b74259434e-metrics-client-ca\") pod \"node-exporter-5hwc2\" (UID: \"ee650e41-f4c6-4ff7-bcf1-e9b74259434e\") " pod="openshift-monitoring/node-exporter-5hwc2"
Jul 21 17:27:20.313823 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.313707    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ee650e41-f4c6-4ff7-bcf1-e9b74259434e-root\") pod \"node-exporter-5hwc2\" (UID: \"ee650e41-f4c6-4ff7-bcf1-e9b74259434e\") " pod="openshift-monitoring/node-exporter-5hwc2"
Jul 21 17:27:20.313823 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.313734    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7fe58db6-f487-48b9-a4cc-16d5a4844298-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-rn2q5\" (UID: \"7fe58db6-f487-48b9-a4cc-16d5a4844298\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rn2q5"
Jul 21 17:27:20.313823 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.313772    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr9wn\" (UniqueName: \"kubernetes.io/projected/ee650e41-f4c6-4ff7-bcf1-e9b74259434e-kube-api-access-hr9wn\") pod \"node-exporter-5hwc2\" (UID: \"ee650e41-f4c6-4ff7-bcf1-e9b74259434e\") " pod="openshift-monitoring/node-exporter-5hwc2"
Jul 21 17:27:20.313823 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.313793    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7fe58db6-f487-48b9-a4cc-16d5a4844298-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-rn2q5\" (UID: \"7fe58db6-f487-48b9-a4cc-16d5a4844298\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rn2q5"
Jul 21 17:27:20.313823 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.313810    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ee650e41-f4c6-4ff7-bcf1-e9b74259434e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5hwc2\" (UID: \"ee650e41-f4c6-4ff7-bcf1-e9b74259434e\") " pod="openshift-monitoring/node-exporter-5hwc2"
Jul 21 17:27:20.313988 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.313845    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ee650e41-f4c6-4ff7-bcf1-e9b74259434e-node-exporter-textfile\") pod \"node-exporter-5hwc2\" (UID: \"ee650e41-f4c6-4ff7-bcf1-e9b74259434e\") " pod="openshift-monitoring/node-exporter-5hwc2"
Jul 21 17:27:20.313988 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.313866    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ee650e41-f4c6-4ff7-bcf1-e9b74259434e-node-exporter-tls\") pod \"node-exporter-5hwc2\" (UID: \"ee650e41-f4c6-4ff7-bcf1-e9b74259434e\") " pod="openshift-monitoring/node-exporter-5hwc2"
Jul 21 17:27:20.324450 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.324422    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g76xh"]
Jul 21 17:27:20.325580 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.325559    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-d8gnt"]
Jul 21 17:27:20.326721 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:27:20.326697    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa6ca187_d1f3_4d33_bd20_f70784dbbe02.slice/crio-7ce345b2b5e1d4bcd836d83a47cf64601b8d2af056081092ab7c108a1b3d4df0 WatchSource:0}: Error finding container 7ce345b2b5e1d4bcd836d83a47cf64601b8d2af056081092ab7c108a1b3d4df0: Status 404 returned error can't find the container with id 7ce345b2b5e1d4bcd836d83a47cf64601b8d2af056081092ab7c108a1b3d4df0
Jul 21 17:27:20.327458 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:27:20.327424    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb92707b0_e59d_4123_82e2_658e4361cb6e.slice/crio-c0d69fcbe6c2da4d6c059d27eabab4054d448e3a1847562ae4c6fd90f977c056 WatchSource:0}: Error finding container c0d69fcbe6c2da4d6c059d27eabab4054d448e3a1847562ae4c6fd90f977c056: Status 404 returned error can't find the container with id c0d69fcbe6c2da4d6c059d27eabab4054d448e3a1847562ae4c6fd90f977c056
Jul 21 17:27:20.414570 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.414544    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ee650e41-f4c6-4ff7-bcf1-e9b74259434e-node-exporter-tls\") pod \"node-exporter-5hwc2\" (UID: \"ee650e41-f4c6-4ff7-bcf1-e9b74259434e\") " pod="openshift-monitoring/node-exporter-5hwc2"
Jul 21 17:27:20.414697 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.414579    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7fe58db6-f487-48b9-a4cc-16d5a4844298-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-rn2q5\" (UID: \"7fe58db6-f487-48b9-a4cc-16d5a4844298\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rn2q5"
Jul 21 17:27:20.414697 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.414600    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjlj6\" (UniqueName: \"kubernetes.io/projected/7fe58db6-f487-48b9-a4cc-16d5a4844298-kube-api-access-zjlj6\") pod \"kube-state-metrics-56c5594674-rn2q5\" (UID: \"7fe58db6-f487-48b9-a4cc-16d5a4844298\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rn2q5"
Jul 21 17:27:20.414766 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.414741    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee650e41-f4c6-4ff7-bcf1-e9b74259434e-sys\") pod \"node-exporter-5hwc2\" (UID: \"ee650e41-f4c6-4ff7-bcf1-e9b74259434e\") " pod="openshift-monitoring/node-exporter-5hwc2"
Jul 21 17:27:20.414807 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.414793    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ee650e41-f4c6-4ff7-bcf1-e9b74259434e-node-exporter-wtmp\") pod \"node-exporter-5hwc2\" (UID: \"ee650e41-f4c6-4ff7-bcf1-e9b74259434e\") " pod="openshift-monitoring/node-exporter-5hwc2"
Jul 21 17:27:20.414843 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.414817    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7fe58db6-f487-48b9-a4cc-16d5a4844298-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-rn2q5\" (UID: \"7fe58db6-f487-48b9-a4cc-16d5a4844298\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rn2q5"
Jul 21 17:27:20.414843 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.414815    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee650e41-f4c6-4ff7-bcf1-e9b74259434e-sys\") pod \"node-exporter-5hwc2\" (UID: \"ee650e41-f4c6-4ff7-bcf1-e9b74259434e\") " pod="openshift-monitoring/node-exporter-5hwc2"
Jul 21 17:27:20.414929 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.414863    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ee650e41-f4c6-4ff7-bcf1-e9b74259434e-node-exporter-accelerators-collector-config\") pod \"node-exporter-5hwc2\" (UID: \"ee650e41-f4c6-4ff7-bcf1-e9b74259434e\") " pod="openshift-monitoring/node-exporter-5hwc2"
Jul 21 17:27:20.414929 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.414886    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7fe58db6-f487-48b9-a4cc-16d5a4844298-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-rn2q5\" (UID: \"7fe58db6-f487-48b9-a4cc-16d5a4844298\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rn2q5"
Jul 21 17:27:20.414929 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.414900    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee650e41-f4c6-4ff7-bcf1-e9b74259434e-metrics-client-ca\") pod \"node-exporter-5hwc2\" (UID: \"ee650e41-f4c6-4ff7-bcf1-e9b74259434e\") " pod="openshift-monitoring/node-exporter-5hwc2"
Jul 21 17:27:20.414929 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:27:20.414926    2563 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 21 17:27:20.415117 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.414932    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ee650e41-f4c6-4ff7-bcf1-e9b74259434e-root\") pod \"node-exporter-5hwc2\" (UID: \"ee650e41-f4c6-4ff7-bcf1-e9b74259434e\") " pod="openshift-monitoring/node-exporter-5hwc2"
Jul 21 17:27:20.415117 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.414955    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7fe58db6-f487-48b9-a4cc-16d5a4844298-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-rn2q5\" (UID: \"7fe58db6-f487-48b9-a4cc-16d5a4844298\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rn2q5"
Jul 21 17:27:20.415117 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:27:20.414981    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fe58db6-f487-48b9-a4cc-16d5a4844298-kube-state-metrics-tls podName:7fe58db6-f487-48b9-a4cc-16d5a4844298 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:20.914962117 +0000 UTC m=+68.677101698 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/7fe58db6-f487-48b9-a4cc-16d5a4844298-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-rn2q5" (UID: "7fe58db6-f487-48b9-a4cc-16d5a4844298") : secret "kube-state-metrics-tls" not found
Jul 21 17:27:20.415117 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.414981    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ee650e41-f4c6-4ff7-bcf1-e9b74259434e-node-exporter-wtmp\") pod \"node-exporter-5hwc2\" (UID: \"ee650e41-f4c6-4ff7-bcf1-e9b74259434e\") " pod="openshift-monitoring/node-exporter-5hwc2"
Jul 21 17:27:20.415117 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.415020    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr9wn\" (UniqueName: \"kubernetes.io/projected/ee650e41-f4c6-4ff7-bcf1-e9b74259434e-kube-api-access-hr9wn\") pod \"node-exporter-5hwc2\" (UID: \"ee650e41-f4c6-4ff7-bcf1-e9b74259434e\") " pod="openshift-monitoring/node-exporter-5hwc2"
Jul 21 17:27:20.415117 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.415047    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7fe58db6-f487-48b9-a4cc-16d5a4844298-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-rn2q5\" (UID: \"7fe58db6-f487-48b9-a4cc-16d5a4844298\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rn2q5"
Jul 21 17:27:20.415117 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.415074    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ee650e41-f4c6-4ff7-bcf1-e9b74259434e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5hwc2\" (UID: \"ee650e41-f4c6-4ff7-bcf1-e9b74259434e\") " pod="openshift-monitoring/node-exporter-5hwc2"
Jul 21 17:27:20.415488 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.415123    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ee650e41-f4c6-4ff7-bcf1-e9b74259434e-node-exporter-textfile\") pod \"node-exporter-5hwc2\" (UID: \"ee650e41-f4c6-4ff7-bcf1-e9b74259434e\") " pod="openshift-monitoring/node-exporter-5hwc2"
Jul 21 17:27:20.415488 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.415316    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7fe58db6-f487-48b9-a4cc-16d5a4844298-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-rn2q5\" (UID: \"7fe58db6-f487-48b9-a4cc-16d5a4844298\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rn2q5"
Jul 21 17:27:20.415488 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.415477    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ee650e41-f4c6-4ff7-bcf1-e9b74259434e-node-exporter-textfile\") pod \"node-exporter-5hwc2\" (UID: \"ee650e41-f4c6-4ff7-bcf1-e9b74259434e\") " pod="openshift-monitoring/node-exporter-5hwc2"
Jul 21 17:27:20.415661 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.415642    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee650e41-f4c6-4ff7-bcf1-e9b74259434e-metrics-client-ca\") pod \"node-exporter-5hwc2\" (UID: \"ee650e41-f4c6-4ff7-bcf1-e9b74259434e\") " pod="openshift-monitoring/node-exporter-5hwc2"
Jul 21 17:27:20.415734 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.415716    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ee650e41-f4c6-4ff7-bcf1-e9b74259434e-root\") pod \"node-exporter-5hwc2\" (UID: \"ee650e41-f4c6-4ff7-bcf1-e9b74259434e\") " pod="openshift-monitoring/node-exporter-5hwc2"
Jul 21 17:27:20.415839 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.415819    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7fe58db6-f487-48b9-a4cc-16d5a4844298-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-rn2q5\" (UID: \"7fe58db6-f487-48b9-a4cc-16d5a4844298\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rn2q5"
Jul 21 17:27:20.416161 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.415962    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7fe58db6-f487-48b9-a4cc-16d5a4844298-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-rn2q5\" (UID: \"7fe58db6-f487-48b9-a4cc-16d5a4844298\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rn2q5"
Jul 21 17:27:20.416161 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.415992    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ee650e41-f4c6-4ff7-bcf1-e9b74259434e-node-exporter-accelerators-collector-config\") pod \"node-exporter-5hwc2\" (UID: \"ee650e41-f4c6-4ff7-bcf1-e9b74259434e\") " pod="openshift-monitoring/node-exporter-5hwc2"
Jul 21 17:27:20.417377 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.417355    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ee650e41-f4c6-4ff7-bcf1-e9b74259434e-node-exporter-tls\") pod \"node-exporter-5hwc2\" (UID: \"ee650e41-f4c6-4ff7-bcf1-e9b74259434e\") " pod="openshift-monitoring/node-exporter-5hwc2"
Jul 21 17:27:20.417477 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.417393    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7fe58db6-f487-48b9-a4cc-16d5a4844298-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-rn2q5\" (UID: \"7fe58db6-f487-48b9-a4cc-16d5a4844298\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rn2q5"
Jul 21 17:27:20.417695 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.417679    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ee650e41-f4c6-4ff7-bcf1-e9b74259434e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5hwc2\" (UID: \"ee650e41-f4c6-4ff7-bcf1-e9b74259434e\") " pod="openshift-monitoring/node-exporter-5hwc2"
Jul 21 17:27:20.433640 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.433616    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr9wn\" (UniqueName: \"kubernetes.io/projected/ee650e41-f4c6-4ff7-bcf1-e9b74259434e-kube-api-access-hr9wn\") pod \"node-exporter-5hwc2\" (UID: \"ee650e41-f4c6-4ff7-bcf1-e9b74259434e\") " pod="openshift-monitoring/node-exporter-5hwc2"
Jul 21 17:27:20.436225 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.436206    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjlj6\" (UniqueName: \"kubernetes.io/projected/7fe58db6-f487-48b9-a4cc-16d5a4844298-kube-api-access-zjlj6\") pod \"kube-state-metrics-56c5594674-rn2q5\" (UID: \"7fe58db6-f487-48b9-a4cc-16d5a4844298\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rn2q5"
Jul 21 17:27:20.564385 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.564308    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5hwc2"
Jul 21 17:27:20.573818 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:27:20.573791    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee650e41_f4c6_4ff7_bcf1_e9b74259434e.slice/crio-282bcf56c8f27f333dd7f00b831be9a73928b9f846ecb72894fb74f04911798e WatchSource:0}: Error finding container 282bcf56c8f27f333dd7f00b831be9a73928b9f846ecb72894fb74f04911798e: Status 404 returned error can't find the container with id 282bcf56c8f27f333dd7f00b831be9a73928b9f846ecb72894fb74f04911798e
Jul 21 17:27:20.918864 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.918812    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7fe58db6-f487-48b9-a4cc-16d5a4844298-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-rn2q5\" (UID: \"7fe58db6-f487-48b9-a4cc-16d5a4844298\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rn2q5"
Jul 21 17:27:20.922278 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:20.922248    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7fe58db6-f487-48b9-a4cc-16d5a4844298-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-rn2q5\" (UID: \"7fe58db6-f487-48b9-a4cc-16d5a4844298\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rn2q5"
Jul 21 17:27:21.040867 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:21.040832    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dptd4" event={"ID":"15d6706f-5736-4ae6-b58d-677e456478c7","Type":"ContainerStarted","Data":"02cddb4c6ef0fa10d8493acf7ea4a4d4f0bf69b25f1ecb10805b7fd6ff46f252"}
Jul 21 17:27:21.042384 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:21.042355    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-g76xh" event={"ID":"fa6ca187-d1f3-4d33-bd20-f70784dbbe02","Type":"ContainerStarted","Data":"7ce345b2b5e1d4bcd836d83a47cf64601b8d2af056081092ab7c108a1b3d4df0"}
Jul 21 17:27:21.045301 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:21.045273    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-d8gnt" event={"ID":"b92707b0-e59d-4123-82e2-658e4361cb6e","Type":"ContainerStarted","Data":"c0d69fcbe6c2da4d6c059d27eabab4054d448e3a1847562ae4c6fd90f977c056"}
Jul 21 17:27:21.047696 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:21.047663    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5hwc2" event={"ID":"ee650e41-f4c6-4ff7-bcf1-e9b74259434e","Type":"ContainerStarted","Data":"282bcf56c8f27f333dd7f00b831be9a73928b9f846ecb72894fb74f04911798e"}
Jul 21 17:27:21.049053 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:21.048993    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sxmzk" event={"ID":"82279305-9475-429f-a495-143fc616fb34","Type":"ContainerStarted","Data":"2f0a949c122cb107bea048135e87cca22c0c9aef0339298d31dd82a70b2a1fe6"}
Jul 21 17:27:21.061585 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:21.061329    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dptd4" podStartSLOduration=1.91492704 podStartE2EDuration="15.061316082s" totalImagesPullingTime="13.146389042s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:27:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:21.059630159 +0000 UTC m=+68.821769764" watchObservedRunningTime="2026-07-21 17:27:21.061316082 +0000 UTC m=+68.823455685"
Jul 21 17:27:21.151176 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:21.150911    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-rn2q5"
Jul 21 17:27:21.344260 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:21.344012    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-rn2q5"]
Jul 21 17:27:21.347520 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:27:21.347486    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fe58db6_f487_48b9_a4cc_16d5a4844298.slice/crio-b6e1fdc3ac2a56b228df7f4f892a5e1af6d3a3cdbc3f96f6986ad459cd78a368 WatchSource:0}: Error finding container b6e1fdc3ac2a56b228df7f4f892a5e1af6d3a3cdbc3f96f6986ad459cd78a368: Status 404 returned error can't find the container with id b6e1fdc3ac2a56b228df7f4f892a5e1af6d3a3cdbc3f96f6986ad459cd78a368
Jul 21 17:27:22.056189 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:22.056151    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sxmzk" event={"ID":"82279305-9475-429f-a495-143fc616fb34","Type":"ContainerStarted","Data":"bdcd0729eaad33fddda93a3d52019ad51ae07a1cc153bbf095d1bd509a81ac08"}
Jul 21 17:27:22.056347 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:22.056220    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sxmzk" event={"ID":"82279305-9475-429f-a495-143fc616fb34","Type":"ContainerStarted","Data":"98b2a1196f7fce42c5894d5c103e8da9eaa7ef7934cc17151b0635d1f6e79fcd"}
Jul 21 17:27:22.058550 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:22.058447    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-rn2q5" event={"ID":"7fe58db6-f487-48b9-a4cc-16d5a4844298","Type":"ContainerStarted","Data":"b6e1fdc3ac2a56b228df7f4f892a5e1af6d3a3cdbc3f96f6986ad459cd78a368"}
Jul 21 17:27:22.062237 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:22.062200    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5hwc2" event={"ID":"ee650e41-f4c6-4ff7-bcf1-e9b74259434e","Type":"ContainerStarted","Data":"cf9aad4418236e5f579d82947681d4adc758ae55740056e5aa5bbaf92c993c39"}
Jul 21 17:27:22.075773 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:22.075526    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-sxmzk" podStartSLOduration=67.992496205 podStartE2EDuration="1m9.075507092s" totalImagesPullingTime="1.083010887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:26:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:22.073752879 +0000 UTC m=+69.835892507" watchObservedRunningTime="2026-07-21 17:27:22.075507092 +0000 UTC m=+69.837646697"
Jul 21 17:27:23.001670 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:23.001644    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-c9sl8"
Jul 21 17:27:23.068892 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:23.068861    2563 generic.go:356] "Generic (PLEG): container finished" podID="ee650e41-f4c6-4ff7-bcf1-e9b74259434e" containerID="cf9aad4418236e5f579d82947681d4adc758ae55740056e5aa5bbaf92c993c39" exitCode=0
Jul 21 17:27:23.069090 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:23.069031    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5hwc2" event={"ID":"ee650e41-f4c6-4ff7-bcf1-e9b74259434e","Type":"ContainerDied","Data":"cf9aad4418236e5f579d82947681d4adc758ae55740056e5aa5bbaf92c993c39"}
Jul 21 17:27:24.828389 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:24.828356    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-d2htd"]
Jul 21 17:27:24.830639 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:24.830617    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-d2htd"
Jul 21 17:27:24.833743 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:24.833383    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-z5bj2\""
Jul 21 17:27:24.833743 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:24.833595    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 21 17:27:24.841234 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:24.841066    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-d2htd"]
Jul 21 17:27:24.954334 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:24.954292    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/79e3c744-e62b-4582-b305-6a3586a7a37e-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-d2htd\" (UID: \"79e3c744-e62b-4582-b305-6a3586a7a37e\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-d2htd"
Jul 21 17:27:24.954509 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:24.954369    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k985\" (UniqueName: \"kubernetes.io/projected/79e3c744-e62b-4582-b305-6a3586a7a37e-kube-api-access-8k985\") pod \"monitoring-plugin-7b7b7bc8bf-d2htd\" (UID: \"79e3c744-e62b-4582-b305-6a3586a7a37e\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-d2htd"
Jul 21 17:27:25.054752 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:25.054713    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k985\" (UniqueName: \"kubernetes.io/projected/79e3c744-e62b-4582-b305-6a3586a7a37e-kube-api-access-8k985\") pod \"monitoring-plugin-7b7b7bc8bf-d2htd\" (UID: \"79e3c744-e62b-4582-b305-6a3586a7a37e\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-d2htd"
Jul 21 17:27:25.054947 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:25.054804    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/79e3c744-e62b-4582-b305-6a3586a7a37e-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-d2htd\" (UID: \"79e3c744-e62b-4582-b305-6a3586a7a37e\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-d2htd"
Jul 21 17:27:25.057530 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:25.057501    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/79e3c744-e62b-4582-b305-6a3586a7a37e-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-d2htd\" (UID: \"79e3c744-e62b-4582-b305-6a3586a7a37e\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-d2htd"
Jul 21 17:27:25.062715 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:25.062689    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k985\" (UniqueName: \"kubernetes.io/projected/79e3c744-e62b-4582-b305-6a3586a7a37e-kube-api-access-8k985\") pod \"monitoring-plugin-7b7b7bc8bf-d2htd\" (UID: \"79e3c744-e62b-4582-b305-6a3586a7a37e\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-d2htd"
Jul 21 17:27:25.143286 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:25.143255    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-d2htd"
Jul 21 17:27:25.643219 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:25.643164    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-d2htd"]
Jul 21 17:27:25.656424 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:27:25.656389    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e3c744_e62b_4582_b305_6a3586a7a37e.slice/crio-a2cb6bc3b9f8580d456b7348fbb73e4135c246847a3836b3b6181fd94345900e WatchSource:0}: Error finding container a2cb6bc3b9f8580d456b7348fbb73e4135c246847a3836b3b6181fd94345900e: Status 404 returned error can't find the container with id a2cb6bc3b9f8580d456b7348fbb73e4135c246847a3836b3b6181fd94345900e
Jul 21 17:27:26.081596 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.081525    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-d8gnt" event={"ID":"b92707b0-e59d-4123-82e2-658e4361cb6e","Type":"ContainerStarted","Data":"f6416d3344067e1896db7b0dfdd62e3304fe3229b8c35eb6cf9a4e0545e09e7c"}
Jul 21 17:27:26.082045 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.081605    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-d8gnt"
Jul 21 17:27:26.083750 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.083728    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-rn2q5" event={"ID":"7fe58db6-f487-48b9-a4cc-16d5a4844298","Type":"ContainerStarted","Data":"b4aa6c3c4cb778fa529a72c22191c9dc3bbfec458c84c7f96bf9feb62f262624"}
Jul 21 17:27:26.083874 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.083781    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-rn2q5" event={"ID":"7fe58db6-f487-48b9-a4cc-16d5a4844298","Type":"ContainerStarted","Data":"9ff7791fe36ae3b6f50c32626e4df64bba47e4c4e1796d3a3788a0f168195abe"}
Jul 21 17:27:26.083874 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.083810    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-rn2q5" event={"ID":"7fe58db6-f487-48b9-a4cc-16d5a4844298","Type":"ContainerStarted","Data":"576ed8bcb0df279db5c941f3f378656b2e47825e5220f03a368804e271a71aaf"}
Jul 21 17:27:26.085627 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.085606    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5hwc2" event={"ID":"ee650e41-f4c6-4ff7-bcf1-e9b74259434e","Type":"ContainerStarted","Data":"2163a10e54260e9c8b825b09e059ad63dea38a9bb222a5a535032ec77c072aa1"}
Jul 21 17:27:26.085748 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.085642    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5hwc2" event={"ID":"ee650e41-f4c6-4ff7-bcf1-e9b74259434e","Type":"ContainerStarted","Data":"9fb52ae494e7aca5d91b9b3e3fd3a8d2be7ee5add0beb2e2bfe1d4bd0d430e49"}
Jul 21 17:27:26.087080 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.087021    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-d2htd" event={"ID":"79e3c744-e62b-4582-b305-6a3586a7a37e","Type":"ContainerStarted","Data":"a2cb6bc3b9f8580d456b7348fbb73e4135c246847a3836b3b6181fd94345900e"}
Jul 21 17:27:26.089642 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.089622    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-g76xh" event={"ID":"fa6ca187-d1f3-4d33-bd20-f70784dbbe02","Type":"ContainerStarted","Data":"8a23226908c1a42d8607d23c62c5855f357afadfa06dc061a4103aa9dfb9d761"}
Jul 21 17:27:26.107821 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.107780    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-d8gnt" podStartSLOduration=67.94828006 podStartE2EDuration="1m13.107767397s" totalImagesPullingTime="5.159487337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:26:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:26.1055503 +0000 UTC m=+73.867689903" watchObservedRunningTime="2026-07-21 17:27:26.107767397 +0000 UTC m=+73.869907000"
Jul 21 17:27:26.178027 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.177973    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-rn2q5" podStartSLOduration=2.046940577 podStartE2EDuration="6.177955823s" totalImagesPullingTime="4.131015246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:27:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:26.140086353 +0000 UTC m=+73.902225958" watchObservedRunningTime="2026-07-21 17:27:26.177955823 +0000 UTC m=+73.940095430"
Jul 21 17:27:26.178703 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.178665    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5hwc2" podStartSLOduration=4.751114804 podStartE2EDuration="6.178654307s" totalImagesPullingTime="1.344987538s" totalInitContainerRuntime="82.551965ms" isStatefulPod=true podCreationTimestamp="2026-07-21 17:27:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:26.177908442 +0000 UTC m=+73.940048045" watchObservedRunningTime="2026-07-21 17:27:26.178654307 +0000 UTC m=+73.940793911"
Jul 21 17:27:26.564542 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.564493    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-g76xh" podStartSLOduration=68.392217251 podStartE2EDuration="1m13.564473292s" totalImagesPullingTime="5.172256041s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:26:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:26.198293159 +0000 UTC m=+73.960432754" watchObservedRunningTime="2026-07-21 17:27:26.564473292 +0000 UTC m=+74.326612895"
Jul 21 17:27:26.564872 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.564852    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 21 17:27:26.573866 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.573833    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.578285 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.577823    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 21 17:27:26.578285 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.577846    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 21 17:27:26.578285 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.577853    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 21 17:27:26.578285 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.577974    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 21 17:27:26.578285 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.577820    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 21 17:27:26.578285 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.578049    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 21 17:27:26.578285 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.578093    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 21 17:27:26.578285 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.578185    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 21 17:27:26.578770 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.578423    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 21 17:27:26.578770 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.578498    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 21 17:27:26.578770 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.578518    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-19uced80bb7e5\""
Jul 21 17:27:26.578770 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.578601    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 21 17:27:26.578770 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.578705    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lp86g\""
Jul 21 17:27:26.578770 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.578705    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 21 17:27:26.579070 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.578787    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 21 17:27:26.579070 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.578989    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 21 17:27:26.580797 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.580746    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 21 17:27:26.592624 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.592600    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 21 17:27:26.677781 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.677742    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.677970 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.677791    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.677970 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.677820    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcac0b1f-28d9-4bfe-a032-30dd161d324c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.677970 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.677845    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcac0b1f-28d9-4bfe-a032-30dd161d324c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.677970 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.677922    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fcac0b1f-28d9-4bfe-a032-30dd161d324c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.678223 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.677975    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-web-config\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.678223 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.678004    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.678223 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.678036    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8wr4\" (UniqueName: \"kubernetes.io/projected/fcac0b1f-28d9-4bfe-a032-30dd161d324c-kube-api-access-v8wr4\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.678223 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.678074    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.678223 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.678097    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fcac0b1f-28d9-4bfe-a032-30dd161d324c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.678223 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.678114    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcac0b1f-28d9-4bfe-a032-30dd161d324c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.678223 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.678169    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fcac0b1f-28d9-4bfe-a032-30dd161d324c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.678223 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.678191    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fcac0b1f-28d9-4bfe-a032-30dd161d324c-config-out\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.678585 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.678226    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcac0b1f-28d9-4bfe-a032-30dd161d324c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.678585 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.678260    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fcac0b1f-28d9-4bfe-a032-30dd161d324c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.678585 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.678290    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.678585 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.678316    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fcac0b1f-28d9-4bfe-a032-30dd161d324c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.678585 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.678349    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.678585 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.678409    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-config\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.678585 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.678433    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.780535 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.780494    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.780735 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.780556    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8wr4\" (UniqueName: \"kubernetes.io/projected/fcac0b1f-28d9-4bfe-a032-30dd161d324c-kube-api-access-v8wr4\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.780735 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.780587    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.780735 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.780607    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fcac0b1f-28d9-4bfe-a032-30dd161d324c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.780735 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.780630    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcac0b1f-28d9-4bfe-a032-30dd161d324c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.780735 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.780678    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fcac0b1f-28d9-4bfe-a032-30dd161d324c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.780735 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.780702    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fcac0b1f-28d9-4bfe-a032-30dd161d324c-config-out\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.780735 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.780725    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcac0b1f-28d9-4bfe-a032-30dd161d324c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.780735 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.780740    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fcac0b1f-28d9-4bfe-a032-30dd161d324c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.781117 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.780756    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.781117 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.780772    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fcac0b1f-28d9-4bfe-a032-30dd161d324c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.781117 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.780794    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.781117 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.780820    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-config\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.781117 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.780842    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.781117 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.780869    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.781117 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.780894    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.781117 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.780916    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcac0b1f-28d9-4bfe-a032-30dd161d324c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.781117 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.780937    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcac0b1f-28d9-4bfe-a032-30dd161d324c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.781117 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.780960    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fcac0b1f-28d9-4bfe-a032-30dd161d324c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.781117 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.780986    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-web-config\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.782104 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.782043    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fcac0b1f-28d9-4bfe-a032-30dd161d324c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.783260 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.782914    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcac0b1f-28d9-4bfe-a032-30dd161d324c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.784408 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.784381    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fcac0b1f-28d9-4bfe-a032-30dd161d324c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.784781 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.784704    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcac0b1f-28d9-4bfe-a032-30dd161d324c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.785287 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.785262    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.786046 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.786019    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcac0b1f-28d9-4bfe-a032-30dd161d324c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.786239 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.786216    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.786585 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.786530    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-web-config\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.787226 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.787175    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fcac0b1f-28d9-4bfe-a032-30dd161d324c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.787324 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.787264    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcac0b1f-28d9-4bfe-a032-30dd161d324c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.787324 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.787290    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fcac0b1f-28d9-4bfe-a032-30dd161d324c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.787652 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.787632    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.788498 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.788421    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.788954 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.788934    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.789683 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.789451    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-config\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.789683 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.789665    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.790356 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.790332    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fcac0b1f-28d9-4bfe-a032-30dd161d324c-config-out\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.790447 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.790370    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.791564 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.791543    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fcac0b1f-28d9-4bfe-a032-30dd161d324c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.795800 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.795776    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8wr4\" (UniqueName: \"kubernetes.io/projected/fcac0b1f-28d9-4bfe-a032-30dd161d324c-kube-api-access-v8wr4\") pod \"prometheus-k8s-0\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:26.887882 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:26.887851    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:27.226730 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:27.226634    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 21 17:27:27.228783 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:27:27.228759    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcac0b1f_28d9_4bfe_a032_30dd161d324c.slice/crio-affb4f1d63ba4191050355a04d3a280c4b415d9f31b194fc477fd3deaf1be68f WatchSource:0}: Error finding container affb4f1d63ba4191050355a04d3a280c4b415d9f31b194fc477fd3deaf1be68f: Status 404 returned error can't find the container with id affb4f1d63ba4191050355a04d3a280c4b415d9f31b194fc477fd3deaf1be68f
Jul 21 17:27:28.103703 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:28.103668    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-d2htd" event={"ID":"79e3c744-e62b-4582-b305-6a3586a7a37e","Type":"ContainerStarted","Data":"c3f4af10f524ab93c0bfc5a10f0a61ff755845876610b471c8a502bb25a0d0d6"}
Jul 21 17:27:28.103832 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:28.103765    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-d2htd"
Jul 21 17:27:28.105650 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:28.105505    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcac0b1f-28d9-4bfe-a032-30dd161d324c","Type":"ContainerStarted","Data":"8dd1ae2c0ce23878b18a919130277656472eef2266493939e19ebd8d47a122f6"}
Jul 21 17:27:28.105650 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:28.105579    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcac0b1f-28d9-4bfe-a032-30dd161d324c","Type":"ContainerStarted","Data":"affb4f1d63ba4191050355a04d3a280c4b415d9f31b194fc477fd3deaf1be68f"}
Jul 21 17:27:28.109739 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:28.109718    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-d2htd"
Jul 21 17:27:28.125005 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:28.124966    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-d2htd" podStartSLOduration=2.680400083 podStartE2EDuration="4.124952862s" totalImagesPullingTime="1.444552779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:27:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:28.124283411 +0000 UTC m=+75.886423016" watchObservedRunningTime="2026-07-21 17:27:28.124952862 +0000 UTC m=+75.887092487"
Jul 21 17:27:29.110430 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:29.110400    2563 generic.go:356] "Generic (PLEG): container finished" podID="fcac0b1f-28d9-4bfe-a032-30dd161d324c" containerID="8dd1ae2c0ce23878b18a919130277656472eef2266493939e19ebd8d47a122f6" exitCode=0
Jul 21 17:27:29.110810 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:29.110495    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcac0b1f-28d9-4bfe-a032-30dd161d324c","Type":"ContainerDied","Data":"8dd1ae2c0ce23878b18a919130277656472eef2266493939e19ebd8d47a122f6"}
Jul 21 17:27:33.130220 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:33.130061    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcac0b1f-28d9-4bfe-a032-30dd161d324c","Type":"ContainerStarted","Data":"08bf9fb215c401d9ed897af025766949d13883fae2c0229be2050a840ed7d3a4"}
Jul 21 17:27:33.130220 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:33.130178    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcac0b1f-28d9-4bfe-a032-30dd161d324c","Type":"ContainerStarted","Data":"a1ea37899d9c960c678e68a8aa7b07a00e5b40a2e5d65867317f64bbf2b4d3bb"}
Jul 21 17:27:34.138150 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:34.138106    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcac0b1f-28d9-4bfe-a032-30dd161d324c","Type":"ContainerStarted","Data":"ab9b51fdab5f94989443967b7860067977d2d5c6fd79cb6926246763fae1eeb9"}
Jul 21 17:27:34.138468 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:34.138189    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcac0b1f-28d9-4bfe-a032-30dd161d324c","Type":"ContainerStarted","Data":"028c4966e1251ff8378cfb189651c804204c465b25e5bb76e2c90bc2cffe5984"}
Jul 21 17:27:35.149668 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:35.149636    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcac0b1f-28d9-4bfe-a032-30dd161d324c","Type":"ContainerStarted","Data":"b38f7067898436adb73c6e3f07aafed054f86d7fe85078dac10e94d950217814"}
Jul 21 17:27:35.150076 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:35.149722    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcac0b1f-28d9-4bfe-a032-30dd161d324c","Type":"ContainerStarted","Data":"6745a4ad64f9209368cd244d25a3cd646abfb77ae47b34ef626cd13616d79a1c"}
Jul 21 17:27:35.186748 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:35.186688    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.604481453 podStartE2EDuration="9.186670658s" totalImagesPullingTime="5.559907255s" totalInitContainerRuntime="22.28195ms" isStatefulPod=false podCreationTimestamp="2026-07-21 17:27:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:35.185096951 +0000 UTC m=+82.947236595" watchObservedRunningTime="2026-07-21 17:27:35.186670658 +0000 UTC m=+82.948810286"
Jul 21 17:27:36.888375 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:36.888341    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:57.096040 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:27:57.096008    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-d8gnt"
Jul 21 17:28:26.888444 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:26.888405    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:26.907856 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:26.907830    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:27.343068 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:27.342990    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:44.880326 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:44.880287    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 21 17:28:44.880927 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:44.880656    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fcac0b1f-28d9-4bfe-a032-30dd161d324c" containerName="prometheus" containerID="cri-o://a1ea37899d9c960c678e68a8aa7b07a00e5b40a2e5d65867317f64bbf2b4d3bb" gracePeriod=600
Jul 21 17:28:44.880927 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:44.880710    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fcac0b1f-28d9-4bfe-a032-30dd161d324c" containerName="kube-rbac-proxy-thanos" containerID="cri-o://b38f7067898436adb73c6e3f07aafed054f86d7fe85078dac10e94d950217814" gracePeriod=600
Jul 21 17:28:44.880927 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:44.880764    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fcac0b1f-28d9-4bfe-a032-30dd161d324c" containerName="kube-rbac-proxy" containerID="cri-o://6745a4ad64f9209368cd244d25a3cd646abfb77ae47b34ef626cd13616d79a1c" gracePeriod=600
Jul 21 17:28:44.880927 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:44.880813    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fcac0b1f-28d9-4bfe-a032-30dd161d324c" containerName="thanos-sidecar" containerID="cri-o://028c4966e1251ff8378cfb189651c804204c465b25e5bb76e2c90bc2cffe5984" gracePeriod=600
Jul 21 17:28:44.880927 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:44.880856    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fcac0b1f-28d9-4bfe-a032-30dd161d324c" containerName="config-reloader" containerID="cri-o://08bf9fb215c401d9ed897af025766949d13883fae2c0229be2050a840ed7d3a4" gracePeriod=600
Jul 21 17:28:44.880927 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:44.880857    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fcac0b1f-28d9-4bfe-a032-30dd161d324c" containerName="kube-rbac-proxy-web" containerID="cri-o://ab9b51fdab5f94989443967b7860067977d2d5c6fd79cb6926246763fae1eeb9" gracePeriod=600
Jul 21 17:28:45.131550 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.131398    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.181124 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.181090    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-secret-metrics-client-certs\") pod \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") "
Jul 21 17:28:45.181124 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.181154    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fcac0b1f-28d9-4bfe-a032-30dd161d324c-prometheus-k8s-rulefiles-2\") pod \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") "
Jul 21 17:28:45.181426 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.181201    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-secret-grpc-tls\") pod \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") "
Jul 21 17:28:45.181850 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.181785    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcac0b1f-28d9-4bfe-a032-30dd161d324c-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "fcac0b1f-28d9-4bfe-a032-30dd161d324c" (UID: "fcac0b1f-28d9-4bfe-a032-30dd161d324c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:28:45.183046 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.182022    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fcac0b1f-28d9-4bfe-a032-30dd161d324c-config-out\") pod \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") "
Jul 21 17:28:45.183046 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.182070    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-config\") pod \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") "
Jul 21 17:28:45.183046 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.182103    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcac0b1f-28d9-4bfe-a032-30dd161d324c-configmap-kubelet-serving-ca-bundle\") pod \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") "
Jul 21 17:28:45.183046 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.182151    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") "
Jul 21 17:28:45.183046 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.182176    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fcac0b1f-28d9-4bfe-a032-30dd161d324c-prometheus-k8s-db\") pod \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") "
Jul 21 17:28:45.183046 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.182209    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8wr4\" (UniqueName: \"kubernetes.io/projected/fcac0b1f-28d9-4bfe-a032-30dd161d324c-kube-api-access-v8wr4\") pod \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") "
Jul 21 17:28:45.183046 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.182234    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fcac0b1f-28d9-4bfe-a032-30dd161d324c-prometheus-k8s-rulefiles-1\") pod \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") "
Jul 21 17:28:45.183046 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.182260    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") "
Jul 21 17:28:45.183046 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.182290    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcac0b1f-28d9-4bfe-a032-30dd161d324c-configmap-serving-certs-ca-bundle\") pod \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") "
Jul 21 17:28:45.183046 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.182315    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcac0b1f-28d9-4bfe-a032-30dd161d324c-configmap-metrics-client-ca\") pod \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") "
Jul 21 17:28:45.183046 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.182345    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcac0b1f-28d9-4bfe-a032-30dd161d324c-prometheus-trusted-ca-bundle\") pod \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") "
Jul 21 17:28:45.183046 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.182370    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-thanos-prometheus-http-client-file\") pod \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") "
Jul 21 17:28:45.183046 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.182399    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fcac0b1f-28d9-4bfe-a032-30dd161d324c-tls-assets\") pod \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") "
Jul 21 17:28:45.183046 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.182425    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-web-config\") pod \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") "
Jul 21 17:28:45.183046 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.182460    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fcac0b1f-28d9-4bfe-a032-30dd161d324c-prometheus-k8s-rulefiles-0\") pod \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") "
Jul 21 17:28:45.183046 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.182508    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-secret-kube-rbac-proxy\") pod \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") "
Jul 21 17:28:45.183046 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.182563    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-secret-prometheus-k8s-tls\") pod \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\" (UID: \"fcac0b1f-28d9-4bfe-a032-30dd161d324c\") "
Jul 21 17:28:45.183046 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.182769    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fcac0b1f-28d9-4bfe-a032-30dd161d324c-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:28:45.184339 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.184300    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcac0b1f-28d9-4bfe-a032-30dd161d324c-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "fcac0b1f-28d9-4bfe-a032-30dd161d324c" (UID: "fcac0b1f-28d9-4bfe-a032-30dd161d324c"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:28:45.184770 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.184504    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcac0b1f-28d9-4bfe-a032-30dd161d324c-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "fcac0b1f-28d9-4bfe-a032-30dd161d324c" (UID: "fcac0b1f-28d9-4bfe-a032-30dd161d324c"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:28:45.186348 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.185416    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcac0b1f-28d9-4bfe-a032-30dd161d324c-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "fcac0b1f-28d9-4bfe-a032-30dd161d324c" (UID: "fcac0b1f-28d9-4bfe-a032-30dd161d324c"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:28:45.186933 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.186743    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcac0b1f-28d9-4bfe-a032-30dd161d324c-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "fcac0b1f-28d9-4bfe-a032-30dd161d324c" (UID: "fcac0b1f-28d9-4bfe-a032-30dd161d324c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:28:45.187292 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.187181    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcac0b1f-28d9-4bfe-a032-30dd161d324c-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "fcac0b1f-28d9-4bfe-a032-30dd161d324c" (UID: "fcac0b1f-28d9-4bfe-a032-30dd161d324c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:28:45.187292 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.187194    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcac0b1f-28d9-4bfe-a032-30dd161d324c-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "fcac0b1f-28d9-4bfe-a032-30dd161d324c" (UID: "fcac0b1f-28d9-4bfe-a032-30dd161d324c"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:28:45.187292 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.187257    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcac0b1f-28d9-4bfe-a032-30dd161d324c-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "fcac0b1f-28d9-4bfe-a032-30dd161d324c" (UID: "fcac0b1f-28d9-4bfe-a032-30dd161d324c"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 17:28:45.187525 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.187488    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "fcac0b1f-28d9-4bfe-a032-30dd161d324c" (UID: "fcac0b1f-28d9-4bfe-a032-30dd161d324c"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:28:45.188119 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.188084    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "fcac0b1f-28d9-4bfe-a032-30dd161d324c" (UID: "fcac0b1f-28d9-4bfe-a032-30dd161d324c"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:28:45.188600 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.188566    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "fcac0b1f-28d9-4bfe-a032-30dd161d324c" (UID: "fcac0b1f-28d9-4bfe-a032-30dd161d324c"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:28:45.188991 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.188926    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "fcac0b1f-28d9-4bfe-a032-30dd161d324c" (UID: "fcac0b1f-28d9-4bfe-a032-30dd161d324c"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:28:45.188991 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.188971    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcac0b1f-28d9-4bfe-a032-30dd161d324c-kube-api-access-v8wr4" (OuterVolumeSpecName: "kube-api-access-v8wr4") pod "fcac0b1f-28d9-4bfe-a032-30dd161d324c" (UID: "fcac0b1f-28d9-4bfe-a032-30dd161d324c"). InnerVolumeSpecName "kube-api-access-v8wr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:28:45.189155 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.189018    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "fcac0b1f-28d9-4bfe-a032-30dd161d324c" (UID: "fcac0b1f-28d9-4bfe-a032-30dd161d324c"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:28:45.189327 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.189289    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "fcac0b1f-28d9-4bfe-a032-30dd161d324c" (UID: "fcac0b1f-28d9-4bfe-a032-30dd161d324c"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:28:45.189505 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.189479    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcac0b1f-28d9-4bfe-a032-30dd161d324c-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "fcac0b1f-28d9-4bfe-a032-30dd161d324c" (UID: "fcac0b1f-28d9-4bfe-a032-30dd161d324c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:28:45.190011 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.189977    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-config" (OuterVolumeSpecName: "config") pod "fcac0b1f-28d9-4bfe-a032-30dd161d324c" (UID: "fcac0b1f-28d9-4bfe-a032-30dd161d324c"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:28:45.190901 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.190875    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "fcac0b1f-28d9-4bfe-a032-30dd161d324c" (UID: "fcac0b1f-28d9-4bfe-a032-30dd161d324c"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:28:45.190990 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.190967    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcac0b1f-28d9-4bfe-a032-30dd161d324c-config-out" (OuterVolumeSpecName: "config-out") pod "fcac0b1f-28d9-4bfe-a032-30dd161d324c" (UID: "fcac0b1f-28d9-4bfe-a032-30dd161d324c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 17:28:45.200102 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.200073    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-web-config" (OuterVolumeSpecName: "web-config") pod "fcac0b1f-28d9-4bfe-a032-30dd161d324c" (UID: "fcac0b1f-28d9-4bfe-a032-30dd161d324c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:28:45.283622 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.283579    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-secret-prometheus-k8s-tls\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:28:45.283622 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.283611    2563 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-secret-metrics-client-certs\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:28:45.283622 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.283622    2563 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-secret-grpc-tls\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:28:45.283622 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.283631    2563 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fcac0b1f-28d9-4bfe-a032-30dd161d324c-config-out\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:28:45.283886 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.283639    2563 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-config\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:28:45.283886 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.283649    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcac0b1f-28d9-4bfe-a032-30dd161d324c-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:28:45.283886 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.283658    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:28:45.283886 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.283669    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fcac0b1f-28d9-4bfe-a032-30dd161d324c-prometheus-k8s-db\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:28:45.283886 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.283678    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8wr4\" (UniqueName: \"kubernetes.io/projected/fcac0b1f-28d9-4bfe-a032-30dd161d324c-kube-api-access-v8wr4\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:28:45.283886 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.283686    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fcac0b1f-28d9-4bfe-a032-30dd161d324c-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:28:45.283886 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.283694    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:28:45.283886 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.283702    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcac0b1f-28d9-4bfe-a032-30dd161d324c-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:28:45.283886 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.283710    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcac0b1f-28d9-4bfe-a032-30dd161d324c-configmap-metrics-client-ca\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:28:45.283886 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.283717    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcac0b1f-28d9-4bfe-a032-30dd161d324c-prometheus-trusted-ca-bundle\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:28:45.283886 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.283725    2563 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-thanos-prometheus-http-client-file\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:28:45.283886 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.283734    2563 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fcac0b1f-28d9-4bfe-a032-30dd161d324c-tls-assets\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:28:45.283886 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.283762    2563 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-web-config\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:28:45.283886 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.283770    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fcac0b1f-28d9-4bfe-a032-30dd161d324c-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:28:45.283886 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.283779    2563 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fcac0b1f-28d9-4bfe-a032-30dd161d324c-secret-kube-rbac-proxy\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:28:45.392863 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.392781    2563 generic.go:356] "Generic (PLEG): container finished" podID="fcac0b1f-28d9-4bfe-a032-30dd161d324c" containerID="b38f7067898436adb73c6e3f07aafed054f86d7fe85078dac10e94d950217814" exitCode=0
Jul 21 17:28:45.392863 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.392807    2563 generic.go:356] "Generic (PLEG): container finished" podID="fcac0b1f-28d9-4bfe-a032-30dd161d324c" containerID="6745a4ad64f9209368cd244d25a3cd646abfb77ae47b34ef626cd13616d79a1c" exitCode=0
Jul 21 17:28:45.392863 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.392814    2563 generic.go:356] "Generic (PLEG): container finished" podID="fcac0b1f-28d9-4bfe-a032-30dd161d324c" containerID="ab9b51fdab5f94989443967b7860067977d2d5c6fd79cb6926246763fae1eeb9" exitCode=0
Jul 21 17:28:45.392863 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.392819    2563 generic.go:356] "Generic (PLEG): container finished" podID="fcac0b1f-28d9-4bfe-a032-30dd161d324c" containerID="028c4966e1251ff8378cfb189651c804204c465b25e5bb76e2c90bc2cffe5984" exitCode=0
Jul 21 17:28:45.392863 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.392824    2563 generic.go:356] "Generic (PLEG): container finished" podID="fcac0b1f-28d9-4bfe-a032-30dd161d324c" containerID="08bf9fb215c401d9ed897af025766949d13883fae2c0229be2050a840ed7d3a4" exitCode=0
Jul 21 17:28:45.392863 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.392829    2563 generic.go:356] "Generic (PLEG): container finished" podID="fcac0b1f-28d9-4bfe-a032-30dd161d324c" containerID="a1ea37899d9c960c678e68a8aa7b07a00e5b40a2e5d65867317f64bbf2b4d3bb" exitCode=0
Jul 21 17:28:45.392863 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.392850    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcac0b1f-28d9-4bfe-a032-30dd161d324c","Type":"ContainerDied","Data":"b38f7067898436adb73c6e3f07aafed054f86d7fe85078dac10e94d950217814"}
Jul 21 17:28:45.392863 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.392863    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.393287 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.392933    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcac0b1f-28d9-4bfe-a032-30dd161d324c","Type":"ContainerDied","Data":"6745a4ad64f9209368cd244d25a3cd646abfb77ae47b34ef626cd13616d79a1c"}
Jul 21 17:28:45.393287 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.392961    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcac0b1f-28d9-4bfe-a032-30dd161d324c","Type":"ContainerDied","Data":"ab9b51fdab5f94989443967b7860067977d2d5c6fd79cb6926246763fae1eeb9"}
Jul 21 17:28:45.393287 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.392984    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcac0b1f-28d9-4bfe-a032-30dd161d324c","Type":"ContainerDied","Data":"028c4966e1251ff8378cfb189651c804204c465b25e5bb76e2c90bc2cffe5984"}
Jul 21 17:28:45.393287 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.393002    2563 scope.go:117] "RemoveContainer" containerID="b38f7067898436adb73c6e3f07aafed054f86d7fe85078dac10e94d950217814"
Jul 21 17:28:45.393287 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.393007    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcac0b1f-28d9-4bfe-a032-30dd161d324c","Type":"ContainerDied","Data":"08bf9fb215c401d9ed897af025766949d13883fae2c0229be2050a840ed7d3a4"}
Jul 21 17:28:45.393287 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.393034    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcac0b1f-28d9-4bfe-a032-30dd161d324c","Type":"ContainerDied","Data":"a1ea37899d9c960c678e68a8aa7b07a00e5b40a2e5d65867317f64bbf2b4d3bb"}
Jul 21 17:28:45.393287 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.393058    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcac0b1f-28d9-4bfe-a032-30dd161d324c","Type":"ContainerDied","Data":"affb4f1d63ba4191050355a04d3a280c4b415d9f31b194fc477fd3deaf1be68f"}
Jul 21 17:28:45.401670 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.401650    2563 scope.go:117] "RemoveContainer" containerID="6745a4ad64f9209368cd244d25a3cd646abfb77ae47b34ef626cd13616d79a1c"
Jul 21 17:28:45.408539 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.408522    2563 scope.go:117] "RemoveContainer" containerID="ab9b51fdab5f94989443967b7860067977d2d5c6fd79cb6926246763fae1eeb9"
Jul 21 17:28:45.415048 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.415030    2563 scope.go:117] "RemoveContainer" containerID="028c4966e1251ff8378cfb189651c804204c465b25e5bb76e2c90bc2cffe5984"
Jul 21 17:28:45.421558 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.421533    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 21 17:28:45.422767 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.422749    2563 scope.go:117] "RemoveContainer" containerID="08bf9fb215c401d9ed897af025766949d13883fae2c0229be2050a840ed7d3a4"
Jul 21 17:28:45.428773 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.428747    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 21 17:28:45.430104 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.430088    2563 scope.go:117] "RemoveContainer" containerID="a1ea37899d9c960c678e68a8aa7b07a00e5b40a2e5d65867317f64bbf2b4d3bb"
Jul 21 17:28:45.437049 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.437035    2563 scope.go:117] "RemoveContainer" containerID="8dd1ae2c0ce23878b18a919130277656472eef2266493939e19ebd8d47a122f6"
Jul 21 17:28:45.443205 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.443183    2563 scope.go:117] "RemoveContainer" containerID="b38f7067898436adb73c6e3f07aafed054f86d7fe85078dac10e94d950217814"
Jul 21 17:28:45.443670 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:28:45.443651    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b38f7067898436adb73c6e3f07aafed054f86d7fe85078dac10e94d950217814\": container with ID starting with b38f7067898436adb73c6e3f07aafed054f86d7fe85078dac10e94d950217814 not found: ID does not exist" containerID="b38f7067898436adb73c6e3f07aafed054f86d7fe85078dac10e94d950217814"
Jul 21 17:28:45.443731 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.443679    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b38f7067898436adb73c6e3f07aafed054f86d7fe85078dac10e94d950217814"} err="failed to get container status \"b38f7067898436adb73c6e3f07aafed054f86d7fe85078dac10e94d950217814\": rpc error: code = NotFound desc = could not find container \"b38f7067898436adb73c6e3f07aafed054f86d7fe85078dac10e94d950217814\": container with ID starting with b38f7067898436adb73c6e3f07aafed054f86d7fe85078dac10e94d950217814 not found: ID does not exist"
Jul 21 17:28:45.443731 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.443711    2563 scope.go:117] "RemoveContainer" containerID="6745a4ad64f9209368cd244d25a3cd646abfb77ae47b34ef626cd13616d79a1c"
Jul 21 17:28:45.443954 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:28:45.443938    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6745a4ad64f9209368cd244d25a3cd646abfb77ae47b34ef626cd13616d79a1c\": container with ID starting with 6745a4ad64f9209368cd244d25a3cd646abfb77ae47b34ef626cd13616d79a1c not found: ID does not exist" containerID="6745a4ad64f9209368cd244d25a3cd646abfb77ae47b34ef626cd13616d79a1c"
Jul 21 17:28:45.443994 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.443961    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6745a4ad64f9209368cd244d25a3cd646abfb77ae47b34ef626cd13616d79a1c"} err="failed to get container status \"6745a4ad64f9209368cd244d25a3cd646abfb77ae47b34ef626cd13616d79a1c\": rpc error: code = NotFound desc = could not find container \"6745a4ad64f9209368cd244d25a3cd646abfb77ae47b34ef626cd13616d79a1c\": container with ID starting with 6745a4ad64f9209368cd244d25a3cd646abfb77ae47b34ef626cd13616d79a1c not found: ID does not exist"
Jul 21 17:28:45.443994 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.443977    2563 scope.go:117] "RemoveContainer" containerID="ab9b51fdab5f94989443967b7860067977d2d5c6fd79cb6926246763fae1eeb9"
Jul 21 17:28:45.444235 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:28:45.444219    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab9b51fdab5f94989443967b7860067977d2d5c6fd79cb6926246763fae1eeb9\": container with ID starting with ab9b51fdab5f94989443967b7860067977d2d5c6fd79cb6926246763fae1eeb9 not found: ID does not exist" containerID="ab9b51fdab5f94989443967b7860067977d2d5c6fd79cb6926246763fae1eeb9"
Jul 21 17:28:45.444291 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.444240    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab9b51fdab5f94989443967b7860067977d2d5c6fd79cb6926246763fae1eeb9"} err="failed to get container status \"ab9b51fdab5f94989443967b7860067977d2d5c6fd79cb6926246763fae1eeb9\": rpc error: code = NotFound desc = could not find container \"ab9b51fdab5f94989443967b7860067977d2d5c6fd79cb6926246763fae1eeb9\": container with ID starting with ab9b51fdab5f94989443967b7860067977d2d5c6fd79cb6926246763fae1eeb9 not found: ID does not exist"
Jul 21 17:28:45.444291 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.444258    2563 scope.go:117] "RemoveContainer" containerID="028c4966e1251ff8378cfb189651c804204c465b25e5bb76e2c90bc2cffe5984"
Jul 21 17:28:45.444484 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:28:45.444468    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"028c4966e1251ff8378cfb189651c804204c465b25e5bb76e2c90bc2cffe5984\": container with ID starting with 028c4966e1251ff8378cfb189651c804204c465b25e5bb76e2c90bc2cffe5984 not found: ID does not exist" containerID="028c4966e1251ff8378cfb189651c804204c465b25e5bb76e2c90bc2cffe5984"
Jul 21 17:28:45.444523 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.444498    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"028c4966e1251ff8378cfb189651c804204c465b25e5bb76e2c90bc2cffe5984"} err="failed to get container status \"028c4966e1251ff8378cfb189651c804204c465b25e5bb76e2c90bc2cffe5984\": rpc error: code = NotFound desc = could not find container \"028c4966e1251ff8378cfb189651c804204c465b25e5bb76e2c90bc2cffe5984\": container with ID starting with 028c4966e1251ff8378cfb189651c804204c465b25e5bb76e2c90bc2cffe5984 not found: ID does not exist"
Jul 21 17:28:45.444523 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.444513    2563 scope.go:117] "RemoveContainer" containerID="08bf9fb215c401d9ed897af025766949d13883fae2c0229be2050a840ed7d3a4"
Jul 21 17:28:45.444736 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:28:45.444721    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08bf9fb215c401d9ed897af025766949d13883fae2c0229be2050a840ed7d3a4\": container with ID starting with 08bf9fb215c401d9ed897af025766949d13883fae2c0229be2050a840ed7d3a4 not found: ID does not exist" containerID="08bf9fb215c401d9ed897af025766949d13883fae2c0229be2050a840ed7d3a4"
Jul 21 17:28:45.444780 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.444738    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08bf9fb215c401d9ed897af025766949d13883fae2c0229be2050a840ed7d3a4"} err="failed to get container status \"08bf9fb215c401d9ed897af025766949d13883fae2c0229be2050a840ed7d3a4\": rpc error: code = NotFound desc = could not find container \"08bf9fb215c401d9ed897af025766949d13883fae2c0229be2050a840ed7d3a4\": container with ID starting with 08bf9fb215c401d9ed897af025766949d13883fae2c0229be2050a840ed7d3a4 not found: ID does not exist"
Jul 21 17:28:45.444780 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.444751    2563 scope.go:117] "RemoveContainer" containerID="a1ea37899d9c960c678e68a8aa7b07a00e5b40a2e5d65867317f64bbf2b4d3bb"
Jul 21 17:28:45.444917 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:28:45.444904    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1ea37899d9c960c678e68a8aa7b07a00e5b40a2e5d65867317f64bbf2b4d3bb\": container with ID starting with a1ea37899d9c960c678e68a8aa7b07a00e5b40a2e5d65867317f64bbf2b4d3bb not found: ID does not exist" containerID="a1ea37899d9c960c678e68a8aa7b07a00e5b40a2e5d65867317f64bbf2b4d3bb"
Jul 21 17:28:45.444954 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.444920    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1ea37899d9c960c678e68a8aa7b07a00e5b40a2e5d65867317f64bbf2b4d3bb"} err="failed to get container status \"a1ea37899d9c960c678e68a8aa7b07a00e5b40a2e5d65867317f64bbf2b4d3bb\": rpc error: code = NotFound desc = could not find container \"a1ea37899d9c960c678e68a8aa7b07a00e5b40a2e5d65867317f64bbf2b4d3bb\": container with ID starting with a1ea37899d9c960c678e68a8aa7b07a00e5b40a2e5d65867317f64bbf2b4d3bb not found: ID does not exist"
Jul 21 17:28:45.444954 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.444932    2563 scope.go:117] "RemoveContainer" containerID="8dd1ae2c0ce23878b18a919130277656472eef2266493939e19ebd8d47a122f6"
Jul 21 17:28:45.445157 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:28:45.445119    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dd1ae2c0ce23878b18a919130277656472eef2266493939e19ebd8d47a122f6\": container with ID starting with 8dd1ae2c0ce23878b18a919130277656472eef2266493939e19ebd8d47a122f6 not found: ID does not exist" containerID="8dd1ae2c0ce23878b18a919130277656472eef2266493939e19ebd8d47a122f6"
Jul 21 17:28:45.445214 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.445161    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dd1ae2c0ce23878b18a919130277656472eef2266493939e19ebd8d47a122f6"} err="failed to get container status \"8dd1ae2c0ce23878b18a919130277656472eef2266493939e19ebd8d47a122f6\": rpc error: code = NotFound desc = could not find container \"8dd1ae2c0ce23878b18a919130277656472eef2266493939e19ebd8d47a122f6\": container with ID starting with 8dd1ae2c0ce23878b18a919130277656472eef2266493939e19ebd8d47a122f6 not found: ID does not exist"
Jul 21 17:28:45.445214 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.445182    2563 scope.go:117] "RemoveContainer" containerID="b38f7067898436adb73c6e3f07aafed054f86d7fe85078dac10e94d950217814"
Jul 21 17:28:45.445409 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.445392    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b38f7067898436adb73c6e3f07aafed054f86d7fe85078dac10e94d950217814"} err="failed to get container status \"b38f7067898436adb73c6e3f07aafed054f86d7fe85078dac10e94d950217814\": rpc error: code = NotFound desc = could not find container \"b38f7067898436adb73c6e3f07aafed054f86d7fe85078dac10e94d950217814\": container with ID starting with b38f7067898436adb73c6e3f07aafed054f86d7fe85078dac10e94d950217814 not found: ID does not exist"
Jul 21 17:28:45.445456 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.445409    2563 scope.go:117] "RemoveContainer" containerID="6745a4ad64f9209368cd244d25a3cd646abfb77ae47b34ef626cd13616d79a1c"
Jul 21 17:28:45.445574 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.445559    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6745a4ad64f9209368cd244d25a3cd646abfb77ae47b34ef626cd13616d79a1c"} err="failed to get container status \"6745a4ad64f9209368cd244d25a3cd646abfb77ae47b34ef626cd13616d79a1c\": rpc error: code = NotFound desc = could not find container \"6745a4ad64f9209368cd244d25a3cd646abfb77ae47b34ef626cd13616d79a1c\": container with ID starting with 6745a4ad64f9209368cd244d25a3cd646abfb77ae47b34ef626cd13616d79a1c not found: ID does not exist"
Jul 21 17:28:45.445613 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.445573    2563 scope.go:117] "RemoveContainer" containerID="ab9b51fdab5f94989443967b7860067977d2d5c6fd79cb6926246763fae1eeb9"
Jul 21 17:28:45.445761 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.445744    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab9b51fdab5f94989443967b7860067977d2d5c6fd79cb6926246763fae1eeb9"} err="failed to get container status \"ab9b51fdab5f94989443967b7860067977d2d5c6fd79cb6926246763fae1eeb9\": rpc error: code = NotFound desc = could not find container \"ab9b51fdab5f94989443967b7860067977d2d5c6fd79cb6926246763fae1eeb9\": container with ID starting with ab9b51fdab5f94989443967b7860067977d2d5c6fd79cb6926246763fae1eeb9 not found: ID does not exist"
Jul 21 17:28:45.445808 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.445763    2563 scope.go:117] "RemoveContainer" containerID="028c4966e1251ff8378cfb189651c804204c465b25e5bb76e2c90bc2cffe5984"
Jul 21 17:28:45.445946 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.445928    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"028c4966e1251ff8378cfb189651c804204c465b25e5bb76e2c90bc2cffe5984"} err="failed to get container status \"028c4966e1251ff8378cfb189651c804204c465b25e5bb76e2c90bc2cffe5984\": rpc error: code = NotFound desc = could not find container \"028c4966e1251ff8378cfb189651c804204c465b25e5bb76e2c90bc2cffe5984\": container with ID starting with 028c4966e1251ff8378cfb189651c804204c465b25e5bb76e2c90bc2cffe5984 not found: ID does not exist"
Jul 21 17:28:45.446012 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.445946    2563 scope.go:117] "RemoveContainer" containerID="08bf9fb215c401d9ed897af025766949d13883fae2c0229be2050a840ed7d3a4"
Jul 21 17:28:45.446231 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.446215    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08bf9fb215c401d9ed897af025766949d13883fae2c0229be2050a840ed7d3a4"} err="failed to get container status \"08bf9fb215c401d9ed897af025766949d13883fae2c0229be2050a840ed7d3a4\": rpc error: code = NotFound desc = could not find container \"08bf9fb215c401d9ed897af025766949d13883fae2c0229be2050a840ed7d3a4\": container with ID starting with 08bf9fb215c401d9ed897af025766949d13883fae2c0229be2050a840ed7d3a4 not found: ID does not exist"
Jul 21 17:28:45.446275 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.446231    2563 scope.go:117] "RemoveContainer" containerID="a1ea37899d9c960c678e68a8aa7b07a00e5b40a2e5d65867317f64bbf2b4d3bb"
Jul 21 17:28:45.446444 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.446427    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1ea37899d9c960c678e68a8aa7b07a00e5b40a2e5d65867317f64bbf2b4d3bb"} err="failed to get container status \"a1ea37899d9c960c678e68a8aa7b07a00e5b40a2e5d65867317f64bbf2b4d3bb\": rpc error: code = NotFound desc = could not find container \"a1ea37899d9c960c678e68a8aa7b07a00e5b40a2e5d65867317f64bbf2b4d3bb\": container with ID starting with a1ea37899d9c960c678e68a8aa7b07a00e5b40a2e5d65867317f64bbf2b4d3bb not found: ID does not exist"
Jul 21 17:28:45.446514 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.446446    2563 scope.go:117] "RemoveContainer" containerID="8dd1ae2c0ce23878b18a919130277656472eef2266493939e19ebd8d47a122f6"
Jul 21 17:28:45.446679 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.446663    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dd1ae2c0ce23878b18a919130277656472eef2266493939e19ebd8d47a122f6"} err="failed to get container status \"8dd1ae2c0ce23878b18a919130277656472eef2266493939e19ebd8d47a122f6\": rpc error: code = NotFound desc = could not find container \"8dd1ae2c0ce23878b18a919130277656472eef2266493939e19ebd8d47a122f6\": container with ID starting with 8dd1ae2c0ce23878b18a919130277656472eef2266493939e19ebd8d47a122f6 not found: ID does not exist"
Jul 21 17:28:45.446723 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.446679    2563 scope.go:117] "RemoveContainer" containerID="b38f7067898436adb73c6e3f07aafed054f86d7fe85078dac10e94d950217814"
Jul 21 17:28:45.446877 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.446858    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b38f7067898436adb73c6e3f07aafed054f86d7fe85078dac10e94d950217814"} err="failed to get container status \"b38f7067898436adb73c6e3f07aafed054f86d7fe85078dac10e94d950217814\": rpc error: code = NotFound desc = could not find container \"b38f7067898436adb73c6e3f07aafed054f86d7fe85078dac10e94d950217814\": container with ID starting with b38f7067898436adb73c6e3f07aafed054f86d7fe85078dac10e94d950217814 not found: ID does not exist"
Jul 21 17:28:45.446942 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.446878    2563 scope.go:117] "RemoveContainer" containerID="6745a4ad64f9209368cd244d25a3cd646abfb77ae47b34ef626cd13616d79a1c"
Jul 21 17:28:45.447069 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.447053    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6745a4ad64f9209368cd244d25a3cd646abfb77ae47b34ef626cd13616d79a1c"} err="failed to get container status \"6745a4ad64f9209368cd244d25a3cd646abfb77ae47b34ef626cd13616d79a1c\": rpc error: code = NotFound desc = could not find container \"6745a4ad64f9209368cd244d25a3cd646abfb77ae47b34ef626cd13616d79a1c\": container with ID starting with 6745a4ad64f9209368cd244d25a3cd646abfb77ae47b34ef626cd13616d79a1c not found: ID does not exist"
Jul 21 17:28:45.447108 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.447068    2563 scope.go:117] "RemoveContainer" containerID="ab9b51fdab5f94989443967b7860067977d2d5c6fd79cb6926246763fae1eeb9"
Jul 21 17:28:45.447305 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.447283    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab9b51fdab5f94989443967b7860067977d2d5c6fd79cb6926246763fae1eeb9"} err="failed to get container status \"ab9b51fdab5f94989443967b7860067977d2d5c6fd79cb6926246763fae1eeb9\": rpc error: code = NotFound desc = could not find container \"ab9b51fdab5f94989443967b7860067977d2d5c6fd79cb6926246763fae1eeb9\": container with ID starting with ab9b51fdab5f94989443967b7860067977d2d5c6fd79cb6926246763fae1eeb9 not found: ID does not exist"
Jul 21 17:28:45.447355 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.447307    2563 scope.go:117] "RemoveContainer" containerID="028c4966e1251ff8378cfb189651c804204c465b25e5bb76e2c90bc2cffe5984"
Jul 21 17:28:45.447533 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.447508    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"028c4966e1251ff8378cfb189651c804204c465b25e5bb76e2c90bc2cffe5984"} err="failed to get container status \"028c4966e1251ff8378cfb189651c804204c465b25e5bb76e2c90bc2cffe5984\": rpc error: code = NotFound desc = could not find container \"028c4966e1251ff8378cfb189651c804204c465b25e5bb76e2c90bc2cffe5984\": container with ID starting with 028c4966e1251ff8378cfb189651c804204c465b25e5bb76e2c90bc2cffe5984 not found: ID does not exist"
Jul 21 17:28:45.447574 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.447533    2563 scope.go:117] "RemoveContainer" containerID="08bf9fb215c401d9ed897af025766949d13883fae2c0229be2050a840ed7d3a4"
Jul 21 17:28:45.447748 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.447731    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08bf9fb215c401d9ed897af025766949d13883fae2c0229be2050a840ed7d3a4"} err="failed to get container status \"08bf9fb215c401d9ed897af025766949d13883fae2c0229be2050a840ed7d3a4\": rpc error: code = NotFound desc = could not find container \"08bf9fb215c401d9ed897af025766949d13883fae2c0229be2050a840ed7d3a4\": container with ID starting with 08bf9fb215c401d9ed897af025766949d13883fae2c0229be2050a840ed7d3a4 not found: ID does not exist"
Jul 21 17:28:45.447789 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.447748    2563 scope.go:117] "RemoveContainer" containerID="a1ea37899d9c960c678e68a8aa7b07a00e5b40a2e5d65867317f64bbf2b4d3bb"
Jul 21 17:28:45.447937 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.447921    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1ea37899d9c960c678e68a8aa7b07a00e5b40a2e5d65867317f64bbf2b4d3bb"} err="failed to get container status \"a1ea37899d9c960c678e68a8aa7b07a00e5b40a2e5d65867317f64bbf2b4d3bb\": rpc error: code = NotFound desc = could not find container \"a1ea37899d9c960c678e68a8aa7b07a00e5b40a2e5d65867317f64bbf2b4d3bb\": container with ID starting with a1ea37899d9c960c678e68a8aa7b07a00e5b40a2e5d65867317f64bbf2b4d3bb not found: ID does not exist"
Jul 21 17:28:45.447974 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.447938    2563 scope.go:117] "RemoveContainer" containerID="8dd1ae2c0ce23878b18a919130277656472eef2266493939e19ebd8d47a122f6"
Jul 21 17:28:45.448118 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.448103    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dd1ae2c0ce23878b18a919130277656472eef2266493939e19ebd8d47a122f6"} err="failed to get container status \"8dd1ae2c0ce23878b18a919130277656472eef2266493939e19ebd8d47a122f6\": rpc error: code = NotFound desc = could not find container \"8dd1ae2c0ce23878b18a919130277656472eef2266493939e19ebd8d47a122f6\": container with ID starting with 8dd1ae2c0ce23878b18a919130277656472eef2266493939e19ebd8d47a122f6 not found: ID does not exist"
Jul 21 17:28:45.448182 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.448118    2563 scope.go:117] "RemoveContainer" containerID="b38f7067898436adb73c6e3f07aafed054f86d7fe85078dac10e94d950217814"
Jul 21 17:28:45.448313 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.448297    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b38f7067898436adb73c6e3f07aafed054f86d7fe85078dac10e94d950217814"} err="failed to get container status \"b38f7067898436adb73c6e3f07aafed054f86d7fe85078dac10e94d950217814\": rpc error: code = NotFound desc = could not find container \"b38f7067898436adb73c6e3f07aafed054f86d7fe85078dac10e94d950217814\": container with ID starting with b38f7067898436adb73c6e3f07aafed054f86d7fe85078dac10e94d950217814 not found: ID does not exist"
Jul 21 17:28:45.448353 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.448314    2563 scope.go:117] "RemoveContainer" containerID="6745a4ad64f9209368cd244d25a3cd646abfb77ae47b34ef626cd13616d79a1c"
Jul 21 17:28:45.448490 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.448474    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6745a4ad64f9209368cd244d25a3cd646abfb77ae47b34ef626cd13616d79a1c"} err="failed to get container status \"6745a4ad64f9209368cd244d25a3cd646abfb77ae47b34ef626cd13616d79a1c\": rpc error: code = NotFound desc = could not find container \"6745a4ad64f9209368cd244d25a3cd646abfb77ae47b34ef626cd13616d79a1c\": container with ID starting with 6745a4ad64f9209368cd244d25a3cd646abfb77ae47b34ef626cd13616d79a1c not found: ID does not exist"
Jul 21 17:28:45.448527 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.448489    2563 scope.go:117] "RemoveContainer" containerID="ab9b51fdab5f94989443967b7860067977d2d5c6fd79cb6926246763fae1eeb9"
Jul 21 17:28:45.448665 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.448649    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab9b51fdab5f94989443967b7860067977d2d5c6fd79cb6926246763fae1eeb9"} err="failed to get container status \"ab9b51fdab5f94989443967b7860067977d2d5c6fd79cb6926246763fae1eeb9\": rpc error: code = NotFound desc = could not find container \"ab9b51fdab5f94989443967b7860067977d2d5c6fd79cb6926246763fae1eeb9\": container with ID starting with ab9b51fdab5f94989443967b7860067977d2d5c6fd79cb6926246763fae1eeb9 not found: ID does not exist"
Jul 21 17:28:45.448736 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.448665    2563 scope.go:117] "RemoveContainer" containerID="028c4966e1251ff8378cfb189651c804204c465b25e5bb76e2c90bc2cffe5984"
Jul 21 17:28:45.448887 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.448871    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"028c4966e1251ff8378cfb189651c804204c465b25e5bb76e2c90bc2cffe5984"} err="failed to get container status \"028c4966e1251ff8378cfb189651c804204c465b25e5bb76e2c90bc2cffe5984\": rpc error: code = NotFound desc = could not find container \"028c4966e1251ff8378cfb189651c804204c465b25e5bb76e2c90bc2cffe5984\": container with ID starting with 028c4966e1251ff8378cfb189651c804204c465b25e5bb76e2c90bc2cffe5984 not found: ID does not exist"
Jul 21 17:28:45.448939 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.448888    2563 scope.go:117] "RemoveContainer" containerID="08bf9fb215c401d9ed897af025766949d13883fae2c0229be2050a840ed7d3a4"
Jul 21 17:28:45.449071 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.449052    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08bf9fb215c401d9ed897af025766949d13883fae2c0229be2050a840ed7d3a4"} err="failed to get container status \"08bf9fb215c401d9ed897af025766949d13883fae2c0229be2050a840ed7d3a4\": rpc error: code = NotFound desc = could not find container \"08bf9fb215c401d9ed897af025766949d13883fae2c0229be2050a840ed7d3a4\": container with ID starting with 08bf9fb215c401d9ed897af025766949d13883fae2c0229be2050a840ed7d3a4 not found: ID does not exist"
Jul 21 17:28:45.449126 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.449072    2563 scope.go:117] "RemoveContainer" containerID="a1ea37899d9c960c678e68a8aa7b07a00e5b40a2e5d65867317f64bbf2b4d3bb"
Jul 21 17:28:45.449278 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.449263    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1ea37899d9c960c678e68a8aa7b07a00e5b40a2e5d65867317f64bbf2b4d3bb"} err="failed to get container status \"a1ea37899d9c960c678e68a8aa7b07a00e5b40a2e5d65867317f64bbf2b4d3bb\": rpc error: code = NotFound desc = could not find container \"a1ea37899d9c960c678e68a8aa7b07a00e5b40a2e5d65867317f64bbf2b4d3bb\": container with ID starting with a1ea37899d9c960c678e68a8aa7b07a00e5b40a2e5d65867317f64bbf2b4d3bb not found: ID does not exist"
Jul 21 17:28:45.449320 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.449279    2563 scope.go:117] "RemoveContainer" containerID="8dd1ae2c0ce23878b18a919130277656472eef2266493939e19ebd8d47a122f6"
Jul 21 17:28:45.449462 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.449446    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dd1ae2c0ce23878b18a919130277656472eef2266493939e19ebd8d47a122f6"} err="failed to get container status \"8dd1ae2c0ce23878b18a919130277656472eef2266493939e19ebd8d47a122f6\": rpc error: code = NotFound desc = could not find container \"8dd1ae2c0ce23878b18a919130277656472eef2266493939e19ebd8d47a122f6\": container with ID starting with 8dd1ae2c0ce23878b18a919130277656472eef2266493939e19ebd8d47a122f6 not found: ID does not exist"
Jul 21 17:28:45.449502 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.449463    2563 scope.go:117] "RemoveContainer" containerID="b38f7067898436adb73c6e3f07aafed054f86d7fe85078dac10e94d950217814"
Jul 21 17:28:45.449665 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.449650    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b38f7067898436adb73c6e3f07aafed054f86d7fe85078dac10e94d950217814"} err="failed to get container status \"b38f7067898436adb73c6e3f07aafed054f86d7fe85078dac10e94d950217814\": rpc error: code = NotFound desc = could not find container \"b38f7067898436adb73c6e3f07aafed054f86d7fe85078dac10e94d950217814\": container with ID starting with b38f7067898436adb73c6e3f07aafed054f86d7fe85078dac10e94d950217814 not found: ID does not exist"
Jul 21 17:28:45.449665 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.449665    2563 scope.go:117] "RemoveContainer" containerID="6745a4ad64f9209368cd244d25a3cd646abfb77ae47b34ef626cd13616d79a1c"
Jul 21 17:28:45.449823 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.449805    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6745a4ad64f9209368cd244d25a3cd646abfb77ae47b34ef626cd13616d79a1c"} err="failed to get container status \"6745a4ad64f9209368cd244d25a3cd646abfb77ae47b34ef626cd13616d79a1c\": rpc error: code = NotFound desc = could not find container \"6745a4ad64f9209368cd244d25a3cd646abfb77ae47b34ef626cd13616d79a1c\": container with ID starting with 6745a4ad64f9209368cd244d25a3cd646abfb77ae47b34ef626cd13616d79a1c not found: ID does not exist"
Jul 21 17:28:45.449863 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.449824    2563 scope.go:117] "RemoveContainer" containerID="ab9b51fdab5f94989443967b7860067977d2d5c6fd79cb6926246763fae1eeb9"
Jul 21 17:28:45.449987 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.449972    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab9b51fdab5f94989443967b7860067977d2d5c6fd79cb6926246763fae1eeb9"} err="failed to get container status \"ab9b51fdab5f94989443967b7860067977d2d5c6fd79cb6926246763fae1eeb9\": rpc error: code = NotFound desc = could not find container \"ab9b51fdab5f94989443967b7860067977d2d5c6fd79cb6926246763fae1eeb9\": container with ID starting with ab9b51fdab5f94989443967b7860067977d2d5c6fd79cb6926246763fae1eeb9 not found: ID does not exist"
Jul 21 17:28:45.450024 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.449987    2563 scope.go:117] "RemoveContainer" containerID="028c4966e1251ff8378cfb189651c804204c465b25e5bb76e2c90bc2cffe5984"
Jul 21 17:28:45.450219 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.450196    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"028c4966e1251ff8378cfb189651c804204c465b25e5bb76e2c90bc2cffe5984"} err="failed to get container status \"028c4966e1251ff8378cfb189651c804204c465b25e5bb76e2c90bc2cffe5984\": rpc error: code = NotFound desc = could not find container \"028c4966e1251ff8378cfb189651c804204c465b25e5bb76e2c90bc2cffe5984\": container with ID starting with 028c4966e1251ff8378cfb189651c804204c465b25e5bb76e2c90bc2cffe5984 not found: ID does not exist"
Jul 21 17:28:45.450219 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.450212    2563 scope.go:117] "RemoveContainer" containerID="08bf9fb215c401d9ed897af025766949d13883fae2c0229be2050a840ed7d3a4"
Jul 21 17:28:45.450437 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.450414    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08bf9fb215c401d9ed897af025766949d13883fae2c0229be2050a840ed7d3a4"} err="failed to get container status \"08bf9fb215c401d9ed897af025766949d13883fae2c0229be2050a840ed7d3a4\": rpc error: code = NotFound desc = could not find container \"08bf9fb215c401d9ed897af025766949d13883fae2c0229be2050a840ed7d3a4\": container with ID starting with 08bf9fb215c401d9ed897af025766949d13883fae2c0229be2050a840ed7d3a4 not found: ID does not exist"
Jul 21 17:28:45.450484 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.450439    2563 scope.go:117] "RemoveContainer" containerID="a1ea37899d9c960c678e68a8aa7b07a00e5b40a2e5d65867317f64bbf2b4d3bb"
Jul 21 17:28:45.450665 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.450648    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1ea37899d9c960c678e68a8aa7b07a00e5b40a2e5d65867317f64bbf2b4d3bb"} err="failed to get container status \"a1ea37899d9c960c678e68a8aa7b07a00e5b40a2e5d65867317f64bbf2b4d3bb\": rpc error: code = NotFound desc = could not find container \"a1ea37899d9c960c678e68a8aa7b07a00e5b40a2e5d65867317f64bbf2b4d3bb\": container with ID starting with a1ea37899d9c960c678e68a8aa7b07a00e5b40a2e5d65867317f64bbf2b4d3bb not found: ID does not exist"
Jul 21 17:28:45.450710 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.450665    2563 scope.go:117] "RemoveContainer" containerID="8dd1ae2c0ce23878b18a919130277656472eef2266493939e19ebd8d47a122f6"
Jul 21 17:28:45.450831 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.450813    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dd1ae2c0ce23878b18a919130277656472eef2266493939e19ebd8d47a122f6"} err="failed to get container status \"8dd1ae2c0ce23878b18a919130277656472eef2266493939e19ebd8d47a122f6\": rpc error: code = NotFound desc = could not find container \"8dd1ae2c0ce23878b18a919130277656472eef2266493939e19ebd8d47a122f6\": container with ID starting with 8dd1ae2c0ce23878b18a919130277656472eef2266493939e19ebd8d47a122f6 not found: ID does not exist"
Jul 21 17:28:45.450876 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.450832    2563 scope.go:117] "RemoveContainer" containerID="b38f7067898436adb73c6e3f07aafed054f86d7fe85078dac10e94d950217814"
Jul 21 17:28:45.451035 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.451020    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b38f7067898436adb73c6e3f07aafed054f86d7fe85078dac10e94d950217814"} err="failed to get container status \"b38f7067898436adb73c6e3f07aafed054f86d7fe85078dac10e94d950217814\": rpc error: code = NotFound desc = could not find container \"b38f7067898436adb73c6e3f07aafed054f86d7fe85078dac10e94d950217814\": container with ID starting with b38f7067898436adb73c6e3f07aafed054f86d7fe85078dac10e94d950217814 not found: ID does not exist"
Jul 21 17:28:45.451085 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.451035    2563 scope.go:117] "RemoveContainer" containerID="6745a4ad64f9209368cd244d25a3cd646abfb77ae47b34ef626cd13616d79a1c"
Jul 21 17:28:45.451292 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.451276    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6745a4ad64f9209368cd244d25a3cd646abfb77ae47b34ef626cd13616d79a1c"} err="failed to get container status \"6745a4ad64f9209368cd244d25a3cd646abfb77ae47b34ef626cd13616d79a1c\": rpc error: code = NotFound desc = could not find container \"6745a4ad64f9209368cd244d25a3cd646abfb77ae47b34ef626cd13616d79a1c\": container with ID starting with 6745a4ad64f9209368cd244d25a3cd646abfb77ae47b34ef626cd13616d79a1c not found: ID does not exist"
Jul 21 17:28:45.451346 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.451293    2563 scope.go:117] "RemoveContainer" containerID="ab9b51fdab5f94989443967b7860067977d2d5c6fd79cb6926246763fae1eeb9"
Jul 21 17:28:45.451502 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.451488    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab9b51fdab5f94989443967b7860067977d2d5c6fd79cb6926246763fae1eeb9"} err="failed to get container status \"ab9b51fdab5f94989443967b7860067977d2d5c6fd79cb6926246763fae1eeb9\": rpc error: code = NotFound desc = could not find container \"ab9b51fdab5f94989443967b7860067977d2d5c6fd79cb6926246763fae1eeb9\": container with ID starting with ab9b51fdab5f94989443967b7860067977d2d5c6fd79cb6926246763fae1eeb9 not found: ID does not exist"
Jul 21 17:28:45.451561 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.451503    2563 scope.go:117] "RemoveContainer" containerID="028c4966e1251ff8378cfb189651c804204c465b25e5bb76e2c90bc2cffe5984"
Jul 21 17:28:45.451715 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.451696    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"028c4966e1251ff8378cfb189651c804204c465b25e5bb76e2c90bc2cffe5984"} err="failed to get container status \"028c4966e1251ff8378cfb189651c804204c465b25e5bb76e2c90bc2cffe5984\": rpc error: code = NotFound desc = could not find container \"028c4966e1251ff8378cfb189651c804204c465b25e5bb76e2c90bc2cffe5984\": container with ID starting with 028c4966e1251ff8378cfb189651c804204c465b25e5bb76e2c90bc2cffe5984 not found: ID does not exist"
Jul 21 17:28:45.451786 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.451717    2563 scope.go:117] "RemoveContainer" containerID="08bf9fb215c401d9ed897af025766949d13883fae2c0229be2050a840ed7d3a4"
Jul 21 17:28:45.451934 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.451919    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08bf9fb215c401d9ed897af025766949d13883fae2c0229be2050a840ed7d3a4"} err="failed to get container status \"08bf9fb215c401d9ed897af025766949d13883fae2c0229be2050a840ed7d3a4\": rpc error: code = NotFound desc = could not find container \"08bf9fb215c401d9ed897af025766949d13883fae2c0229be2050a840ed7d3a4\": container with ID starting with 08bf9fb215c401d9ed897af025766949d13883fae2c0229be2050a840ed7d3a4 not found: ID does not exist"
Jul 21 17:28:45.451979 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.451935    2563 scope.go:117] "RemoveContainer" containerID="a1ea37899d9c960c678e68a8aa7b07a00e5b40a2e5d65867317f64bbf2b4d3bb"
Jul 21 17:28:45.452194 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.452176    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1ea37899d9c960c678e68a8aa7b07a00e5b40a2e5d65867317f64bbf2b4d3bb"} err="failed to get container status \"a1ea37899d9c960c678e68a8aa7b07a00e5b40a2e5d65867317f64bbf2b4d3bb\": rpc error: code = NotFound desc = could not find container \"a1ea37899d9c960c678e68a8aa7b07a00e5b40a2e5d65867317f64bbf2b4d3bb\": container with ID starting with a1ea37899d9c960c678e68a8aa7b07a00e5b40a2e5d65867317f64bbf2b4d3bb not found: ID does not exist"
Jul 21 17:28:45.452270 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.452197    2563 scope.go:117] "RemoveContainer" containerID="8dd1ae2c0ce23878b18a919130277656472eef2266493939e19ebd8d47a122f6"
Jul 21 17:28:45.452399 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.452383    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dd1ae2c0ce23878b18a919130277656472eef2266493939e19ebd8d47a122f6"} err="failed to get container status \"8dd1ae2c0ce23878b18a919130277656472eef2266493939e19ebd8d47a122f6\": rpc error: code = NotFound desc = could not find container \"8dd1ae2c0ce23878b18a919130277656472eef2266493939e19ebd8d47a122f6\": container with ID starting with 8dd1ae2c0ce23878b18a919130277656472eef2266493939e19ebd8d47a122f6 not found: ID does not exist"
Jul 21 17:28:45.468650 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.468622    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 21 17:28:45.468890 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.468877    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcac0b1f-28d9-4bfe-a032-30dd161d324c" containerName="prometheus"
Jul 21 17:28:45.468942 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.468893    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcac0b1f-28d9-4bfe-a032-30dd161d324c" containerName="prometheus"
Jul 21 17:28:45.468942 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.468904    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcac0b1f-28d9-4bfe-a032-30dd161d324c" containerName="init-config-reloader"
Jul 21 17:28:45.468942 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.468909    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcac0b1f-28d9-4bfe-a032-30dd161d324c" containerName="init-config-reloader"
Jul 21 17:28:45.468942 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.468916    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcac0b1f-28d9-4bfe-a032-30dd161d324c" containerName="thanos-sidecar"
Jul 21 17:28:45.468942 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.468921    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcac0b1f-28d9-4bfe-a032-30dd161d324c" containerName="thanos-sidecar"
Jul 21 17:28:45.468942 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.468928    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcac0b1f-28d9-4bfe-a032-30dd161d324c" containerName="kube-rbac-proxy-web"
Jul 21 17:28:45.468942 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.468934    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcac0b1f-28d9-4bfe-a032-30dd161d324c" containerName="kube-rbac-proxy-web"
Jul 21 17:28:45.468942 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.468941    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcac0b1f-28d9-4bfe-a032-30dd161d324c" containerName="config-reloader"
Jul 21 17:28:45.469214 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.468946    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcac0b1f-28d9-4bfe-a032-30dd161d324c" containerName="config-reloader"
Jul 21 17:28:45.469214 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.468953    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcac0b1f-28d9-4bfe-a032-30dd161d324c" containerName="kube-rbac-proxy"
Jul 21 17:28:45.469214 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.468958    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcac0b1f-28d9-4bfe-a032-30dd161d324c" containerName="kube-rbac-proxy"
Jul 21 17:28:45.469214 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.468966    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcac0b1f-28d9-4bfe-a032-30dd161d324c" containerName="kube-rbac-proxy-thanos"
Jul 21 17:28:45.469214 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.468971    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcac0b1f-28d9-4bfe-a032-30dd161d324c" containerName="kube-rbac-proxy-thanos"
Jul 21 17:28:45.469214 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.469013    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcac0b1f-28d9-4bfe-a032-30dd161d324c" containerName="kube-rbac-proxy-web"
Jul 21 17:28:45.469214 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.469020    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcac0b1f-28d9-4bfe-a032-30dd161d324c" containerName="config-reloader"
Jul 21 17:28:45.469214 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.469026    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcac0b1f-28d9-4bfe-a032-30dd161d324c" containerName="thanos-sidecar"
Jul 21 17:28:45.469214 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.469032    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcac0b1f-28d9-4bfe-a032-30dd161d324c" containerName="kube-rbac-proxy"
Jul 21 17:28:45.469214 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.469037    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcac0b1f-28d9-4bfe-a032-30dd161d324c" containerName="prometheus"
Jul 21 17:28:45.469214 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.469042    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcac0b1f-28d9-4bfe-a032-30dd161d324c" containerName="kube-rbac-proxy-thanos"
Jul 21 17:28:45.472615 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.472596    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.477079 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.477057    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 21 17:28:45.477329 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.477315    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 21 17:28:45.477423 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.477344    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 21 17:28:45.477692 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.477678    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 21 17:28:45.477887 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.477857    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 21 17:28:45.477887 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.477861    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 21 17:28:45.478231 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.477889    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 21 17:28:45.478231 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.477969    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 21 17:28:45.478231 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.478213    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 21 17:28:45.478695 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.478675    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 21 17:28:45.478784 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.478764    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 21 17:28:45.478951 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.478930    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lp86g\""
Jul 21 17:28:45.479229 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.479204    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 21 17:28:45.479955 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.479934    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 21 17:28:45.480200 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.480044    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-19uced80bb7e5\""
Jul 21 17:28:45.482168 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.482151    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 21 17:28:45.488276 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.488258    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 21 17:28:45.495623 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.495602    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 21 17:28:45.585644 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.585601    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df270e60-53ca-4479-9e00-78d0d0378cb1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.585644 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.585641    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/df270e60-53ca-4479-9e00-78d0d0378cb1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.585894 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.585667    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df270e60-53ca-4479-9e00-78d0d0378cb1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.585894 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.585692    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/df270e60-53ca-4479-9e00-78d0d0378cb1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.585894 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.585737    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/df270e60-53ca-4479-9e00-78d0d0378cb1-config\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.585894 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.585763    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df270e60-53ca-4479-9e00-78d0d0378cb1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.585894 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.585793    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df270e60-53ca-4479-9e00-78d0d0378cb1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.585894 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.585817    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/df270e60-53ca-4479-9e00-78d0d0378cb1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.585894 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.585840    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/df270e60-53ca-4479-9e00-78d0d0378cb1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.585894 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.585872    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59lnk\" (UniqueName: \"kubernetes.io/projected/df270e60-53ca-4479-9e00-78d0d0378cb1-kube-api-access-59lnk\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.586277 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.585899    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/df270e60-53ca-4479-9e00-78d0d0378cb1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.586277 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.585922    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/df270e60-53ca-4479-9e00-78d0d0378cb1-config-out\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.586277 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.585948    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/df270e60-53ca-4479-9e00-78d0d0378cb1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.586277 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.585973    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/df270e60-53ca-4479-9e00-78d0d0378cb1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.586277 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.586015    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df270e60-53ca-4479-9e00-78d0d0378cb1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.586277 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.586052    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/df270e60-53ca-4479-9e00-78d0d0378cb1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.586277 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.586074    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/df270e60-53ca-4479-9e00-78d0d0378cb1-web-config\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.586277 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.586103    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df270e60-53ca-4479-9e00-78d0d0378cb1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.586277 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.586184    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/df270e60-53ca-4479-9e00-78d0d0378cb1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.586277 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.586253    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/df270e60-53ca-4479-9e00-78d0d0378cb1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.687656 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.687568    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/df270e60-53ca-4479-9e00-78d0d0378cb1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.687656 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.687612    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/df270e60-53ca-4479-9e00-78d0d0378cb1-config\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.687656 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.687635    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df270e60-53ca-4479-9e00-78d0d0378cb1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.687903 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.687663    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df270e60-53ca-4479-9e00-78d0d0378cb1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.687903 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.687686    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/df270e60-53ca-4479-9e00-78d0d0378cb1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.687903 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.687707    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/df270e60-53ca-4479-9e00-78d0d0378cb1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.687903 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.687735    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59lnk\" (UniqueName: \"kubernetes.io/projected/df270e60-53ca-4479-9e00-78d0d0378cb1-kube-api-access-59lnk\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.687903 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.687757    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/df270e60-53ca-4479-9e00-78d0d0378cb1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.687903 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.687782    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/df270e60-53ca-4479-9e00-78d0d0378cb1-config-out\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.687903 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.687805    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/df270e60-53ca-4479-9e00-78d0d0378cb1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.687903 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.687829    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/df270e60-53ca-4479-9e00-78d0d0378cb1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.687903 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.687873    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df270e60-53ca-4479-9e00-78d0d0378cb1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.688365 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.687909    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/df270e60-53ca-4479-9e00-78d0d0378cb1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.688365 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.687937    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/df270e60-53ca-4479-9e00-78d0d0378cb1-web-config\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.688365 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.687970    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df270e60-53ca-4479-9e00-78d0d0378cb1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.688365 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.687998    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/df270e60-53ca-4479-9e00-78d0d0378cb1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.688365 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.688029    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/df270e60-53ca-4479-9e00-78d0d0378cb1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.688365 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.688082    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df270e60-53ca-4479-9e00-78d0d0378cb1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.688365 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.688105    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/df270e60-53ca-4479-9e00-78d0d0378cb1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.688365 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.688151    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df270e60-53ca-4479-9e00-78d0d0378cb1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.688757 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.688511    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df270e60-53ca-4479-9e00-78d0d0378cb1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.688971 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.688949    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df270e60-53ca-4479-9e00-78d0d0378cb1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.691116 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.689242    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/df270e60-53ca-4479-9e00-78d0d0378cb1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.691116 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.689766    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df270e60-53ca-4479-9e00-78d0d0378cb1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.691116 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.689802    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/df270e60-53ca-4479-9e00-78d0d0378cb1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.691116 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.690786    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df270e60-53ca-4479-9e00-78d0d0378cb1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.691116 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.691027    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/df270e60-53ca-4479-9e00-78d0d0378cb1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.691116 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.691042    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/df270e60-53ca-4479-9e00-78d0d0378cb1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.691510 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.691486    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df270e60-53ca-4479-9e00-78d0d0378cb1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.692836 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.692254    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/df270e60-53ca-4479-9e00-78d0d0378cb1-web-config\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.692836 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.692646    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/df270e60-53ca-4479-9e00-78d0d0378cb1-config\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.692836 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.692817    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/df270e60-53ca-4479-9e00-78d0d0378cb1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.693063 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.692855    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/df270e60-53ca-4479-9e00-78d0d0378cb1-config-out\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.693063 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.692948    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/df270e60-53ca-4479-9e00-78d0d0378cb1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.693218 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.693114    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/df270e60-53ca-4479-9e00-78d0d0378cb1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.693332 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.693313    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/df270e60-53ca-4479-9e00-78d0d0378cb1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.693381 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.693315    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df270e60-53ca-4479-9e00-78d0d0378cb1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.693759 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.693740    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/df270e60-53ca-4479-9e00-78d0d0378cb1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.693825 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.693767    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/df270e60-53ca-4479-9e00-78d0d0378cb1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.699774 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.699749    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59lnk\" (UniqueName: \"kubernetes.io/projected/df270e60-53ca-4479-9e00-78d0d0378cb1-kube-api-access-59lnk\") pod \"prometheus-k8s-0\" (UID: \"df270e60-53ca-4479-9e00-78d0d0378cb1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.783296 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.783260    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:45.927946 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:45.927917    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 21 17:28:45.930475 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:28:45.930447    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf270e60_53ca_4479_9e00_78d0d0378cb1.slice/crio-fe712bc6015f0177d94f72a43262506658e98106d2d7fcd4bfb63ec0b2fe730a WatchSource:0}: Error finding container fe712bc6015f0177d94f72a43262506658e98106d2d7fcd4bfb63ec0b2fe730a: Status 404 returned error can't find the container with id fe712bc6015f0177d94f72a43262506658e98106d2d7fcd4bfb63ec0b2fe730a
Jul 21 17:28:46.397885 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:46.397849    2563 generic.go:356] "Generic (PLEG): container finished" podID="df270e60-53ca-4479-9e00-78d0d0378cb1" containerID="ce59d2606c3a8ccd0c27f97318c5b55cd9023e713d01562da1755e52f30c390b" exitCode=0
Jul 21 17:28:46.398049 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:46.397935    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"df270e60-53ca-4479-9e00-78d0d0378cb1","Type":"ContainerDied","Data":"ce59d2606c3a8ccd0c27f97318c5b55cd9023e713d01562da1755e52f30c390b"}
Jul 21 17:28:46.398049 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:46.398016    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"df270e60-53ca-4479-9e00-78d0d0378cb1","Type":"ContainerStarted","Data":"fe712bc6015f0177d94f72a43262506658e98106d2d7fcd4bfb63ec0b2fe730a"}
Jul 21 17:28:46.764110 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:46.764079    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcac0b1f-28d9-4bfe-a032-30dd161d324c" path="/var/lib/kubelet/pods/fcac0b1f-28d9-4bfe-a032-30dd161d324c/volumes"
Jul 21 17:28:47.406164 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:47.406118    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"df270e60-53ca-4479-9e00-78d0d0378cb1","Type":"ContainerStarted","Data":"344783a870c58f31c24b6091967bb0704fe350442373801e907699a6362b6851"}
Jul 21 17:28:47.406565 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:47.406200    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"df270e60-53ca-4479-9e00-78d0d0378cb1","Type":"ContainerStarted","Data":"b23ca37fe592a2aaabef5769bf4dca01245ee981af3364b38d350e1b49d6c077"}
Jul 21 17:28:47.406565 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:47.406226    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"df270e60-53ca-4479-9e00-78d0d0378cb1","Type":"ContainerStarted","Data":"609c5a3b54627ec40fa47264d457cf5d6a0a6d8d15983d76af768b5ebdf2b4c2"}
Jul 21 17:28:47.406565 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:47.406250    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"df270e60-53ca-4479-9e00-78d0d0378cb1","Type":"ContainerStarted","Data":"094ec635af641f4a38867543f70d34e4f84659c1cba3d44ea22490646ea8dac3"}
Jul 21 17:28:47.406565 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:47.406274    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"df270e60-53ca-4479-9e00-78d0d0378cb1","Type":"ContainerStarted","Data":"10313e426a6e1edffafbcdc1c5171b28db237afdbf3af79e47ee63c1469ed7ad"}
Jul 21 17:28:47.406565 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:47.406296    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"df270e60-53ca-4479-9e00-78d0d0378cb1","Type":"ContainerStarted","Data":"fe34d268b5206267a2329f7d4d5a090dc1767457d733afdaf8cc4b626055d268"}
Jul 21 17:28:47.444713 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:47.444644    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.425179232 podStartE2EDuration="2.444622974s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.443742ms" isStatefulPod=false podCreationTimestamp="2026-07-21 17:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:28:47.444167111 +0000 UTC m=+155.206306730" watchObservedRunningTime="2026-07-21 17:28:47.444622974 +0000 UTC m=+155.206762582"
Jul 21 17:28:50.784114 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:28:50.784069    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:45.784194 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:29:45.784154    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:45.799885 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:29:45.799859    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:46.626080 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:29:46.626057    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:31:12.605688 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:31:12.605663    2563 kubelet.go:1648] "Image garbage collection succeeded"
Jul 21 17:34:10.282976 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:10.282942    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-cnzn7"]
Jul 21 17:34:10.285122 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:10.285106    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-cnzn7"
Jul 21 17:34:10.289318 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:10.289297    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 21 17:34:10.290534 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:10.290368    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-4mtpm\""
Jul 21 17:34:10.290534 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:10.290410    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 21 17:34:10.290534 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:10.290432    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 21 17:34:10.299869 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:10.299846    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-qwdl2"]
Jul 21 17:34:10.302194 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:10.302173    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-qwdl2"
Jul 21 17:34:10.302287 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:10.302242    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-cnzn7"]
Jul 21 17:34:10.304854 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:10.304834    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 21 17:34:10.304954 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:10.304899    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-wl66s\""
Jul 21 17:34:10.315558 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:10.315532    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-qwdl2"]
Jul 21 17:34:10.388446 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:10.388410    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnrzk\" (UniqueName: \"kubernetes.io/projected/75a8d379-853b-456d-9434-1ae6451caf47-kube-api-access-nnrzk\") pod \"model-serving-api-fd65d7d6-cnzn7\" (UID: \"75a8d379-853b-456d-9434-1ae6451caf47\") " pod="kserve/model-serving-api-fd65d7d6-cnzn7"
Jul 21 17:34:10.388620 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:10.388466    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/52ad3d48-f859-4ac0-b406-4944f6e62afa-cert\") pod \"odh-model-controller-84b54b7f97-qwdl2\" (UID: \"52ad3d48-f859-4ac0-b406-4944f6e62afa\") " pod="kserve/odh-model-controller-84b54b7f97-qwdl2"
Jul 21 17:34:10.388620 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:10.388526    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75a8d379-853b-456d-9434-1ae6451caf47-tls-certs\") pod \"model-serving-api-fd65d7d6-cnzn7\" (UID: \"75a8d379-853b-456d-9434-1ae6451caf47\") " pod="kserve/model-serving-api-fd65d7d6-cnzn7"
Jul 21 17:34:10.388620 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:10.388580    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw2z4\" (UniqueName: \"kubernetes.io/projected/52ad3d48-f859-4ac0-b406-4944f6e62afa-kube-api-access-rw2z4\") pod \"odh-model-controller-84b54b7f97-qwdl2\" (UID: \"52ad3d48-f859-4ac0-b406-4944f6e62afa\") " pod="kserve/odh-model-controller-84b54b7f97-qwdl2"
Jul 21 17:34:10.489893 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:10.489861    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/52ad3d48-f859-4ac0-b406-4944f6e62afa-cert\") pod \"odh-model-controller-84b54b7f97-qwdl2\" (UID: \"52ad3d48-f859-4ac0-b406-4944f6e62afa\") " pod="kserve/odh-model-controller-84b54b7f97-qwdl2"
Jul 21 17:34:10.489893 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:10.489890    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75a8d379-853b-456d-9434-1ae6451caf47-tls-certs\") pod \"model-serving-api-fd65d7d6-cnzn7\" (UID: \"75a8d379-853b-456d-9434-1ae6451caf47\") " pod="kserve/model-serving-api-fd65d7d6-cnzn7"
Jul 21 17:34:10.490099 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:10.489919    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw2z4\" (UniqueName: \"kubernetes.io/projected/52ad3d48-f859-4ac0-b406-4944f6e62afa-kube-api-access-rw2z4\") pod \"odh-model-controller-84b54b7f97-qwdl2\" (UID: \"52ad3d48-f859-4ac0-b406-4944f6e62afa\") " pod="kserve/odh-model-controller-84b54b7f97-qwdl2"
Jul 21 17:34:10.490099 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:10.490045    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnrzk\" (UniqueName: \"kubernetes.io/projected/75a8d379-853b-456d-9434-1ae6451caf47-kube-api-access-nnrzk\") pod \"model-serving-api-fd65d7d6-cnzn7\" (UID: \"75a8d379-853b-456d-9434-1ae6451caf47\") " pod="kserve/model-serving-api-fd65d7d6-cnzn7"
Jul 21 17:34:10.492464 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:10.492439    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75a8d379-853b-456d-9434-1ae6451caf47-tls-certs\") pod \"model-serving-api-fd65d7d6-cnzn7\" (UID: \"75a8d379-853b-456d-9434-1ae6451caf47\") " pod="kserve/model-serving-api-fd65d7d6-cnzn7"
Jul 21 17:34:10.492619 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:10.492596    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/52ad3d48-f859-4ac0-b406-4944f6e62afa-cert\") pod \"odh-model-controller-84b54b7f97-qwdl2\" (UID: \"52ad3d48-f859-4ac0-b406-4944f6e62afa\") " pod="kserve/odh-model-controller-84b54b7f97-qwdl2"
Jul 21 17:34:10.501257 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:10.501226    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw2z4\" (UniqueName: \"kubernetes.io/projected/52ad3d48-f859-4ac0-b406-4944f6e62afa-kube-api-access-rw2z4\") pod \"odh-model-controller-84b54b7f97-qwdl2\" (UID: \"52ad3d48-f859-4ac0-b406-4944f6e62afa\") " pod="kserve/odh-model-controller-84b54b7f97-qwdl2"
Jul 21 17:34:10.501589 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:10.501572    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnrzk\" (UniqueName: \"kubernetes.io/projected/75a8d379-853b-456d-9434-1ae6451caf47-kube-api-access-nnrzk\") pod \"model-serving-api-fd65d7d6-cnzn7\" (UID: \"75a8d379-853b-456d-9434-1ae6451caf47\") " pod="kserve/model-serving-api-fd65d7d6-cnzn7"
Jul 21 17:34:10.595694 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:10.595605    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-cnzn7"
Jul 21 17:34:10.611874 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:10.611841    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-qwdl2"
Jul 21 17:34:10.774227 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:10.774122    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 21 17:34:10.774502 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:10.774418    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-cnzn7"]
Jul 21 17:34:10.790882 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:10.790854    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-qwdl2"]
Jul 21 17:34:10.796166 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:34:10.796112    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ad3d48_f859_4ac0_b406_4944f6e62afa.slice/crio-3e6d321de6fcb2c0e905edff84b6e2955fcfdd31488b9cdfdbb8c98d0a57e0c6 WatchSource:0}: Error finding container 3e6d321de6fcb2c0e905edff84b6e2955fcfdd31488b9cdfdbb8c98d0a57e0c6: Status 404 returned error can't find the container with id 3e6d321de6fcb2c0e905edff84b6e2955fcfdd31488b9cdfdbb8c98d0a57e0c6
Jul 21 17:34:11.494891 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:11.494853    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-qwdl2" event={"ID":"52ad3d48-f859-4ac0-b406-4944f6e62afa","Type":"ContainerStarted","Data":"3e6d321de6fcb2c0e905edff84b6e2955fcfdd31488b9cdfdbb8c98d0a57e0c6"}
Jul 21 17:34:11.497000 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:11.496921    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-cnzn7" event={"ID":"75a8d379-853b-456d-9434-1ae6451caf47","Type":"ContainerStarted","Data":"447dafbc3e2988320acedb60c3b34171c356e3770ed638ac9c295d126c1f308c"}
Jul 21 17:34:14.514072 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:14.514036    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-qwdl2" event={"ID":"52ad3d48-f859-4ac0-b406-4944f6e62afa","Type":"ContainerStarted","Data":"e94b6676f74ff66cff3a43222370d5e7adde22640133174dce07e0d813f6c325"}
Jul 21 17:34:14.514516 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:14.514157    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-qwdl2"
Jul 21 17:34:14.515530 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:14.515507    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-cnzn7" event={"ID":"75a8d379-853b-456d-9434-1ae6451caf47","Type":"ContainerStarted","Data":"473bbe8907d68b4e442dffb62311e8e12bce19dfd146e51bf2ab9534be019a3f"}
Jul 21 17:34:14.515634 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:14.515548    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-cnzn7"
Jul 21 17:34:14.552528 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:14.552476    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-qwdl2" podStartSLOduration=0.96022926 podStartE2EDuration="4.552459987s" totalImagesPullingTime="3.592230727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:34:14.531626721 +0000 UTC m=+482.293766325" watchObservedRunningTime="2026-07-21 17:34:14.552459987 +0000 UTC m=+482.314599591"
Jul 21 17:34:14.552844 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:14.552823    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-cnzn7" podStartSLOduration=0.982473124 podStartE2EDuration="4.552818534s" totalImagesPullingTime="3.57034541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:34:14.552518386 +0000 UTC m=+482.314657992" watchObservedRunningTime="2026-07-21 17:34:14.552818534 +0000 UTC m=+482.314958139"
Jul 21 17:34:25.526258 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:25.526230    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-qwdl2"
Jul 21 17:34:25.528203 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:25.528182    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-cnzn7"
Jul 21 17:34:26.425172 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:26.425122    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-gjxfp"]
Jul 21 17:34:26.428775 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:26.428758    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-gjxfp"
Jul 21 17:34:26.431403 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:26.431387    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 21 17:34:26.431505 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:26.431384    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-xsj9q\""
Jul 21 17:34:26.438503 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:26.438483    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-gjxfp"]
Jul 21 17:34:26.519824 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:26.519788    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt7sr\" (UniqueName: \"kubernetes.io/projected/0153b238-ec76-4384-b471-226b0feb6df7-kube-api-access-vt7sr\") pod \"s3-init-gjxfp\" (UID: \"0153b238-ec76-4384-b471-226b0feb6df7\") " pod="kserve/s3-init-gjxfp"
Jul 21 17:34:26.519971 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:26.519831    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0153b238-ec76-4384-b471-226b0feb6df7-model-cache\") pod \"s3-init-gjxfp\" (UID: \"0153b238-ec76-4384-b471-226b0feb6df7\") " pod="kserve/s3-init-gjxfp"
Jul 21 17:34:26.620859 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:26.620825    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt7sr\" (UniqueName: \"kubernetes.io/projected/0153b238-ec76-4384-b471-226b0feb6df7-kube-api-access-vt7sr\") pod \"s3-init-gjxfp\" (UID: \"0153b238-ec76-4384-b471-226b0feb6df7\") " pod="kserve/s3-init-gjxfp"
Jul 21 17:34:26.621208 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:26.620872    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0153b238-ec76-4384-b471-226b0feb6df7-model-cache\") pod \"s3-init-gjxfp\" (UID: \"0153b238-ec76-4384-b471-226b0feb6df7\") " pod="kserve/s3-init-gjxfp"
Jul 21 17:34:26.621274 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:26.621255    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0153b238-ec76-4384-b471-226b0feb6df7-model-cache\") pod \"s3-init-gjxfp\" (UID: \"0153b238-ec76-4384-b471-226b0feb6df7\") " pod="kserve/s3-init-gjxfp"
Jul 21 17:34:26.635379 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:26.635356    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt7sr\" (UniqueName: \"kubernetes.io/projected/0153b238-ec76-4384-b471-226b0feb6df7-kube-api-access-vt7sr\") pod \"s3-init-gjxfp\" (UID: \"0153b238-ec76-4384-b471-226b0feb6df7\") " pod="kserve/s3-init-gjxfp"
Jul 21 17:34:26.754105 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:26.754036    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-gjxfp"
Jul 21 17:34:26.882757 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:26.882735    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-gjxfp"]
Jul 21 17:34:26.885024 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:34:26.884997    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0153b238_ec76_4384_b471_226b0feb6df7.slice/crio-3b2b82ff29ffe12bf92239f56732944195c4c5fa373a9aebf17a95d2cefb2406 WatchSource:0}: Error finding container 3b2b82ff29ffe12bf92239f56732944195c4c5fa373a9aebf17a95d2cefb2406: Status 404 returned error can't find the container with id 3b2b82ff29ffe12bf92239f56732944195c4c5fa373a9aebf17a95d2cefb2406
Jul 21 17:34:27.568684 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:27.568640    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-gjxfp" event={"ID":"0153b238-ec76-4384-b471-226b0feb6df7","Type":"ContainerStarted","Data":"3b2b82ff29ffe12bf92239f56732944195c4c5fa373a9aebf17a95d2cefb2406"}
Jul 21 17:34:30.581826 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:30.581781    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-gjxfp" event={"ID":"0153b238-ec76-4384-b471-226b0feb6df7","Type":"ContainerStarted","Data":"990516bc60a4b415e2dba5cc1e4d7868eee98f90bd7ef2ab7a86386ae6245066"}
Jul 21 17:34:33.594741 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:33.594711    2563 generic.go:356] "Generic (PLEG): container finished" podID="0153b238-ec76-4384-b471-226b0feb6df7" containerID="990516bc60a4b415e2dba5cc1e4d7868eee98f90bd7ef2ab7a86386ae6245066" exitCode=0
Jul 21 17:34:33.595149 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:33.594794    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-gjxfp" event={"ID":"0153b238-ec76-4384-b471-226b0feb6df7","Type":"ContainerDied","Data":"990516bc60a4b415e2dba5cc1e4d7868eee98f90bd7ef2ab7a86386ae6245066"}
Jul 21 17:34:38.616807 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:38.616763    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-gjxfp" event={"ID":"0153b238-ec76-4384-b471-226b0feb6df7","Type":"ContainerStarted","Data":"e364a217a6986953c367c215fd5d87a5034d44ff7729afcfcc68d01185a80c48"}
Jul 21 17:34:38.635656 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:34:38.635609    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-gjxfp" podStartSLOduration=2.523299791 podStartE2EDuration="12.635594249s" totalImagesPullingTime="7.858987651s" totalInitContainerRuntime="2.253306807s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:34:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:34:38.633968876 +0000 UTC m=+506.396108493" watchObservedRunningTime="2026-07-21 17:34:38.635594249 +0000 UTC m=+506.397733852"
Jul 21 17:35:43.851970 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:43.851936    2563 generic.go:356] "Generic (PLEG): container finished" podID="0153b238-ec76-4384-b471-226b0feb6df7" containerID="e364a217a6986953c367c215fd5d87a5034d44ff7729afcfcc68d01185a80c48" exitCode=0
Jul 21 17:35:43.852390 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:43.852009    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-gjxfp" event={"ID":"0153b238-ec76-4384-b471-226b0feb6df7","Type":"ContainerDied","Data":"e364a217a6986953c367c215fd5d87a5034d44ff7729afcfcc68d01185a80c48"}
Jul 21 17:35:44.980078 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:44.980054    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-gjxfp"
Jul 21 17:35:45.020757 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:45.020735    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0153b238-ec76-4384-b471-226b0feb6df7-model-cache\") pod \"0153b238-ec76-4384-b471-226b0feb6df7\" (UID: \"0153b238-ec76-4384-b471-226b0feb6df7\") "
Jul 21 17:35:45.020909 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:45.020822    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vt7sr\" (UniqueName: \"kubernetes.io/projected/0153b238-ec76-4384-b471-226b0feb6df7-kube-api-access-vt7sr\") pod \"0153b238-ec76-4384-b471-226b0feb6df7\" (UID: \"0153b238-ec76-4384-b471-226b0feb6df7\") "
Jul 21 17:35:45.023038 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:45.023012    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0153b238-ec76-4384-b471-226b0feb6df7-kube-api-access-vt7sr" (OuterVolumeSpecName: "kube-api-access-vt7sr") pod "0153b238-ec76-4384-b471-226b0feb6df7" (UID: "0153b238-ec76-4384-b471-226b0feb6df7"). InnerVolumeSpecName "kube-api-access-vt7sr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:35:45.042354 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:45.042328    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0153b238-ec76-4384-b471-226b0feb6df7-model-cache" (OuterVolumeSpecName: "model-cache") pod "0153b238-ec76-4384-b471-226b0feb6df7" (UID: "0153b238-ec76-4384-b471-226b0feb6df7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 17:35:45.121523 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:45.121498    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt7sr\" (UniqueName: \"kubernetes.io/projected/0153b238-ec76-4384-b471-226b0feb6df7-kube-api-access-vt7sr\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:35:45.121523 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:45.121523    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0153b238-ec76-4384-b471-226b0feb6df7-model-cache\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:35:45.860722 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:45.860698    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-gjxfp"
Jul 21 17:35:45.860883 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:45.860698    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-gjxfp" event={"ID":"0153b238-ec76-4384-b471-226b0feb6df7","Type":"ContainerDied","Data":"3b2b82ff29ffe12bf92239f56732944195c4c5fa373a9aebf17a95d2cefb2406"}
Jul 21 17:35:45.860883 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:45.860809    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b2b82ff29ffe12bf92239f56732944195c4c5fa373a9aebf17a95d2cefb2406"
Jul 21 17:35:58.277751 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.277720    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7"]
Jul 21 17:35:58.278200 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.278009    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0153b238-ec76-4384-b471-226b0feb6df7" containerName="s3-init"
Jul 21 17:35:58.278200 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.278019    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0153b238-ec76-4384-b471-226b0feb6df7" containerName="s3-init"
Jul 21 17:35:58.278200 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.278028    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0153b238-ec76-4384-b471-226b0feb6df7" containerName="download-hf-model"
Jul 21 17:35:58.278200 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.278034    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0153b238-ec76-4384-b471-226b0feb6df7" containerName="download-hf-model"
Jul 21 17:35:58.278200 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.278088    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0153b238-ec76-4384-b471-226b0feb6df7" containerName="s3-init"
Jul 21 17:35:58.280398 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.280382    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7"
Jul 21 17:35:58.283756 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.283725    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4dfe2-kube-rbac-proxy-sar-config\""
Jul 21 17:35:58.284950 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.284932    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4dfe2-predictor-serving-cert\""
Jul 21 17:35:58.284950 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.284940    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 21 17:35:58.285169 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.284987    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wx68x\""
Jul 21 17:35:58.285169 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.284996    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 21 17:35:58.296318 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.296297    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7"]
Jul 21 17:35:58.418488 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.418451    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blc88\" (UniqueName: \"kubernetes.io/projected/2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46-kube-api-access-blc88\") pod \"success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7\" (UID: \"2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46\") " pod="kserve-ci-e2e-test/success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7"
Jul 21 17:35:58.418664 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.418495    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46-proxy-tls\") pod \"success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7\" (UID: \"2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46\") " pod="kserve-ci-e2e-test/success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7"
Jul 21 17:35:58.418664 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.418528    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7\" (UID: \"2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46\") " pod="kserve-ci-e2e-test/success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7"
Jul 21 17:35:58.519303 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.519269    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blc88\" (UniqueName: \"kubernetes.io/projected/2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46-kube-api-access-blc88\") pod \"success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7\" (UID: \"2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46\") " pod="kserve-ci-e2e-test/success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7"
Jul 21 17:35:58.519468 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.519313    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46-proxy-tls\") pod \"success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7\" (UID: \"2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46\") " pod="kserve-ci-e2e-test/success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7"
Jul 21 17:35:58.519468 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.519451    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7\" (UID: \"2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46\") " pod="kserve-ci-e2e-test/success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7"
Jul 21 17:35:58.520094 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.520021    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7\" (UID: \"2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46\") " pod="kserve-ci-e2e-test/success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7"
Jul 21 17:35:58.521868 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.521848    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46-proxy-tls\") pod \"success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7\" (UID: \"2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46\") " pod="kserve-ci-e2e-test/success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7"
Jul 21 17:35:58.529887 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.529837    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blc88\" (UniqueName: \"kubernetes.io/projected/2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46-kube-api-access-blc88\") pod \"success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7\" (UID: \"2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46\") " pod="kserve-ci-e2e-test/success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7"
Jul 21 17:35:58.581020 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.580993    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt"]
Jul 21 17:35:58.583430 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.583413    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt"
Jul 21 17:35:58.587232 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.587215    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\""
Jul 21 17:35:58.587321 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.587213    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 17:35:58.587380 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.587361    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-predictor-serving-cert\""
Jul 21 17:35:58.587450 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.587425    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 21 17:35:58.590444 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.590387    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7"
Jul 21 17:35:58.596490 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.596469    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt"]
Jul 21 17:35:58.672584 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.672545    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw"]
Jul 21 17:35:58.676332 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.676312    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw"
Jul 21 17:35:58.680938 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.680452    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4dfe2-predictor-serving-cert\""
Jul 21 17:35:58.680938 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.680466    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4dfe2-kube-rbac-proxy-sar-config\""
Jul 21 17:35:58.690641 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.690552    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw"]
Jul 21 17:35:58.720489 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.720461    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69e0a43b-e6af-4859-9a8f-872ada531c9d-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt\" (UID: \"69e0a43b-e6af-4859-9a8f-872ada531c9d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt"
Jul 21 17:35:58.720651 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.720503    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69e0a43b-e6af-4859-9a8f-872ada531c9d-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt\" (UID: \"69e0a43b-e6af-4859-9a8f-872ada531c9d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt"
Jul 21 17:35:58.720651 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.720525    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgc9m\" (UniqueName: \"kubernetes.io/projected/e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf-kube-api-access-rgc9m\") pod \"error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw\" (UID: \"e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf\") " pod="kserve-ci-e2e-test/error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw"
Jul 21 17:35:58.720651 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.720589    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqhh4\" (UniqueName: \"kubernetes.io/projected/69e0a43b-e6af-4859-9a8f-872ada531c9d-kube-api-access-bqhh4\") pod \"isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt\" (UID: \"69e0a43b-e6af-4859-9a8f-872ada531c9d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt"
Jul 21 17:35:58.720651 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.720644    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69e0a43b-e6af-4859-9a8f-872ada531c9d-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt\" (UID: \"69e0a43b-e6af-4859-9a8f-872ada531c9d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt"
Jul 21 17:35:58.720882 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.720696    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw\" (UID: \"e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf\") " pod="kserve-ci-e2e-test/error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw"
Jul 21 17:35:58.720882 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.720727    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf-proxy-tls\") pod \"error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw\" (UID: \"e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf\") " pod="kserve-ci-e2e-test/error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw"
Jul 21 17:35:58.720882 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.720747    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69e0a43b-e6af-4859-9a8f-872ada531c9d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt\" (UID: \"69e0a43b-e6af-4859-9a8f-872ada531c9d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt"
Jul 21 17:35:58.733695 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.730221    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7"]
Jul 21 17:35:58.738301 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:35:58.738261    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ecdb2de_9c9b_4548_bc44_c9cc92f1ec46.slice/crio-6b59d56bb75c2701deeee32c965f774195f9566094d03fadd4f9d3c8a7ee3b1e WatchSource:0}: Error finding container 6b59d56bb75c2701deeee32c965f774195f9566094d03fadd4f9d3c8a7ee3b1e: Status 404 returned error can't find the container with id 6b59d56bb75c2701deeee32c965f774195f9566094d03fadd4f9d3c8a7ee3b1e
Jul 21 17:35:58.797117 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.797052    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs"]
Jul 21 17:35:58.799805 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.799790    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs"
Jul 21 17:35:58.802222 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.802203    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-predictor-serving-cert\""
Jul 21 17:35:58.802568 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.802554    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-kube-rbac-proxy-sar-config\""
Jul 21 17:35:58.811088 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.811064    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs"]
Jul 21 17:35:58.821435 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.821413    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69e0a43b-e6af-4859-9a8f-872ada531c9d-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt\" (UID: \"69e0a43b-e6af-4859-9a8f-872ada531c9d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt"
Jul 21 17:35:58.821564 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.821454    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgc9m\" (UniqueName: \"kubernetes.io/projected/e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf-kube-api-access-rgc9m\") pod \"error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw\" (UID: \"e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf\") " pod="kserve-ci-e2e-test/error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw"
Jul 21 17:35:58.821564 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.821488    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqhh4\" (UniqueName: \"kubernetes.io/projected/69e0a43b-e6af-4859-9a8f-872ada531c9d-kube-api-access-bqhh4\") pod \"isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt\" (UID: \"69e0a43b-e6af-4859-9a8f-872ada531c9d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt"
Jul 21 17:35:58.821564 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.821548    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69e0a43b-e6af-4859-9a8f-872ada531c9d-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt\" (UID: \"69e0a43b-e6af-4859-9a8f-872ada531c9d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt"
Jul 21 17:35:58.821721 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.821594    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw\" (UID: \"e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf\") " pod="kserve-ci-e2e-test/error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw"
Jul 21 17:35:58.821721 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.821635    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf-proxy-tls\") pod \"error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw\" (UID: \"e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf\") " pod="kserve-ci-e2e-test/error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw"
Jul 21 17:35:58.821721 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.821662    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69e0a43b-e6af-4859-9a8f-872ada531c9d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt\" (UID: \"69e0a43b-e6af-4859-9a8f-872ada531c9d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt"
Jul 21 17:35:58.821721 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.821705    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69e0a43b-e6af-4859-9a8f-872ada531c9d-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt\" (UID: \"69e0a43b-e6af-4859-9a8f-872ada531c9d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt"
Jul 21 17:35:58.821904 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.821843    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69e0a43b-e6af-4859-9a8f-872ada531c9d-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt\" (UID: \"69e0a43b-e6af-4859-9a8f-872ada531c9d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt"
Jul 21 17:35:58.821950 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:35:58.821933    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-4dfe2-predictor-serving-cert: secret "error-404-isvc-4dfe2-predictor-serving-cert" not found
Jul 21 17:35:58.822145 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:35:58.822002    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf-proxy-tls podName:e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf nodeName:}" failed. No retries permitted until 2026-07-21 17:35:59.32198199 +0000 UTC m=+587.084121572 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf-proxy-tls") pod "error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw" (UID: "e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf") : secret "error-404-isvc-4dfe2-predictor-serving-cert" not found
Jul 21 17:35:58.822481 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.822450    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69e0a43b-e6af-4859-9a8f-872ada531c9d-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt\" (UID: \"69e0a43b-e6af-4859-9a8f-872ada531c9d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt"
Jul 21 17:35:58.822594 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.822450    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw\" (UID: \"e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf\") " pod="kserve-ci-e2e-test/error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw"
Jul 21 17:35:58.822594 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.822531    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69e0a43b-e6af-4859-9a8f-872ada531c9d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt\" (UID: \"69e0a43b-e6af-4859-9a8f-872ada531c9d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt"
Jul 21 17:35:58.824686 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.824664    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69e0a43b-e6af-4859-9a8f-872ada531c9d-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt\" (UID: \"69e0a43b-e6af-4859-9a8f-872ada531c9d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt"
Jul 21 17:35:58.836424 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.836402    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgc9m\" (UniqueName: \"kubernetes.io/projected/e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf-kube-api-access-rgc9m\") pod \"error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw\" (UID: \"e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf\") " pod="kserve-ci-e2e-test/error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw"
Jul 21 17:35:58.836773 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.836752    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqhh4\" (UniqueName: \"kubernetes.io/projected/69e0a43b-e6af-4859-9a8f-872ada531c9d-kube-api-access-bqhh4\") pod \"isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt\" (UID: \"69e0a43b-e6af-4859-9a8f-872ada531c9d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt"
Jul 21 17:35:58.894177 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.894119    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt"
Jul 21 17:35:58.899752 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.899724    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c"]
Jul 21 17:35:58.902197 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.902180    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c"
Jul 21 17:35:58.904780 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.904758    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\""
Jul 21 17:35:58.904881 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.904848    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\""
Jul 21 17:35:58.911038 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.911013    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7" event={"ID":"2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46","Type":"ContainerStarted","Data":"6b59d56bb75c2701deeee32c965f774195f9566094d03fadd4f9d3c8a7ee3b1e"}
Jul 21 17:35:58.914281 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.914260    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c"]
Jul 21 17:35:58.922237 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.922210    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2w9gs\" (UID: \"8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs"
Jul 21 17:35:58.922342 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.922252    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmg7g\" (UniqueName: \"kubernetes.io/projected/8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10-kube-api-access-qmg7g\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2w9gs\" (UID: \"8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs"
Jul 21 17:35:58.922342 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.922319    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2w9gs\" (UID: \"8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs"
Jul 21 17:35:58.922451 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.922408    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2w9gs\" (UID: \"8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs"
Jul 21 17:35:58.922549 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:58.922528    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2w9gs\" (UID: \"8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs"
Jul 21 17:35:59.023828 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:59.023792    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2w9gs\" (UID: \"8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs"
Jul 21 17:35:59.023828 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:59.023830    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e8d920d-8b7d-4536-bae9-593c6439d608-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-868d77456c-snz9c\" (UID: \"6e8d920d-8b7d-4536-bae9-593c6439d608\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c"
Jul 21 17:35:59.024006 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:59.023851    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmg7g\" (UniqueName: \"kubernetes.io/projected/8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10-kube-api-access-qmg7g\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2w9gs\" (UID: \"8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs"
Jul 21 17:35:59.024006 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:59.023888    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e8d920d-8b7d-4536-bae9-593c6439d608-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-868d77456c-snz9c\" (UID: \"6e8d920d-8b7d-4536-bae9-593c6439d608\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c"
Jul 21 17:35:59.024006 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:35:59.023967    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-predictor-serving-cert: secret "isvc-xgboost-graph-predictor-serving-cert" not found
Jul 21 17:35:59.024175 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:35:59.024028    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10-proxy-tls podName:8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10 nodeName:}" failed. No retries permitted until 2026-07-21 17:35:59.524009383 +0000 UTC m=+587.286148965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10-proxy-tls") pod "isvc-xgboost-graph-predictor-555fc69d56-2w9gs" (UID: "8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10") : secret "isvc-xgboost-graph-predictor-serving-cert" not found
Jul 21 17:35:59.024175 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:59.024071    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2w9gs\" (UID: \"8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs"
Jul 21 17:35:59.024175 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:59.024155    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2w9gs\" (UID: \"8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs"
Jul 21 17:35:59.024328 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:59.024218    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e8d920d-8b7d-4536-bae9-593c6439d608-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-868d77456c-snz9c\" (UID: \"6e8d920d-8b7d-4536-bae9-593c6439d608\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c"
Jul 21 17:35:59.024328 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:59.024254    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww47z\" (UniqueName: \"kubernetes.io/projected/6e8d920d-8b7d-4536-bae9-593c6439d608-kube-api-access-ww47z\") pod \"isvc-sklearn-graph-2-predictor-868d77456c-snz9c\" (UID: \"6e8d920d-8b7d-4536-bae9-593c6439d608\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c"
Jul 21 17:35:59.024442 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:59.024332    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2w9gs\" (UID: \"8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs"
Jul 21 17:35:59.024442 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:59.024371    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e8d920d-8b7d-4536-bae9-593c6439d608-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-868d77456c-snz9c\" (UID: \"6e8d920d-8b7d-4536-bae9-593c6439d608\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c"
Jul 21 17:35:59.024582 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:59.024564    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2w9gs\" (UID: \"8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs"
Jul 21 17:35:59.024770 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:59.024747    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2w9gs\" (UID: \"8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs"
Jul 21 17:35:59.024916 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:59.024898    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2w9gs\" (UID: \"8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs"
Jul 21 17:35:59.036752 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:59.036733    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt"]
Jul 21 17:35:59.039269 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:35:59.039249    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69e0a43b_e6af_4859_9a8f_872ada531c9d.slice/crio-51febc87b0113be6e4674f6c0a6626117319d2c987b1d004de2b826d1a33cfc5 WatchSource:0}: Error finding container 51febc87b0113be6e4674f6c0a6626117319d2c987b1d004de2b826d1a33cfc5: Status 404 returned error can't find the container with id 51febc87b0113be6e4674f6c0a6626117319d2c987b1d004de2b826d1a33cfc5
Jul 21 17:35:59.040999 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:59.040933    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmg7g\" (UniqueName: \"kubernetes.io/projected/8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10-kube-api-access-qmg7g\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2w9gs\" (UID: \"8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs"
Jul 21 17:35:59.125328 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:59.125300    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e8d920d-8b7d-4536-bae9-593c6439d608-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-868d77456c-snz9c\" (UID: \"6e8d920d-8b7d-4536-bae9-593c6439d608\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c"
Jul 21 17:35:59.125488 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:59.125374    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e8d920d-8b7d-4536-bae9-593c6439d608-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-868d77456c-snz9c\" (UID: \"6e8d920d-8b7d-4536-bae9-593c6439d608\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c"
Jul 21 17:35:59.125488 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:59.125404    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e8d920d-8b7d-4536-bae9-593c6439d608-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-868d77456c-snz9c\" (UID: \"6e8d920d-8b7d-4536-bae9-593c6439d608\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c"
Jul 21 17:35:59.125488 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:59.125450    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e8d920d-8b7d-4536-bae9-593c6439d608-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-868d77456c-snz9c\" (UID: \"6e8d920d-8b7d-4536-bae9-593c6439d608\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c"
Jul 21 17:35:59.125488 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:59.125480    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww47z\" (UniqueName: \"kubernetes.io/projected/6e8d920d-8b7d-4536-bae9-593c6439d608-kube-api-access-ww47z\") pod \"isvc-sklearn-graph-2-predictor-868d77456c-snz9c\" (UID: \"6e8d920d-8b7d-4536-bae9-593c6439d608\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c"
Jul 21 17:35:59.125932 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:59.125906    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e8d920d-8b7d-4536-bae9-593c6439d608-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-868d77456c-snz9c\" (UID: \"6e8d920d-8b7d-4536-bae9-593c6439d608\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c"
Jul 21 17:35:59.126210 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:59.126056    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e8d920d-8b7d-4536-bae9-593c6439d608-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-868d77456c-snz9c\" (UID: \"6e8d920d-8b7d-4536-bae9-593c6439d608\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c"
Jul 21 17:35:59.126210 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:59.126055    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e8d920d-8b7d-4536-bae9-593c6439d608-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-868d77456c-snz9c\" (UID: \"6e8d920d-8b7d-4536-bae9-593c6439d608\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c"
Jul 21 17:35:59.127777 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:59.127758    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e8d920d-8b7d-4536-bae9-593c6439d608-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-868d77456c-snz9c\" (UID: \"6e8d920d-8b7d-4536-bae9-593c6439d608\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c"
Jul 21 17:35:59.135961 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:59.135942    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww47z\" (UniqueName: \"kubernetes.io/projected/6e8d920d-8b7d-4536-bae9-593c6439d608-kube-api-access-ww47z\") pod \"isvc-sklearn-graph-2-predictor-868d77456c-snz9c\" (UID: \"6e8d920d-8b7d-4536-bae9-593c6439d608\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c"
Jul 21 17:35:59.214918 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:59.214868    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c"
Jul 21 17:35:59.327551 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:59.327512    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf-proxy-tls\") pod \"error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw\" (UID: \"e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf\") " pod="kserve-ci-e2e-test/error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw"
Jul 21 17:35:59.331454 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:59.331418    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf-proxy-tls\") pod \"error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw\" (UID: \"e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf\") " pod="kserve-ci-e2e-test/error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw"
Jul 21 17:35:59.367666 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:59.367631    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c"]
Jul 21 17:35:59.373800 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:35:59.373767    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e8d920d_8b7d_4536_bae9_593c6439d608.slice/crio-fb2647f9899eede01dd864f412ad3b49f73c88c34a406a0b00f0cb16aa003b55 WatchSource:0}: Error finding container fb2647f9899eede01dd864f412ad3b49f73c88c34a406a0b00f0cb16aa003b55: Status 404 returned error can't find the container with id fb2647f9899eede01dd864f412ad3b49f73c88c34a406a0b00f0cb16aa003b55
Jul 21 17:35:59.530524 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:59.529916    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2w9gs\" (UID: \"8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs"
Jul 21 17:35:59.536021 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:59.535963    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2w9gs\" (UID: \"8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs"
Jul 21 17:35:59.592613 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:59.592267    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw"
Jul 21 17:35:59.710465 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:59.710433    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs"
Jul 21 17:35:59.839083 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:59.838919    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw"]
Jul 21 17:35:59.844811 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:35:59.844694    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9f46c59_d3f7_4ece_9b53_6dcdaacc66cf.slice/crio-8d39c0a67ba586630391bca8bd0a5ae800ac748144edaa278d264ba10cb4a0de WatchSource:0}: Error finding container 8d39c0a67ba586630391bca8bd0a5ae800ac748144edaa278d264ba10cb4a0de: Status 404 returned error can't find the container with id 8d39c0a67ba586630391bca8bd0a5ae800ac748144edaa278d264ba10cb4a0de
Jul 21 17:35:59.925658 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:59.925616    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt" event={"ID":"69e0a43b-e6af-4859-9a8f-872ada531c9d","Type":"ContainerStarted","Data":"64b9a5f1392068f559e848abd403c50994bb9504dbc8fe80cfd6ae2e6764eeb9"}
Jul 21 17:35:59.925823 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:59.925695    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt" event={"ID":"69e0a43b-e6af-4859-9a8f-872ada531c9d","Type":"ContainerStarted","Data":"51febc87b0113be6e4674f6c0a6626117319d2c987b1d004de2b826d1a33cfc5"}
Jul 21 17:35:59.928824 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:59.928739    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw" event={"ID":"e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf","Type":"ContainerStarted","Data":"8d39c0a67ba586630391bca8bd0a5ae800ac748144edaa278d264ba10cb4a0de"}
Jul 21 17:35:59.933287 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:59.932708    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c" event={"ID":"6e8d920d-8b7d-4536-bae9-593c6439d608","Type":"ContainerStarted","Data":"90691b9203ae36f7d65f9657f33e645fa063993776ec1bf23335eab210e8184c"}
Jul 21 17:35:59.933287 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:59.932767    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c" event={"ID":"6e8d920d-8b7d-4536-bae9-593c6439d608","Type":"ContainerStarted","Data":"fb2647f9899eede01dd864f412ad3b49f73c88c34a406a0b00f0cb16aa003b55"}
Jul 21 17:35:59.961625 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:35:59.961599    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs"]
Jul 21 17:35:59.965545 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:35:59.965520    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c5e3cb0_172b_4f1a_8e76_028bbc1c8f10.slice/crio-0bd3ea2da6751d073449f132d8911af69ef21e0a07d55bfee72fad21272e1293 WatchSource:0}: Error finding container 0bd3ea2da6751d073449f132d8911af69ef21e0a07d55bfee72fad21272e1293: Status 404 returned error can't find the container with id 0bd3ea2da6751d073449f132d8911af69ef21e0a07d55bfee72fad21272e1293
Jul 21 17:36:00.947089 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:00.947050    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs" event={"ID":"8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10","Type":"ContainerStarted","Data":"d18a06757e08dc2211708f6aa09f2cac5ff2880d01f657332a4f78687e72397f"}
Jul 21 17:36:00.947570 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:00.947122    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs" event={"ID":"8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10","Type":"ContainerStarted","Data":"0bd3ea2da6751d073449f132d8911af69ef21e0a07d55bfee72fad21272e1293"}
Jul 21 17:36:03.984357 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:03.983354    2563 generic.go:356] "Generic (PLEG): container finished" podID="69e0a43b-e6af-4859-9a8f-872ada531c9d" containerID="64b9a5f1392068f559e848abd403c50994bb9504dbc8fe80cfd6ae2e6764eeb9" exitCode=0
Jul 21 17:36:03.984357 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:03.983512    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt" event={"ID":"69e0a43b-e6af-4859-9a8f-872ada531c9d","Type":"ContainerDied","Data":"64b9a5f1392068f559e848abd403c50994bb9504dbc8fe80cfd6ae2e6764eeb9"}
Jul 21 17:36:03.989709 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:03.988897    2563 generic.go:356] "Generic (PLEG): container finished" podID="6e8d920d-8b7d-4536-bae9-593c6439d608" containerID="90691b9203ae36f7d65f9657f33e645fa063993776ec1bf23335eab210e8184c" exitCode=0
Jul 21 17:36:03.989709 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:03.988928    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c" event={"ID":"6e8d920d-8b7d-4536-bae9-593c6439d608","Type":"ContainerDied","Data":"90691b9203ae36f7d65f9657f33e645fa063993776ec1bf23335eab210e8184c"}
Jul 21 17:36:07.026256 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:07.025179    2563 generic.go:356] "Generic (PLEG): container finished" podID="8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10" containerID="d18a06757e08dc2211708f6aa09f2cac5ff2880d01f657332a4f78687e72397f" exitCode=0
Jul 21 17:36:07.026256 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:07.025273    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs" event={"ID":"8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10","Type":"ContainerDied","Data":"d18a06757e08dc2211708f6aa09f2cac5ff2880d01f657332a4f78687e72397f"}
Jul 21 17:36:23.118041 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:23.118015    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c" event={"ID":"6e8d920d-8b7d-4536-bae9-593c6439d608","Type":"ContainerStarted","Data":"2acee3ed50dc3d42451be0f066ad43026024098831deca3d677a365244920dee"}
Jul 21 17:36:23.119465 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:23.119445    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7" event={"ID":"2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46","Type":"ContainerStarted","Data":"5a7b28908d3d2255b69f1019a816f378c5c4316717fe22ac3d7294ad698b4c1d"}
Jul 21 17:36:23.121100 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:23.121080    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt" event={"ID":"69e0a43b-e6af-4859-9a8f-872ada531c9d","Type":"ContainerStarted","Data":"aa8cc606a24907fa0592f3678ff55d065d8bf1a6fdd487c86626b9ba0f3ecacc"}
Jul 21 17:36:23.122456 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:23.122437    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw" event={"ID":"e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf","Type":"ContainerStarted","Data":"18698faf902a63dde1ec9c210a94489b8d39ad228289a2072bd52d2d2ef37ba0"}
Jul 21 17:36:29.166672 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:29.166621    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw" event={"ID":"e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf","Type":"ContainerStarted","Data":"fd05d6af257c094c2335592006d073f5f62cd6f9fbae9549ea4f6aea36642d79"}
Jul 21 17:36:29.167127 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:29.166732    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw"
Jul 21 17:36:29.169384 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:29.169358    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c" event={"ID":"6e8d920d-8b7d-4536-bae9-593c6439d608","Type":"ContainerStarted","Data":"69afb319b37c95cd8e4771d0ff24eda579a797484e615bb4b1e85e8f04973243"}
Jul 21 17:36:29.169589 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:29.169568    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c"
Jul 21 17:36:29.169668 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:29.169622    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c"
Jul 21 17:36:29.171017 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:29.170981    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c" podUID="6e8d920d-8b7d-4536-bae9-593c6439d608" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 21 17:36:29.171718 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:29.171699    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7" event={"ID":"2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46","Type":"ContainerStarted","Data":"d44c37b448f93e85f46bf91aea29fc4eb7177a1b892a7cc73004be50e266c1f0"}
Jul 21 17:36:29.171910 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:29.171895    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7"
Jul 21 17:36:29.171986 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:29.171944    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7"
Jul 21 17:36:29.172973 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:29.172947    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7" podUID="2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 21 17:36:29.174257 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:29.174240    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt" event={"ID":"69e0a43b-e6af-4859-9a8f-872ada531c9d","Type":"ContainerStarted","Data":"dab00082341dc5087aaa1ffd466d367cddb5fe9b749c3d7ab7803741f0911a0a"}
Jul 21 17:36:29.174403 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:29.174376    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt"
Jul 21 17:36:29.189370 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:29.189334    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw" podStartSLOduration=2.614293959 podStartE2EDuration="31.189321297s" totalImagesPullingTime="28.575027338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:35:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:36:29.187468456 +0000 UTC m=+616.949608061" watchObservedRunningTime="2026-07-21 17:36:29.189321297 +0000 UTC m=+616.951460903"
Jul 21 17:36:29.209704 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:29.209654    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c" podStartSLOduration=2.947203366 podStartE2EDuration="31.20963825s" totalImagesPullingTime="24.440108648s" totalInitContainerRuntime="3.822326236s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:35:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:36:29.207320519 +0000 UTC m=+616.969460139" watchObservedRunningTime="2026-07-21 17:36:29.20963825 +0000 UTC m=+616.971777855"
Jul 21 17:36:29.226354 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:29.226019    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt" podStartSLOduration=2.672374072 podStartE2EDuration="31.226007161s" totalImagesPullingTime="24.391958854s" totalInitContainerRuntime="4.161674235s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:35:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:36:29.225872383 +0000 UTC m=+616.988011991" watchObservedRunningTime="2026-07-21 17:36:29.226007161 +0000 UTC m=+616.988146766"
Jul 21 17:36:29.242732 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:29.242584    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7" podStartSLOduration=1.522049053 podStartE2EDuration="31.242567709s" totalImagesPullingTime="29.720518656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:35:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:36:29.242361048 +0000 UTC m=+617.004500680" watchObservedRunningTime="2026-07-21 17:36:29.242567709 +0000 UTC m=+617.004707314"
Jul 21 17:36:30.179844 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:30.179798    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7" podUID="2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 21 17:36:30.180341 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:30.179957    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c" podUID="6e8d920d-8b7d-4536-bae9-593c6439d608" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 21 17:36:30.180653 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:30.180625    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt"
Jul 21 17:36:30.180768 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:30.180685    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw"
Jul 21 17:36:30.181436 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:30.181415    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt" podUID="69e0a43b-e6af-4859-9a8f-872ada531c9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 21 17:36:30.181675 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:30.181648    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw" podUID="e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 21 17:36:31.185601 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:31.185558    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw" podUID="e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 21 17:36:31.186020 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:31.185669    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt" podUID="69e0a43b-e6af-4859-9a8f-872ada531c9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 21 17:36:35.185162 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:35.185063    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c"
Jul 21 17:36:35.185777 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:35.185754    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7"
Jul 21 17:36:35.185899 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:35.185827    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c" podUID="6e8d920d-8b7d-4536-bae9-593c6439d608" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 21 17:36:35.186261 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:35.186237    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7" podUID="2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 21 17:36:36.190939 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:36.190903    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt"
Jul 21 17:36:36.191520 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:36.191492    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw"
Jul 21 17:36:36.191648 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:36.191551    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt" podUID="69e0a43b-e6af-4859-9a8f-872ada531c9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 21 17:36:36.192003 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:36.191977    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw" podUID="e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 21 17:36:44.244510 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:44.244474    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs" event={"ID":"8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10","Type":"ContainerStarted","Data":"8e10c815cb3e2ddd48a8a1757dee0d9850672d03ddebcb43d86d2aecfa40b0ff"}
Jul 21 17:36:44.244947 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:44.244532    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs" event={"ID":"8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10","Type":"ContainerStarted","Data":"42f0234e5d9226b61019bd27f73a9dc15d5ab576be1472113027d603d3d426ce"}
Jul 21 17:36:44.244947 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:44.244714    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs"
Jul 21 17:36:44.264616 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:44.264566    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs" podStartSLOduration=5.467840891 podStartE2EDuration="46.264548805s" totalImagesPullingTime="36.615097719s" totalInitContainerRuntime="4.181610195s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:35:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:36:44.26284221 +0000 UTC m=+632.024981836" watchObservedRunningTime="2026-07-21 17:36:44.264548805 +0000 UTC m=+632.026688409"
Jul 21 17:36:45.186042 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:45.186001    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c" podUID="6e8d920d-8b7d-4536-bae9-593c6439d608" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 21 17:36:45.191058 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:45.186573    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7" podUID="2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 21 17:36:45.249646 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:45.249523    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs"
Jul 21 17:36:45.250906 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:45.250858    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs" podUID="8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 21 17:36:46.191804 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:46.191764    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt" podUID="69e0a43b-e6af-4859-9a8f-872ada531c9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 21 17:36:46.192049 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:46.191932    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw" podUID="e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 21 17:36:46.254680 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:46.254640    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs" podUID="8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 21 17:36:51.260093 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:51.260063    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs"
Jul 21 17:36:51.260550 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:51.260520    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs" podUID="8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 21 17:36:55.186318 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:55.186276    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7" podUID="2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 21 17:36:55.186318 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:55.186293    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c" podUID="6e8d920d-8b7d-4536-bae9-593c6439d608" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 21 17:36:56.191968 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:56.191931    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw" podUID="e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 21 17:36:56.192364 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:36:56.191963    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt" podUID="69e0a43b-e6af-4859-9a8f-872ada531c9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 21 17:37:01.261360 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:01.261315    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs" podUID="8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 21 17:37:05.185858 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:05.185814    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c" podUID="6e8d920d-8b7d-4536-bae9-593c6439d608" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 21 17:37:05.186285 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:05.186265    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7" podUID="2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 21 17:37:06.191449 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:06.191406    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt" podUID="69e0a43b-e6af-4859-9a8f-872ada531c9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 21 17:37:06.192629 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:06.192605    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw" podUID="e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 21 17:37:11.261479 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:11.261436    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs" podUID="8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 21 17:37:15.186636 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:15.186591    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c" podUID="6e8d920d-8b7d-4536-bae9-593c6439d608" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 21 17:37:15.187050 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:15.186591    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7" podUID="2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 21 17:37:16.192048 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:16.192009    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt" podUID="69e0a43b-e6af-4859-9a8f-872ada531c9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 21 17:37:16.192614 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:16.192597    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw"
Jul 21 17:37:21.260465 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:21.260429    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs" podUID="8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 21 17:37:25.186624 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:25.186579    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c" podUID="6e8d920d-8b7d-4536-bae9-593c6439d608" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 21 17:37:25.187152 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:25.187116    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7"
Jul 21 17:37:26.191562 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:26.191523    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt" podUID="69e0a43b-e6af-4859-9a8f-872ada531c9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 21 17:37:31.260571 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:31.260531    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs" podUID="8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 21 17:37:35.186370 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:35.186332    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c" podUID="6e8d920d-8b7d-4536-bae9-593c6439d608" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 21 17:37:36.191486 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:36.191439    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt" podUID="69e0a43b-e6af-4859-9a8f-872ada531c9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 21 17:37:41.261332 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:41.261297    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs" podUID="8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 21 17:37:43.759601 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:43.759558    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c" podUID="6e8d920d-8b7d-4536-bae9-593c6439d608" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 21 17:37:46.191884 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:46.191856    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt"
Jul 21 17:37:48.580000 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:48.579964    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7"]
Jul 21 17:37:48.580433 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:48.580246    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7" podUID="2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46" containerName="kserve-container" containerID="cri-o://5a7b28908d3d2255b69f1019a816f378c5c4316717fe22ac3d7294ad698b4c1d" gracePeriod=30
Jul 21 17:37:48.580433 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:48.580256    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7" podUID="2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46" containerName="kube-rbac-proxy" containerID="cri-o://d44c37b448f93e85f46bf91aea29fc4eb7177a1b892a7cc73004be50e266c1f0" gracePeriod=30
Jul 21 17:37:48.687833 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:48.687801    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6"]
Jul 21 17:37:48.690848 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:48.690784    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6"
Jul 21 17:37:48.693811 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:48.693781    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-5bf12-predictor-serving-cert\""
Jul 21 17:37:48.693950 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:48.693784    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-5bf12-kube-rbac-proxy-sar-config\""
Jul 21 17:37:48.700230 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:48.700207    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw"]
Jul 21 17:37:48.700432 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:48.700396    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw" podUID="e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf" containerName="kserve-container" containerID="cri-o://18698faf902a63dde1ec9c210a94489b8d39ad228289a2072bd52d2d2ef37ba0" gracePeriod=30
Jul 21 17:37:48.700651 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:48.700489    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw" podUID="e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf" containerName="kube-rbac-proxy" containerID="cri-o://fd05d6af257c094c2335592006d073f5f62cd6f9fbae9549ea4f6aea36642d79" gracePeriod=30
Jul 21 17:37:48.703049 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:48.703029    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6"]
Jul 21 17:37:48.791819 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:48.791782    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l"]
Jul 21 17:37:48.794234 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:48.794215    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l"
Jul 21 17:37:48.796504 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:48.796487    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-5bf12-predictor-serving-cert\""
Jul 21 17:37:48.796504 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:48.796495    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-5bf12-kube-rbac-proxy-sar-config\""
Jul 21 17:37:48.805688 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:48.805658    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l"]
Jul 21 17:37:48.832093 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:48.832021    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aeeeed18-d5cb-44dc-98d8-e896038dee1b-proxy-tls\") pod \"success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6\" (UID: \"aeeeed18-d5cb-44dc-98d8-e896038dee1b\") " pod="kserve-ci-e2e-test/success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6"
Jul 21 17:37:48.832093 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:48.832066    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aeeeed18-d5cb-44dc-98d8-e896038dee1b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6\" (UID: \"aeeeed18-d5cb-44dc-98d8-e896038dee1b\") " pod="kserve-ci-e2e-test/success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6"
Jul 21 17:37:48.832311 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:48.832167    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smrs6\" (UniqueName: \"kubernetes.io/projected/aeeeed18-d5cb-44dc-98d8-e896038dee1b-kube-api-access-smrs6\") pod \"success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6\" (UID: \"aeeeed18-d5cb-44dc-98d8-e896038dee1b\") " pod="kserve-ci-e2e-test/success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6"
Jul 21 17:37:48.933482 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:48.933449    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aeeeed18-d5cb-44dc-98d8-e896038dee1b-proxy-tls\") pod \"success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6\" (UID: \"aeeeed18-d5cb-44dc-98d8-e896038dee1b\") " pod="kserve-ci-e2e-test/success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6"
Jul 21 17:37:48.933482 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:48.933483    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aeeeed18-d5cb-44dc-98d8-e896038dee1b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6\" (UID: \"aeeeed18-d5cb-44dc-98d8-e896038dee1b\") " pod="kserve-ci-e2e-test/success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6"
Jul 21 17:37:48.933703 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:48.933507    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28mnb\" (UniqueName: \"kubernetes.io/projected/ef4ef854-bdcb-419b-b4a7-ae0f4caedc69-kube-api-access-28mnb\") pod \"error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l\" (UID: \"ef4ef854-bdcb-419b-b4a7-ae0f4caedc69\") " pod="kserve-ci-e2e-test/error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l"
Jul 21 17:37:48.933703 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:48.933550    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smrs6\" (UniqueName: \"kubernetes.io/projected/aeeeed18-d5cb-44dc-98d8-e896038dee1b-kube-api-access-smrs6\") pod \"success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6\" (UID: \"aeeeed18-d5cb-44dc-98d8-e896038dee1b\") " pod="kserve-ci-e2e-test/success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6"
Jul 21 17:37:48.933703 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:48.933581    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef4ef854-bdcb-419b-b4a7-ae0f4caedc69-proxy-tls\") pod \"error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l\" (UID: \"ef4ef854-bdcb-419b-b4a7-ae0f4caedc69\") " pod="kserve-ci-e2e-test/error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l"
Jul 21 17:37:48.933703 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:48.933605    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef4ef854-bdcb-419b-b4a7-ae0f4caedc69-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l\" (UID: \"ef4ef854-bdcb-419b-b4a7-ae0f4caedc69\") " pod="kserve-ci-e2e-test/error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l"
Jul 21 17:37:48.934181 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:48.934161    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aeeeed18-d5cb-44dc-98d8-e896038dee1b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6\" (UID: \"aeeeed18-d5cb-44dc-98d8-e896038dee1b\") " pod="kserve-ci-e2e-test/success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6"
Jul 21 17:37:48.935963 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:48.935942    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aeeeed18-d5cb-44dc-98d8-e896038dee1b-proxy-tls\") pod \"success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6\" (UID: \"aeeeed18-d5cb-44dc-98d8-e896038dee1b\") " pod="kserve-ci-e2e-test/success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6"
Jul 21 17:37:48.941501 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:48.941475    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smrs6\" (UniqueName: \"kubernetes.io/projected/aeeeed18-d5cb-44dc-98d8-e896038dee1b-kube-api-access-smrs6\") pod \"success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6\" (UID: \"aeeeed18-d5cb-44dc-98d8-e896038dee1b\") " pod="kserve-ci-e2e-test/success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6"
Jul 21 17:37:49.003917 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:49.003884    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6"
Jul 21 17:37:49.034109 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:49.034080    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef4ef854-bdcb-419b-b4a7-ae0f4caedc69-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l\" (UID: \"ef4ef854-bdcb-419b-b4a7-ae0f4caedc69\") " pod="kserve-ci-e2e-test/error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l"
Jul 21 17:37:49.034261 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:49.034175    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28mnb\" (UniqueName: \"kubernetes.io/projected/ef4ef854-bdcb-419b-b4a7-ae0f4caedc69-kube-api-access-28mnb\") pod \"error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l\" (UID: \"ef4ef854-bdcb-419b-b4a7-ae0f4caedc69\") " pod="kserve-ci-e2e-test/error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l"
Jul 21 17:37:49.034261 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:49.034252    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef4ef854-bdcb-419b-b4a7-ae0f4caedc69-proxy-tls\") pod \"error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l\" (UID: \"ef4ef854-bdcb-419b-b4a7-ae0f4caedc69\") " pod="kserve-ci-e2e-test/error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l"
Jul 21 17:37:49.034371 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:37:49.034356    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-5bf12-predictor-serving-cert: secret "error-404-isvc-5bf12-predictor-serving-cert" not found
Jul 21 17:37:49.034439 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:37:49.034428    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef4ef854-bdcb-419b-b4a7-ae0f4caedc69-proxy-tls podName:ef4ef854-bdcb-419b-b4a7-ae0f4caedc69 nodeName:}" failed. No retries permitted until 2026-07-21 17:37:49.534405399 +0000 UTC m=+697.296544981 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ef4ef854-bdcb-419b-b4a7-ae0f4caedc69-proxy-tls") pod "error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l" (UID: "ef4ef854-bdcb-419b-b4a7-ae0f4caedc69") : secret "error-404-isvc-5bf12-predictor-serving-cert" not found
Jul 21 17:37:49.034821 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:49.034803    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef4ef854-bdcb-419b-b4a7-ae0f4caedc69-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l\" (UID: \"ef4ef854-bdcb-419b-b4a7-ae0f4caedc69\") " pod="kserve-ci-e2e-test/error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l"
Jul 21 17:37:49.042963 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:49.042931    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28mnb\" (UniqueName: \"kubernetes.io/projected/ef4ef854-bdcb-419b-b4a7-ae0f4caedc69-kube-api-access-28mnb\") pod \"error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l\" (UID: \"ef4ef854-bdcb-419b-b4a7-ae0f4caedc69\") " pod="kserve-ci-e2e-test/error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l"
Jul 21 17:37:49.141348 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:49.141291    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6"]
Jul 21 17:37:49.143953 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:37:49.143925    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeeeed18_d5cb_44dc_98d8_e896038dee1b.slice/crio-1c17c9a54f749a2ce9b820778c1b467b0e65ac7455e4674721dfa4044c73a8f0 WatchSource:0}: Error finding container 1c17c9a54f749a2ce9b820778c1b467b0e65ac7455e4674721dfa4044c73a8f0: Status 404 returned error can't find the container with id 1c17c9a54f749a2ce9b820778c1b467b0e65ac7455e4674721dfa4044c73a8f0
Jul 21 17:37:49.511180 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:49.511125    2563 generic.go:356] "Generic (PLEG): container finished" podID="e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf" containerID="fd05d6af257c094c2335592006d073f5f62cd6f9fbae9549ea4f6aea36642d79" exitCode=2
Jul 21 17:37:49.511360 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:49.511202    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw" event={"ID":"e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf","Type":"ContainerDied","Data":"fd05d6af257c094c2335592006d073f5f62cd6f9fbae9549ea4f6aea36642d79"}
Jul 21 17:37:49.512826 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:49.512803    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6" event={"ID":"aeeeed18-d5cb-44dc-98d8-e896038dee1b","Type":"ContainerStarted","Data":"3d2f7a01abcb50f5228bfb1aca202d30140d7ce4bb45386193a9cc46910537b8"}
Jul 21 17:37:49.512968 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:49.512847    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6" event={"ID":"aeeeed18-d5cb-44dc-98d8-e896038dee1b","Type":"ContainerStarted","Data":"7452dbc7929fea9bda0fd1f58e072e7bd11e89eaaf2631a1ad86999c16539f9a"}
Jul 21 17:37:49.512968 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:49.512873    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6"
Jul 21 17:37:49.512968 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:49.512892    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6" event={"ID":"aeeeed18-d5cb-44dc-98d8-e896038dee1b","Type":"ContainerStarted","Data":"1c17c9a54f749a2ce9b820778c1b467b0e65ac7455e4674721dfa4044c73a8f0"}
Jul 21 17:37:49.512968 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:49.512912    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6"
Jul 21 17:37:49.514364 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:49.514338    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6" podUID="aeeeed18-d5cb-44dc-98d8-e896038dee1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 21 17:37:49.514589 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:49.514569    2563 generic.go:356] "Generic (PLEG): container finished" podID="2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46" containerID="d44c37b448f93e85f46bf91aea29fc4eb7177a1b892a7cc73004be50e266c1f0" exitCode=2
Jul 21 17:37:49.514652 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:49.514605    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7" event={"ID":"2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46","Type":"ContainerDied","Data":"d44c37b448f93e85f46bf91aea29fc4eb7177a1b892a7cc73004be50e266c1f0"}
Jul 21 17:37:49.533341 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:49.533295    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6" podStartSLOduration=1.533280022 podStartE2EDuration="1.533280022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:37:49.531388429 +0000 UTC m=+697.293528043" watchObservedRunningTime="2026-07-21 17:37:49.533280022 +0000 UTC m=+697.295419693"
Jul 21 17:37:49.538192 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:49.538171    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef4ef854-bdcb-419b-b4a7-ae0f4caedc69-proxy-tls\") pod \"error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l\" (UID: \"ef4ef854-bdcb-419b-b4a7-ae0f4caedc69\") " pod="kserve-ci-e2e-test/error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l"
Jul 21 17:37:49.540863 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:49.540845    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef4ef854-bdcb-419b-b4a7-ae0f4caedc69-proxy-tls\") pod \"error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l\" (UID: \"ef4ef854-bdcb-419b-b4a7-ae0f4caedc69\") " pod="kserve-ci-e2e-test/error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l"
Jul 21 17:37:49.704308 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:49.704278    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l"
Jul 21 17:37:49.838889 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:49.838865    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l"]
Jul 21 17:37:49.841239 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:37:49.841215    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef4ef854_bdcb_419b_b4a7_ae0f4caedc69.slice/crio-041d53034e1e26a85cf3a6748fc2d64931da6119aa4d34294b433dafca0a16c1 WatchSource:0}: Error finding container 041d53034e1e26a85cf3a6748fc2d64931da6119aa4d34294b433dafca0a16c1: Status 404 returned error can't find the container with id 041d53034e1e26a85cf3a6748fc2d64931da6119aa4d34294b433dafca0a16c1
Jul 21 17:37:50.180290 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:50.180251    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7" podUID="2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.18:8643/healthz\": dial tcp 10.133.0.18:8643: connect: connection refused"
Jul 21 17:37:50.520890 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:50.520802    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l" event={"ID":"ef4ef854-bdcb-419b-b4a7-ae0f4caedc69","Type":"ContainerStarted","Data":"5c511b41ad09bf8a6fd450fb0ac8b0bf4865195f94e08c5fbd61021ccbf44465"}
Jul 21 17:37:50.521057 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:50.521008    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l"
Jul 21 17:37:50.521157 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:50.521055    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l" event={"ID":"ef4ef854-bdcb-419b-b4a7-ae0f4caedc69","Type":"ContainerStarted","Data":"244a840e829c5d11f54b697ef871553652af84ba2102dc15597cfd47ae486faf"}
Jul 21 17:37:50.521157 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:50.521092    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l" event={"ID":"ef4ef854-bdcb-419b-b4a7-ae0f4caedc69","Type":"ContainerStarted","Data":"041d53034e1e26a85cf3a6748fc2d64931da6119aa4d34294b433dafca0a16c1"}
Jul 21 17:37:50.521277 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:50.521157    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l"
Jul 21 17:37:50.521277 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:50.521208    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6" podUID="aeeeed18-d5cb-44dc-98d8-e896038dee1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 21 17:37:50.522198 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:50.522176    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l" podUID="ef4ef854-bdcb-419b-b4a7-ae0f4caedc69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 21 17:37:50.544252 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:50.544207    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l" podStartSLOduration=2.544194052 podStartE2EDuration="2.544194052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:37:50.54265608 +0000 UTC m=+698.304795684" watchObservedRunningTime="2026-07-21 17:37:50.544194052 +0000 UTC m=+698.306333657"
Jul 21 17:37:51.186183 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:51.186108    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw" podUID="e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.20:8643/healthz\": dial tcp 10.133.0.20:8643: connect: connection refused"
Jul 21 17:37:51.261319 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:51.261290    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs"
Jul 21 17:37:51.525577 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:51.525488    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l" podUID="ef4ef854-bdcb-419b-b4a7-ae0f4caedc69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 21 17:37:52.227743 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:52.227719    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7"
Jul 21 17:37:52.364080 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:52.363994    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46-kube-rbac-proxy-sar-config\") pod \"2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46\" (UID: \"2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46\") "
Jul 21 17:37:52.364080 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:52.364049    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46-proxy-tls\") pod \"2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46\" (UID: \"2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46\") "
Jul 21 17:37:52.364313 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:52.364086    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blc88\" (UniqueName: \"kubernetes.io/projected/2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46-kube-api-access-blc88\") pod \"2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46\" (UID: \"2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46\") "
Jul 21 17:37:52.364362 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:52.364343    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46" (UID: "2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:37:52.366373 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:52.366345    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46" (UID: "2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:37:52.366373 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:52.366361    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46-kube-api-access-blc88" (OuterVolumeSpecName: "kube-api-access-blc88") pod "2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46" (UID: "2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46"). InnerVolumeSpecName "kube-api-access-blc88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:37:52.465017 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:52.464978    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46-proxy-tls\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:37:52.465017 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:52.465006    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blc88\" (UniqueName: \"kubernetes.io/projected/2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46-kube-api-access-blc88\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:37:52.465017 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:52.465017    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:37:52.531529 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:52.531493    2563 generic.go:356] "Generic (PLEG): container finished" podID="e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf" containerID="18698faf902a63dde1ec9c210a94489b8d39ad228289a2072bd52d2d2ef37ba0" exitCode=0
Jul 21 17:37:52.531655 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:52.531557    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw" event={"ID":"e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf","Type":"ContainerDied","Data":"18698faf902a63dde1ec9c210a94489b8d39ad228289a2072bd52d2d2ef37ba0"}
Jul 21 17:37:52.533359 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:52.533336    2563 generic.go:356] "Generic (PLEG): container finished" podID="2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46" containerID="5a7b28908d3d2255b69f1019a816f378c5c4316717fe22ac3d7294ad698b4c1d" exitCode=0
Jul 21 17:37:52.533473 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:52.533384    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7"
Jul 21 17:37:52.533473 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:52.533418    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7" event={"ID":"2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46","Type":"ContainerDied","Data":"5a7b28908d3d2255b69f1019a816f378c5c4316717fe22ac3d7294ad698b4c1d"}
Jul 21 17:37:52.533577 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:52.533483    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7" event={"ID":"2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46","Type":"ContainerDied","Data":"6b59d56bb75c2701deeee32c965f774195f9566094d03fadd4f9d3c8a7ee3b1e"}
Jul 21 17:37:52.533577 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:52.533524    2563 scope.go:117] "RemoveContainer" containerID="d44c37b448f93e85f46bf91aea29fc4eb7177a1b892a7cc73004be50e266c1f0"
Jul 21 17:37:52.543045 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:52.543029    2563 scope.go:117] "RemoveContainer" containerID="5a7b28908d3d2255b69f1019a816f378c5c4316717fe22ac3d7294ad698b4c1d"
Jul 21 17:37:52.551919 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:52.551899    2563 scope.go:117] "RemoveContainer" containerID="d44c37b448f93e85f46bf91aea29fc4eb7177a1b892a7cc73004be50e266c1f0"
Jul 21 17:37:52.552272 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:37:52.552253    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d44c37b448f93e85f46bf91aea29fc4eb7177a1b892a7cc73004be50e266c1f0\": container with ID starting with d44c37b448f93e85f46bf91aea29fc4eb7177a1b892a7cc73004be50e266c1f0 not found: ID does not exist" containerID="d44c37b448f93e85f46bf91aea29fc4eb7177a1b892a7cc73004be50e266c1f0"
Jul 21 17:37:52.552356 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:52.552283    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d44c37b448f93e85f46bf91aea29fc4eb7177a1b892a7cc73004be50e266c1f0"} err="failed to get container status \"d44c37b448f93e85f46bf91aea29fc4eb7177a1b892a7cc73004be50e266c1f0\": rpc error: code = NotFound desc = could not find container \"d44c37b448f93e85f46bf91aea29fc4eb7177a1b892a7cc73004be50e266c1f0\": container with ID starting with d44c37b448f93e85f46bf91aea29fc4eb7177a1b892a7cc73004be50e266c1f0 not found: ID does not exist"
Jul 21 17:37:52.552356 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:52.552308    2563 scope.go:117] "RemoveContainer" containerID="5a7b28908d3d2255b69f1019a816f378c5c4316717fe22ac3d7294ad698b4c1d"
Jul 21 17:37:52.552623 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:37:52.552603    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a7b28908d3d2255b69f1019a816f378c5c4316717fe22ac3d7294ad698b4c1d\": container with ID starting with 5a7b28908d3d2255b69f1019a816f378c5c4316717fe22ac3d7294ad698b4c1d not found: ID does not exist" containerID="5a7b28908d3d2255b69f1019a816f378c5c4316717fe22ac3d7294ad698b4c1d"
Jul 21 17:37:52.552679 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:52.552640    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a7b28908d3d2255b69f1019a816f378c5c4316717fe22ac3d7294ad698b4c1d"} err="failed to get container status \"5a7b28908d3d2255b69f1019a816f378c5c4316717fe22ac3d7294ad698b4c1d\": rpc error: code = NotFound desc = could not find container \"5a7b28908d3d2255b69f1019a816f378c5c4316717fe22ac3d7294ad698b4c1d\": container with ID starting with 5a7b28908d3d2255b69f1019a816f378c5c4316717fe22ac3d7294ad698b4c1d not found: ID does not exist"
Jul 21 17:37:52.560925 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:52.560900    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7"]
Jul 21 17:37:52.566339 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:52.566313    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4dfe2-predictor-cf754f6b9-58tp7"]
Jul 21 17:37:52.633219 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:52.633197    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw"
Jul 21 17:37:52.764496 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:52.764462    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46" path="/var/lib/kubelet/pods/2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46/volumes"
Jul 21 17:37:52.767373 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:52.767344    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf-kube-rbac-proxy-sar-config\") pod \"e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf\" (UID: \"e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf\") "
Jul 21 17:37:52.767498 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:52.767388    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf-proxy-tls\") pod \"e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf\" (UID: \"e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf\") "
Jul 21 17:37:52.767498 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:52.767458    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rgc9m\" (UniqueName: \"kubernetes.io/projected/e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf-kube-api-access-rgc9m\") pod \"e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf\" (UID: \"e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf\") "
Jul 21 17:37:52.767655 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:52.767635    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf" (UID: "e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:37:52.769456 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:52.769432    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf" (UID: "e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:37:52.769552 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:52.769495    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf-kube-api-access-rgc9m" (OuterVolumeSpecName: "kube-api-access-rgc9m") pod "e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf" (UID: "e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf"). InnerVolumeSpecName "kube-api-access-rgc9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:37:52.868027 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:52.867992    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgc9m\" (UniqueName: \"kubernetes.io/projected/e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf-kube-api-access-rgc9m\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:37:52.868027 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:52.868021    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:37:52.868027 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:52.868031    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf-proxy-tls\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:37:53.539128 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:53.539100    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw"
Jul 21 17:37:53.539128 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:53.539116    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw" event={"ID":"e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf","Type":"ContainerDied","Data":"8d39c0a67ba586630391bca8bd0a5ae800ac748144edaa278d264ba10cb4a0de"}
Jul 21 17:37:53.539639 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:53.539207    2563 scope.go:117] "RemoveContainer" containerID="fd05d6af257c094c2335592006d073f5f62cd6f9fbae9549ea4f6aea36642d79"
Jul 21 17:37:53.548271 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:53.548253    2563 scope.go:117] "RemoveContainer" containerID="18698faf902a63dde1ec9c210a94489b8d39ad228289a2072bd52d2d2ef37ba0"
Jul 21 17:37:53.562344 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:53.562322    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw"]
Jul 21 17:37:53.566634 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:53.566615    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4dfe2-predictor-65cc4f769b-6whlw"]
Jul 21 17:37:53.761056 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:53.761024    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c"
Jul 21 17:37:54.764221 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:54.764194    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf" path="/var/lib/kubelet/pods/e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf/volumes"
Jul 21 17:37:55.530104 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:55.530079    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6"
Jul 21 17:37:55.530689 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:55.530662    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6" podUID="aeeeed18-d5cb-44dc-98d8-e896038dee1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 21 17:37:56.529837 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:56.529812    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l"
Jul 21 17:37:56.530310 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:37:56.530286    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l" podUID="ef4ef854-bdcb-419b-b4a7-ae0f4caedc69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 21 17:38:05.531109 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:05.531025    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6" podUID="aeeeed18-d5cb-44dc-98d8-e896038dee1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 21 17:38:06.531003 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:06.530963    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l" podUID="ef4ef854-bdcb-419b-b4a7-ae0f4caedc69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 21 17:38:15.530962 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:15.530918    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6" podUID="aeeeed18-d5cb-44dc-98d8-e896038dee1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 21 17:38:16.530752 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:16.530701    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l" podUID="ef4ef854-bdcb-419b-b4a7-ae0f4caedc69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 21 17:38:18.461340 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.461309    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c"]
Jul 21 17:38:18.461739 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.461579    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c" podUID="6e8d920d-8b7d-4536-bae9-593c6439d608" containerName="kserve-container" containerID="cri-o://2acee3ed50dc3d42451be0f066ad43026024098831deca3d677a365244920dee" gracePeriod=30
Jul 21 17:38:18.461739 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.461615    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c" podUID="6e8d920d-8b7d-4536-bae9-593c6439d608" containerName="kube-rbac-proxy" containerID="cri-o://69afb319b37c95cd8e4771d0ff24eda579a797484e615bb4b1e85e8f04973243" gracePeriod=30
Jul 21 17:38:18.511783 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.511753    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt"]
Jul 21 17:38:18.512041 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.512016    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt" podUID="69e0a43b-e6af-4859-9a8f-872ada531c9d" containerName="kserve-container" containerID="cri-o://aa8cc606a24907fa0592f3678ff55d065d8bf1a6fdd487c86626b9ba0f3ecacc" gracePeriod=30
Jul 21 17:38:18.512109 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.512057    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt" podUID="69e0a43b-e6af-4859-9a8f-872ada531c9d" containerName="kube-rbac-proxy" containerID="cri-o://dab00082341dc5087aaa1ffd466d367cddb5fe9b749c3d7ab7803741f0911a0a" gracePeriod=30
Jul 21 17:38:18.553262 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.553232    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm"]
Jul 21 17:38:18.553765 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.553743    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46" containerName="kube-rbac-proxy"
Jul 21 17:38:18.553814 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.553771    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46" containerName="kube-rbac-proxy"
Jul 21 17:38:18.553814 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.553786    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf" containerName="kube-rbac-proxy"
Jul 21 17:38:18.553814 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.553795    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf" containerName="kube-rbac-proxy"
Jul 21 17:38:18.553814 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.553811    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf" containerName="kserve-container"
Jul 21 17:38:18.553955 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.553820    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf" containerName="kserve-container"
Jul 21 17:38:18.553955 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.553854    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46" containerName="kserve-container"
Jul 21 17:38:18.553955 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.553863    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46" containerName="kserve-container"
Jul 21 17:38:18.553955 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.553931    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf" containerName="kserve-container"
Jul 21 17:38:18.553955 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.553945    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46" containerName="kserve-container"
Jul 21 17:38:18.554112 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.553955    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ecdb2de-9c9b-4548-bc44-c9cc92f1ec46" containerName="kube-rbac-proxy"
Jul 21 17:38:18.554112 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.553966    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9f46c59-d3f7-4ece-9b53-6dcdaacc66cf" containerName="kube-rbac-proxy"
Jul 21 17:38:18.558755 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.558737    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm"
Jul 21 17:38:18.561342 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.561322    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8558d-kube-rbac-proxy-sar-config\""
Jul 21 17:38:18.561342 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.561323    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8558d-predictor-serving-cert\""
Jul 21 17:38:18.567688 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.567666    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs"]
Jul 21 17:38:18.568151 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.567977    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs" podUID="8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10" containerName="kserve-container" containerID="cri-o://42f0234e5d9226b61019bd27f73a9dc15d5ab576be1472113027d603d3d426ce" gracePeriod=30
Jul 21 17:38:18.568151 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.567981    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs" podUID="8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10" containerName="kube-rbac-proxy" containerID="cri-o://8e10c815cb3e2ddd48a8a1757dee0d9850672d03ddebcb43d86d2aecfa40b0ff" gracePeriod=30
Jul 21 17:38:18.569658 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.569510    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm"]
Jul 21 17:38:18.578930 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.578904    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9de3561f-aecd-4d01-afa8-60277198d9d1-proxy-tls\") pod \"success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm\" (UID: \"9de3561f-aecd-4d01-afa8-60277198d9d1\") " pod="kserve-ci-e2e-test/success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm"
Jul 21 17:38:18.579024 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.578952    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wjbt\" (UniqueName: \"kubernetes.io/projected/9de3561f-aecd-4d01-afa8-60277198d9d1-kube-api-access-7wjbt\") pod \"success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm\" (UID: \"9de3561f-aecd-4d01-afa8-60277198d9d1\") " pod="kserve-ci-e2e-test/success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm"
Jul 21 17:38:18.579153 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.579045    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9de3561f-aecd-4d01-afa8-60277198d9d1-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm\" (UID: \"9de3561f-aecd-4d01-afa8-60277198d9d1\") " pod="kserve-ci-e2e-test/success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm"
Jul 21 17:38:18.645267 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.645237    2563 generic.go:356] "Generic (PLEG): container finished" podID="6e8d920d-8b7d-4536-bae9-593c6439d608" containerID="69afb319b37c95cd8e4771d0ff24eda579a797484e615bb4b1e85e8f04973243" exitCode=2
Jul 21 17:38:18.645409 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.645320    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c" event={"ID":"6e8d920d-8b7d-4536-bae9-593c6439d608","Type":"ContainerDied","Data":"69afb319b37c95cd8e4771d0ff24eda579a797484e615bb4b1e85e8f04973243"}
Jul 21 17:38:18.647477 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.647451    2563 generic.go:356] "Generic (PLEG): container finished" podID="69e0a43b-e6af-4859-9a8f-872ada531c9d" containerID="dab00082341dc5087aaa1ffd466d367cddb5fe9b749c3d7ab7803741f0911a0a" exitCode=2
Jul 21 17:38:18.647626 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.647538    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt" event={"ID":"69e0a43b-e6af-4859-9a8f-872ada531c9d","Type":"ContainerDied","Data":"dab00082341dc5087aaa1ffd466d367cddb5fe9b749c3d7ab7803741f0911a0a"}
Jul 21 17:38:18.654484 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.654456    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8558d-predictor-5594879d74-cksjh"]
Jul 21 17:38:18.658521 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.658504    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8558d-predictor-5594879d74-cksjh"
Jul 21 17:38:18.660889 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.660863    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8558d-kube-rbac-proxy-sar-config\""
Jul 21 17:38:18.661002 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.660907    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8558d-predictor-serving-cert\""
Jul 21 17:38:18.668583 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.668483    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8558d-predictor-5594879d74-cksjh"]
Jul 21 17:38:18.679764 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.679741    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9de3561f-aecd-4d01-afa8-60277198d9d1-proxy-tls\") pod \"success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm\" (UID: \"9de3561f-aecd-4d01-afa8-60277198d9d1\") " pod="kserve-ci-e2e-test/success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm"
Jul 21 17:38:18.679872 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.679781    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wjbt\" (UniqueName: \"kubernetes.io/projected/9de3561f-aecd-4d01-afa8-60277198d9d1-kube-api-access-7wjbt\") pod \"success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm\" (UID: \"9de3561f-aecd-4d01-afa8-60277198d9d1\") " pod="kserve-ci-e2e-test/success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm"
Jul 21 17:38:18.679872 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.679829    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914-proxy-tls\") pod \"error-404-isvc-8558d-predictor-5594879d74-cksjh\" (UID: \"0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914\") " pod="kserve-ci-e2e-test/error-404-isvc-8558d-predictor-5594879d74-cksjh"
Jul 21 17:38:18.679968 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.679871    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8558d-predictor-5594879d74-cksjh\" (UID: \"0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914\") " pod="kserve-ci-e2e-test/error-404-isvc-8558d-predictor-5594879d74-cksjh"
Jul 21 17:38:18.679968 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.679899    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6xt6\" (UniqueName: \"kubernetes.io/projected/0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914-kube-api-access-v6xt6\") pod \"error-404-isvc-8558d-predictor-5594879d74-cksjh\" (UID: \"0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914\") " pod="kserve-ci-e2e-test/error-404-isvc-8558d-predictor-5594879d74-cksjh"
Jul 21 17:38:18.679968 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:38:18.679909    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-8558d-predictor-serving-cert: secret "success-200-isvc-8558d-predictor-serving-cert" not found
Jul 21 17:38:18.680079 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.679963    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9de3561f-aecd-4d01-afa8-60277198d9d1-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm\" (UID: \"9de3561f-aecd-4d01-afa8-60277198d9d1\") " pod="kserve-ci-e2e-test/success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm"
Jul 21 17:38:18.680079 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:38:18.679987    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9de3561f-aecd-4d01-afa8-60277198d9d1-proxy-tls podName:9de3561f-aecd-4d01-afa8-60277198d9d1 nodeName:}" failed. No retries permitted until 2026-07-21 17:38:19.179966442 +0000 UTC m=+726.942106024 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9de3561f-aecd-4d01-afa8-60277198d9d1-proxy-tls") pod "success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm" (UID: "9de3561f-aecd-4d01-afa8-60277198d9d1") : secret "success-200-isvc-8558d-predictor-serving-cert" not found
Jul 21 17:38:18.680501 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.680483    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9de3561f-aecd-4d01-afa8-60277198d9d1-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm\" (UID: \"9de3561f-aecd-4d01-afa8-60277198d9d1\") " pod="kserve-ci-e2e-test/success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm"
Jul 21 17:38:18.691546 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.691485    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wjbt\" (UniqueName: \"kubernetes.io/projected/9de3561f-aecd-4d01-afa8-60277198d9d1-kube-api-access-7wjbt\") pod \"success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm\" (UID: \"9de3561f-aecd-4d01-afa8-60277198d9d1\") " pod="kserve-ci-e2e-test/success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm"
Jul 21 17:38:18.781308 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.781228    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914-proxy-tls\") pod \"error-404-isvc-8558d-predictor-5594879d74-cksjh\" (UID: \"0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914\") " pod="kserve-ci-e2e-test/error-404-isvc-8558d-predictor-5594879d74-cksjh"
Jul 21 17:38:18.781308 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.781275    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8558d-predictor-5594879d74-cksjh\" (UID: \"0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914\") " pod="kserve-ci-e2e-test/error-404-isvc-8558d-predictor-5594879d74-cksjh"
Jul 21 17:38:18.781308 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.781301    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6xt6\" (UniqueName: \"kubernetes.io/projected/0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914-kube-api-access-v6xt6\") pod \"error-404-isvc-8558d-predictor-5594879d74-cksjh\" (UID: \"0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914\") " pod="kserve-ci-e2e-test/error-404-isvc-8558d-predictor-5594879d74-cksjh"
Jul 21 17:38:18.781574 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:38:18.781367    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-8558d-predictor-serving-cert: secret "error-404-isvc-8558d-predictor-serving-cert" not found
Jul 21 17:38:18.781574 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:38:18.781417    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914-proxy-tls podName:0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914 nodeName:}" failed. No retries permitted until 2026-07-21 17:38:19.281403058 +0000 UTC m=+727.043542639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914-proxy-tls") pod "error-404-isvc-8558d-predictor-5594879d74-cksjh" (UID: "0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914") : secret "error-404-isvc-8558d-predictor-serving-cert" not found
Jul 21 17:38:18.781875 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.781856    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8558d-predictor-5594879d74-cksjh\" (UID: \"0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914\") " pod="kserve-ci-e2e-test/error-404-isvc-8558d-predictor-5594879d74-cksjh"
Jul 21 17:38:18.790807 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:18.790781    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6xt6\" (UniqueName: \"kubernetes.io/projected/0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914-kube-api-access-v6xt6\") pod \"error-404-isvc-8558d-predictor-5594879d74-cksjh\" (UID: \"0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914\") " pod="kserve-ci-e2e-test/error-404-isvc-8558d-predictor-5594879d74-cksjh"
Jul 21 17:38:19.185367 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:19.185332    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9de3561f-aecd-4d01-afa8-60277198d9d1-proxy-tls\") pod \"success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm\" (UID: \"9de3561f-aecd-4d01-afa8-60277198d9d1\") " pod="kserve-ci-e2e-test/success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm"
Jul 21 17:38:19.187755 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:19.187737    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9de3561f-aecd-4d01-afa8-60277198d9d1-proxy-tls\") pod \"success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm\" (UID: \"9de3561f-aecd-4d01-afa8-60277198d9d1\") " pod="kserve-ci-e2e-test/success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm"
Jul 21 17:38:19.286366 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:19.286331    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914-proxy-tls\") pod \"error-404-isvc-8558d-predictor-5594879d74-cksjh\" (UID: \"0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914\") " pod="kserve-ci-e2e-test/error-404-isvc-8558d-predictor-5594879d74-cksjh"
Jul 21 17:38:19.288778 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:19.288757    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914-proxy-tls\") pod \"error-404-isvc-8558d-predictor-5594879d74-cksjh\" (UID: \"0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914\") " pod="kserve-ci-e2e-test/error-404-isvc-8558d-predictor-5594879d74-cksjh"
Jul 21 17:38:19.471716 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:19.471638    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm"
Jul 21 17:38:19.570115 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:19.570086    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8558d-predictor-5594879d74-cksjh"
Jul 21 17:38:19.612621 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:19.612597    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm"]
Jul 21 17:38:19.615442 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:38:19.615412    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9de3561f_aecd_4d01_afa8_60277198d9d1.slice/crio-cfcd10935550433bd7389fd18d0d315d71712d0057eea965b460c917c66773de WatchSource:0}: Error finding container cfcd10935550433bd7389fd18d0d315d71712d0057eea965b460c917c66773de: Status 404 returned error can't find the container with id cfcd10935550433bd7389fd18d0d315d71712d0057eea965b460c917c66773de
Jul 21 17:38:19.665513 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:19.665480    2563 generic.go:356] "Generic (PLEG): container finished" podID="8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10" containerID="8e10c815cb3e2ddd48a8a1757dee0d9850672d03ddebcb43d86d2aecfa40b0ff" exitCode=2
Jul 21 17:38:19.665646 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:19.665559    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs" event={"ID":"8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10","Type":"ContainerDied","Data":"8e10c815cb3e2ddd48a8a1757dee0d9850672d03ddebcb43d86d2aecfa40b0ff"}
Jul 21 17:38:19.667033 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:19.666998    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm" event={"ID":"9de3561f-aecd-4d01-afa8-60277198d9d1","Type":"ContainerStarted","Data":"cfcd10935550433bd7389fd18d0d315d71712d0057eea965b460c917c66773de"}
Jul 21 17:38:19.721919 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:19.721834    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8558d-predictor-5594879d74-cksjh"]
Jul 21 17:38:19.723667 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:38:19.723646    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d3e5f2a_fb6e_48d6_bb0c_d7c9eaf25914.slice/crio-242623dd85bc26f070e9db14487bba00e1d7f7fdb616bb6eb594120c37894186 WatchSource:0}: Error finding container 242623dd85bc26f070e9db14487bba00e1d7f7fdb616bb6eb594120c37894186: Status 404 returned error can't find the container with id 242623dd85bc26f070e9db14487bba00e1d7f7fdb616bb6eb594120c37894186
Jul 21 17:38:20.180808 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:20.180770    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c" podUID="6e8d920d-8b7d-4536-bae9-593c6439d608" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Jul 21 17:38:20.673195 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:20.673160    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm" event={"ID":"9de3561f-aecd-4d01-afa8-60277198d9d1","Type":"ContainerStarted","Data":"2fda7a820f11725cfdc128e121170e26c6a967bba17cbf36c18b19f038d55f1a"}
Jul 21 17:38:20.673694 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:20.673234    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm"
Jul 21 17:38:20.673694 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:20.673263    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm" event={"ID":"9de3561f-aecd-4d01-afa8-60277198d9d1","Type":"ContainerStarted","Data":"e4f6d8721fa2f9540446e5a6614dbdf7c035bab1f481fa35a611d0edefacd25a"}
Jul 21 17:38:20.673694 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:20.673293    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm"
Jul 21 17:38:20.674878 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:20.674843    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm" podUID="9de3561f-aecd-4d01-afa8-60277198d9d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 21 17:38:20.675011 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:20.674957    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8558d-predictor-5594879d74-cksjh" event={"ID":"0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914","Type":"ContainerStarted","Data":"c233fcf6344213c2b6df031801e140b614e17f8d34da857bf5e3bff4d3574f01"}
Jul 21 17:38:20.675011 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:20.675001    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8558d-predictor-5594879d74-cksjh" event={"ID":"0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914","Type":"ContainerStarted","Data":"b010d842ec367d0665206db19eb7144c4bebd42c902e3bfc40639c9a3e835090"}
Jul 21 17:38:20.675118 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:20.675021    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8558d-predictor-5594879d74-cksjh" event={"ID":"0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914","Type":"ContainerStarted","Data":"242623dd85bc26f070e9db14487bba00e1d7f7fdb616bb6eb594120c37894186"}
Jul 21 17:38:20.675199 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:20.675147    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8558d-predictor-5594879d74-cksjh"
Jul 21 17:38:20.694545 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:20.694497    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm" podStartSLOduration=2.6944845859999997 podStartE2EDuration="2.694484586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:38:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:38:20.691765795 +0000 UTC m=+728.453905433" watchObservedRunningTime="2026-07-21 17:38:20.694484586 +0000 UTC m=+728.456624189"
Jul 21 17:38:20.712847 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:20.712801    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-8558d-predictor-5594879d74-cksjh" podStartSLOduration=2.712788711 podStartE2EDuration="2.712788711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:38:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:38:20.709860097 +0000 UTC m=+728.471999700" watchObservedRunningTime="2026-07-21 17:38:20.712788711 +0000 UTC m=+728.474928315"
Jul 21 17:38:21.186654 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:21.186616    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt" podUID="69e0a43b-e6af-4859-9a8f-872ada531c9d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.19:8643/healthz\": dial tcp 10.133.0.19:8643: connect: connection refused"
Jul 21 17:38:21.255909 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:21.255857    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs" podUID="8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
Jul 21 17:38:21.261383 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:21.261352    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs" podUID="8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 21 17:38:21.678789 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:21.678744    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm" podUID="9de3561f-aecd-4d01-afa8-60277198d9d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 21 17:38:21.679187 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:21.678881    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8558d-predictor-5594879d74-cksjh"
Jul 21 17:38:21.679967 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:21.679946    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8558d-predictor-5594879d74-cksjh" podUID="0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 21 17:38:22.683549 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:22.683507    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8558d-predictor-5594879d74-cksjh" podUID="0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 21 17:38:23.215440 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.215419    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs"
Jul 21 17:38:23.319333 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.319251    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10-proxy-tls\") pod \"8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10\" (UID: \"8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10\") "
Jul 21 17:38:23.319333 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.319282    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmg7g\" (UniqueName: \"kubernetes.io/projected/8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10-kube-api-access-qmg7g\") pod \"8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10\" (UID: \"8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10\") "
Jul 21 17:38:23.319333 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.319299    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10-kube-rbac-proxy-sar-config\") pod \"8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10\" (UID: \"8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10\") "
Jul 21 17:38:23.319333 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.319316    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10-kserve-provision-location\") pod \"8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10\" (UID: \"8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10\") "
Jul 21 17:38:23.319659 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.319350    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10-cabundle-cert\") pod \"8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10\" (UID: \"8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10\") "
Jul 21 17:38:23.319718 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.319675    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10" (UID: "8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 17:38:23.319763 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.319710    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10" (UID: "8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:38:23.319798 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.319772    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10" (UID: "8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:38:23.321631 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.321604    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10" (UID: "8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:38:23.321752 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.321655    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10-kube-api-access-qmg7g" (OuterVolumeSpecName: "kube-api-access-qmg7g") pod "8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10" (UID: "8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10"). InnerVolumeSpecName "kube-api-access-qmg7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:38:23.420273 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.420241    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10-proxy-tls\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:38:23.420273 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.420270    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmg7g\" (UniqueName: \"kubernetes.io/projected/8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10-kube-api-access-qmg7g\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:38:23.420273 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.420280    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:38:23.420501 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.420291    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10-kserve-provision-location\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:38:23.420501 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.420300    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10-cabundle-cert\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:38:23.697462 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.697425    2563 generic.go:356] "Generic (PLEG): container finished" podID="6e8d920d-8b7d-4536-bae9-593c6439d608" containerID="2acee3ed50dc3d42451be0f066ad43026024098831deca3d677a365244920dee" exitCode=0
Jul 21 17:38:23.697906 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.697498    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c" event={"ID":"6e8d920d-8b7d-4536-bae9-593c6439d608","Type":"ContainerDied","Data":"2acee3ed50dc3d42451be0f066ad43026024098831deca3d677a365244920dee"}
Jul 21 17:38:23.699307 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.699283    2563 generic.go:356] "Generic (PLEG): container finished" podID="8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10" containerID="42f0234e5d9226b61019bd27f73a9dc15d5ab576be1472113027d603d3d426ce" exitCode=0
Jul 21 17:38:23.699407 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.699354    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs" event={"ID":"8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10","Type":"ContainerDied","Data":"42f0234e5d9226b61019bd27f73a9dc15d5ab576be1472113027d603d3d426ce"}
Jul 21 17:38:23.699407 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.699359    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs"
Jul 21 17:38:23.699407 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.699398    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs" event={"ID":"8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10","Type":"ContainerDied","Data":"0bd3ea2da6751d073449f132d8911af69ef21e0a07d55bfee72fad21272e1293"}
Jul 21 17:38:23.699527 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.699422    2563 scope.go:117] "RemoveContainer" containerID="8e10c815cb3e2ddd48a8a1757dee0d9850672d03ddebcb43d86d2aecfa40b0ff"
Jul 21 17:38:23.709298 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.709276    2563 scope.go:117] "RemoveContainer" containerID="42f0234e5d9226b61019bd27f73a9dc15d5ab576be1472113027d603d3d426ce"
Jul 21 17:38:23.717031 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.717013    2563 scope.go:117] "RemoveContainer" containerID="d18a06757e08dc2211708f6aa09f2cac5ff2880d01f657332a4f78687e72397f"
Jul 21 17:38:23.721950 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.721920    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs"]
Jul 21 17:38:23.726778 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.726695    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2w9gs"]
Jul 21 17:38:23.726843 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.726827    2563 scope.go:117] "RemoveContainer" containerID="8e10c815cb3e2ddd48a8a1757dee0d9850672d03ddebcb43d86d2aecfa40b0ff"
Jul 21 17:38:23.727141 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:38:23.727111    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e10c815cb3e2ddd48a8a1757dee0d9850672d03ddebcb43d86d2aecfa40b0ff\": container with ID starting with 8e10c815cb3e2ddd48a8a1757dee0d9850672d03ddebcb43d86d2aecfa40b0ff not found: ID does not exist" containerID="8e10c815cb3e2ddd48a8a1757dee0d9850672d03ddebcb43d86d2aecfa40b0ff"
Jul 21 17:38:23.727194 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.727153    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e10c815cb3e2ddd48a8a1757dee0d9850672d03ddebcb43d86d2aecfa40b0ff"} err="failed to get container status \"8e10c815cb3e2ddd48a8a1757dee0d9850672d03ddebcb43d86d2aecfa40b0ff\": rpc error: code = NotFound desc = could not find container \"8e10c815cb3e2ddd48a8a1757dee0d9850672d03ddebcb43d86d2aecfa40b0ff\": container with ID starting with 8e10c815cb3e2ddd48a8a1757dee0d9850672d03ddebcb43d86d2aecfa40b0ff not found: ID does not exist"
Jul 21 17:38:23.727194 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.727172    2563 scope.go:117] "RemoveContainer" containerID="42f0234e5d9226b61019bd27f73a9dc15d5ab576be1472113027d603d3d426ce"
Jul 21 17:38:23.727432 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:38:23.727418    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42f0234e5d9226b61019bd27f73a9dc15d5ab576be1472113027d603d3d426ce\": container with ID starting with 42f0234e5d9226b61019bd27f73a9dc15d5ab576be1472113027d603d3d426ce not found: ID does not exist" containerID="42f0234e5d9226b61019bd27f73a9dc15d5ab576be1472113027d603d3d426ce"
Jul 21 17:38:23.727469 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.727434    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42f0234e5d9226b61019bd27f73a9dc15d5ab576be1472113027d603d3d426ce"} err="failed to get container status \"42f0234e5d9226b61019bd27f73a9dc15d5ab576be1472113027d603d3d426ce\": rpc error: code = NotFound desc = could not find container \"42f0234e5d9226b61019bd27f73a9dc15d5ab576be1472113027d603d3d426ce\": container with ID starting with 42f0234e5d9226b61019bd27f73a9dc15d5ab576be1472113027d603d3d426ce not found: ID does not exist"
Jul 21 17:38:23.727469 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.727446    2563 scope.go:117] "RemoveContainer" containerID="d18a06757e08dc2211708f6aa09f2cac5ff2880d01f657332a4f78687e72397f"
Jul 21 17:38:23.727705 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:38:23.727688    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d18a06757e08dc2211708f6aa09f2cac5ff2880d01f657332a4f78687e72397f\": container with ID starting with d18a06757e08dc2211708f6aa09f2cac5ff2880d01f657332a4f78687e72397f not found: ID does not exist" containerID="d18a06757e08dc2211708f6aa09f2cac5ff2880d01f657332a4f78687e72397f"
Jul 21 17:38:23.727763 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.727709    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d18a06757e08dc2211708f6aa09f2cac5ff2880d01f657332a4f78687e72397f"} err="failed to get container status \"d18a06757e08dc2211708f6aa09f2cac5ff2880d01f657332a4f78687e72397f\": rpc error: code = NotFound desc = could not find container \"d18a06757e08dc2211708f6aa09f2cac5ff2880d01f657332a4f78687e72397f\": container with ID starting with d18a06757e08dc2211708f6aa09f2cac5ff2880d01f657332a4f78687e72397f not found: ID does not exist"
Jul 21 17:38:23.812831 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.812811    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c"
Jul 21 17:38:23.923838 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.923805    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e8d920d-8b7d-4536-bae9-593c6439d608-cabundle-cert\") pod \"6e8d920d-8b7d-4536-bae9-593c6439d608\" (UID: \"6e8d920d-8b7d-4536-bae9-593c6439d608\") "
Jul 21 17:38:23.924009 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.923869    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e8d920d-8b7d-4536-bae9-593c6439d608-kserve-provision-location\") pod \"6e8d920d-8b7d-4536-bae9-593c6439d608\" (UID: \"6e8d920d-8b7d-4536-bae9-593c6439d608\") "
Jul 21 17:38:23.924009 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.923893    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e8d920d-8b7d-4536-bae9-593c6439d608-proxy-tls\") pod \"6e8d920d-8b7d-4536-bae9-593c6439d608\" (UID: \"6e8d920d-8b7d-4536-bae9-593c6439d608\") "
Jul 21 17:38:23.924009 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.923912    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ww47z\" (UniqueName: \"kubernetes.io/projected/6e8d920d-8b7d-4536-bae9-593c6439d608-kube-api-access-ww47z\") pod \"6e8d920d-8b7d-4536-bae9-593c6439d608\" (UID: \"6e8d920d-8b7d-4536-bae9-593c6439d608\") "
Jul 21 17:38:23.924200 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.924009    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e8d920d-8b7d-4536-bae9-593c6439d608-kube-rbac-proxy-sar-config\") pod \"6e8d920d-8b7d-4536-bae9-593c6439d608\" (UID: \"6e8d920d-8b7d-4536-bae9-593c6439d608\") "
Jul 21 17:38:23.924200 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.924036    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e8d920d-8b7d-4536-bae9-593c6439d608-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6e8d920d-8b7d-4536-bae9-593c6439d608" (UID: "6e8d920d-8b7d-4536-bae9-593c6439d608"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:38:23.924304 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.924226    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e8d920d-8b7d-4536-bae9-593c6439d608-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6e8d920d-8b7d-4536-bae9-593c6439d608" (UID: "6e8d920d-8b7d-4536-bae9-593c6439d608"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 17:38:23.924304 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.924279    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e8d920d-8b7d-4536-bae9-593c6439d608-cabundle-cert\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:38:23.924409 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.924371    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e8d920d-8b7d-4536-bae9-593c6439d608-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6e8d920d-8b7d-4536-bae9-593c6439d608" (UID: "6e8d920d-8b7d-4536-bae9-593c6439d608"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:38:23.926119 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.926073    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e8d920d-8b7d-4536-bae9-593c6439d608-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6e8d920d-8b7d-4536-bae9-593c6439d608" (UID: "6e8d920d-8b7d-4536-bae9-593c6439d608"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:38:23.926119 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:23.926078    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e8d920d-8b7d-4536-bae9-593c6439d608-kube-api-access-ww47z" (OuterVolumeSpecName: "kube-api-access-ww47z") pod "6e8d920d-8b7d-4536-bae9-593c6439d608" (UID: "6e8d920d-8b7d-4536-bae9-593c6439d608"). InnerVolumeSpecName "kube-api-access-ww47z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:38:24.025648 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.025608    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e8d920d-8b7d-4536-bae9-593c6439d608-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:38:24.025648 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.025645    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e8d920d-8b7d-4536-bae9-593c6439d608-kserve-provision-location\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:38:24.025648 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.025655    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e8d920d-8b7d-4536-bae9-593c6439d608-proxy-tls\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:38:24.025648 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.025665    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww47z\" (UniqueName: \"kubernetes.io/projected/6e8d920d-8b7d-4536-bae9-593c6439d608-kube-api-access-ww47z\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:38:24.366821 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.366801    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt"
Jul 21 17:38:24.429232 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.429202    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqhh4\" (UniqueName: \"kubernetes.io/projected/69e0a43b-e6af-4859-9a8f-872ada531c9d-kube-api-access-bqhh4\") pod \"69e0a43b-e6af-4859-9a8f-872ada531c9d\" (UID: \"69e0a43b-e6af-4859-9a8f-872ada531c9d\") "
Jul 21 17:38:24.429384 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.429248    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69e0a43b-e6af-4859-9a8f-872ada531c9d-kserve-provision-location\") pod \"69e0a43b-e6af-4859-9a8f-872ada531c9d\" (UID: \"69e0a43b-e6af-4859-9a8f-872ada531c9d\") "
Jul 21 17:38:24.429445 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.429376    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69e0a43b-e6af-4859-9a8f-872ada531c9d-cabundle-cert\") pod \"69e0a43b-e6af-4859-9a8f-872ada531c9d\" (UID: \"69e0a43b-e6af-4859-9a8f-872ada531c9d\") "
Jul 21 17:38:24.429445 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.429432    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69e0a43b-e6af-4859-9a8f-872ada531c9d-proxy-tls\") pod \"69e0a43b-e6af-4859-9a8f-872ada531c9d\" (UID: \"69e0a43b-e6af-4859-9a8f-872ada531c9d\") "
Jul 21 17:38:24.429557 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.429466    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69e0a43b-e6af-4859-9a8f-872ada531c9d-kube-rbac-proxy-sar-config\") pod \"69e0a43b-e6af-4859-9a8f-872ada531c9d\" (UID: \"69e0a43b-e6af-4859-9a8f-872ada531c9d\") "
Jul 21 17:38:24.429628 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.429609    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69e0a43b-e6af-4859-9a8f-872ada531c9d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "69e0a43b-e6af-4859-9a8f-872ada531c9d" (UID: "69e0a43b-e6af-4859-9a8f-872ada531c9d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 17:38:24.429728 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.429704    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69e0a43b-e6af-4859-9a8f-872ada531c9d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "69e0a43b-e6af-4859-9a8f-872ada531c9d" (UID: "69e0a43b-e6af-4859-9a8f-872ada531c9d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:38:24.429794 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.429716    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69e0a43b-e6af-4859-9a8f-872ada531c9d-kserve-provision-location\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:38:24.429894 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.429875    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69e0a43b-e6af-4859-9a8f-872ada531c9d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "69e0a43b-e6af-4859-9a8f-872ada531c9d" (UID: "69e0a43b-e6af-4859-9a8f-872ada531c9d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:38:24.431510 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.431479    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69e0a43b-e6af-4859-9a8f-872ada531c9d-kube-api-access-bqhh4" (OuterVolumeSpecName: "kube-api-access-bqhh4") pod "69e0a43b-e6af-4859-9a8f-872ada531c9d" (UID: "69e0a43b-e6af-4859-9a8f-872ada531c9d"). InnerVolumeSpecName "kube-api-access-bqhh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:38:24.431564 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.431523    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69e0a43b-e6af-4859-9a8f-872ada531c9d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "69e0a43b-e6af-4859-9a8f-872ada531c9d" (UID: "69e0a43b-e6af-4859-9a8f-872ada531c9d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:38:24.530157 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.530054    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69e0a43b-e6af-4859-9a8f-872ada531c9d-cabundle-cert\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:38:24.530157 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.530080    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69e0a43b-e6af-4859-9a8f-872ada531c9d-proxy-tls\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:38:24.530157 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.530088    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69e0a43b-e6af-4859-9a8f-872ada531c9d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:38:24.530157 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.530097    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqhh4\" (UniqueName: \"kubernetes.io/projected/69e0a43b-e6af-4859-9a8f-872ada531c9d-kube-api-access-bqhh4\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:38:24.704948 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.704913    2563 generic.go:356] "Generic (PLEG): container finished" podID="69e0a43b-e6af-4859-9a8f-872ada531c9d" containerID="aa8cc606a24907fa0592f3678ff55d065d8bf1a6fdd487c86626b9ba0f3ecacc" exitCode=0
Jul 21 17:38:24.705428 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.704974    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt"
Jul 21 17:38:24.705428 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.704998    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt" event={"ID":"69e0a43b-e6af-4859-9a8f-872ada531c9d","Type":"ContainerDied","Data":"aa8cc606a24907fa0592f3678ff55d065d8bf1a6fdd487c86626b9ba0f3ecacc"}
Jul 21 17:38:24.705428 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.705065    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt" event={"ID":"69e0a43b-e6af-4859-9a8f-872ada531c9d","Type":"ContainerDied","Data":"51febc87b0113be6e4674f6c0a6626117319d2c987b1d004de2b826d1a33cfc5"}
Jul 21 17:38:24.705428 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.705105    2563 scope.go:117] "RemoveContainer" containerID="dab00082341dc5087aaa1ffd466d367cddb5fe9b749c3d7ab7803741f0911a0a"
Jul 21 17:38:24.708757 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.708725    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c" event={"ID":"6e8d920d-8b7d-4536-bae9-593c6439d608","Type":"ContainerDied","Data":"fb2647f9899eede01dd864f412ad3b49f73c88c34a406a0b00f0cb16aa003b55"}
Jul 21 17:38:24.708852 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.708790    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c"
Jul 21 17:38:24.713512 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.713494    2563 scope.go:117] "RemoveContainer" containerID="aa8cc606a24907fa0592f3678ff55d065d8bf1a6fdd487c86626b9ba0f3ecacc"
Jul 21 17:38:24.721943 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.721924    2563 scope.go:117] "RemoveContainer" containerID="64b9a5f1392068f559e848abd403c50994bb9504dbc8fe80cfd6ae2e6764eeb9"
Jul 21 17:38:24.730070 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.730021    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt"]
Jul 21 17:38:24.730115 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.730103    2563 scope.go:117] "RemoveContainer" containerID="dab00082341dc5087aaa1ffd466d367cddb5fe9b749c3d7ab7803741f0911a0a"
Jul 21 17:38:24.730568 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:38:24.730532    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dab00082341dc5087aaa1ffd466d367cddb5fe9b749c3d7ab7803741f0911a0a\": container with ID starting with dab00082341dc5087aaa1ffd466d367cddb5fe9b749c3d7ab7803741f0911a0a not found: ID does not exist" containerID="dab00082341dc5087aaa1ffd466d367cddb5fe9b749c3d7ab7803741f0911a0a"
Jul 21 17:38:24.730755 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.730603    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dab00082341dc5087aaa1ffd466d367cddb5fe9b749c3d7ab7803741f0911a0a"} err="failed to get container status \"dab00082341dc5087aaa1ffd466d367cddb5fe9b749c3d7ab7803741f0911a0a\": rpc error: code = NotFound desc = could not find container \"dab00082341dc5087aaa1ffd466d367cddb5fe9b749c3d7ab7803741f0911a0a\": container with ID starting with dab00082341dc5087aaa1ffd466d367cddb5fe9b749c3d7ab7803741f0911a0a not found: ID does not exist"
Jul 21 17:38:24.730755 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.730631    2563 scope.go:117] "RemoveContainer" containerID="aa8cc606a24907fa0592f3678ff55d065d8bf1a6fdd487c86626b9ba0f3ecacc"
Jul 21 17:38:24.731016 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:38:24.730998    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa8cc606a24907fa0592f3678ff55d065d8bf1a6fdd487c86626b9ba0f3ecacc\": container with ID starting with aa8cc606a24907fa0592f3678ff55d065d8bf1a6fdd487c86626b9ba0f3ecacc not found: ID does not exist" containerID="aa8cc606a24907fa0592f3678ff55d065d8bf1a6fdd487c86626b9ba0f3ecacc"
Jul 21 17:38:24.731102 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.731020    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa8cc606a24907fa0592f3678ff55d065d8bf1a6fdd487c86626b9ba0f3ecacc"} err="failed to get container status \"aa8cc606a24907fa0592f3678ff55d065d8bf1a6fdd487c86626b9ba0f3ecacc\": rpc error: code = NotFound desc = could not find container \"aa8cc606a24907fa0592f3678ff55d065d8bf1a6fdd487c86626b9ba0f3ecacc\": container with ID starting with aa8cc606a24907fa0592f3678ff55d065d8bf1a6fdd487c86626b9ba0f3ecacc not found: ID does not exist"
Jul 21 17:38:24.731102 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.731042    2563 scope.go:117] "RemoveContainer" containerID="64b9a5f1392068f559e848abd403c50994bb9504dbc8fe80cfd6ae2e6764eeb9"
Jul 21 17:38:24.731511 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:38:24.731482    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64b9a5f1392068f559e848abd403c50994bb9504dbc8fe80cfd6ae2e6764eeb9\": container with ID starting with 64b9a5f1392068f559e848abd403c50994bb9504dbc8fe80cfd6ae2e6764eeb9 not found: ID does not exist" containerID="64b9a5f1392068f559e848abd403c50994bb9504dbc8fe80cfd6ae2e6764eeb9"
Jul 21 17:38:24.731589 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.731520    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64b9a5f1392068f559e848abd403c50994bb9504dbc8fe80cfd6ae2e6764eeb9"} err="failed to get container status \"64b9a5f1392068f559e848abd403c50994bb9504dbc8fe80cfd6ae2e6764eeb9\": rpc error: code = NotFound desc = could not find container \"64b9a5f1392068f559e848abd403c50994bb9504dbc8fe80cfd6ae2e6764eeb9\": container with ID starting with 64b9a5f1392068f559e848abd403c50994bb9504dbc8fe80cfd6ae2e6764eeb9 not found: ID does not exist"
Jul 21 17:38:24.731589 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.731542    2563 scope.go:117] "RemoveContainer" containerID="69afb319b37c95cd8e4771d0ff24eda579a797484e615bb4b1e85e8f04973243"
Jul 21 17:38:24.732720 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.732701    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84695d5b5c-x96vt"]
Jul 21 17:38:24.740657 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.740641    2563 scope.go:117] "RemoveContainer" containerID="2acee3ed50dc3d42451be0f066ad43026024098831deca3d677a365244920dee"
Jul 21 17:38:24.745373 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.745355    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c"]
Jul 21 17:38:24.749521 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.749495    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c"]
Jul 21 17:38:24.750212 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.750187    2563 scope.go:117] "RemoveContainer" containerID="90691b9203ae36f7d65f9657f33e645fa063993776ec1bf23335eab210e8184c"
Jul 21 17:38:24.759848 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.759814    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-868d77456c-snz9c" podUID="6e8d920d-8b7d-4536-bae9-593c6439d608" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: i/o timeout"
Jul 21 17:38:24.763542 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.763522    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69e0a43b-e6af-4859-9a8f-872ada531c9d" path="/var/lib/kubelet/pods/69e0a43b-e6af-4859-9a8f-872ada531c9d/volumes"
Jul 21 17:38:24.763974 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.763960    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e8d920d-8b7d-4536-bae9-593c6439d608" path="/var/lib/kubelet/pods/6e8d920d-8b7d-4536-bae9-593c6439d608/volumes"
Jul 21 17:38:24.764684 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:24.764669    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10" path="/var/lib/kubelet/pods/8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10/volumes"
Jul 21 17:38:25.531601 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:25.531561    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6" podUID="aeeeed18-d5cb-44dc-98d8-e896038dee1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 21 17:38:26.530367 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:26.530326    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l" podUID="ef4ef854-bdcb-419b-b4a7-ae0f4caedc69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 21 17:38:26.683975 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:26.683942    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm"
Jul 21 17:38:26.684434 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:26.684405    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm" podUID="9de3561f-aecd-4d01-afa8-60277198d9d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 21 17:38:27.688076 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:27.688050    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8558d-predictor-5594879d74-cksjh"
Jul 21 17:38:27.688570 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:27.688545    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8558d-predictor-5594879d74-cksjh" podUID="0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 21 17:38:35.531272 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:35.531243    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6"
Jul 21 17:38:36.530386 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:36.530341    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l" podUID="ef4ef854-bdcb-419b-b4a7-ae0f4caedc69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 21 17:38:36.684777 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:36.684740    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm" podUID="9de3561f-aecd-4d01-afa8-60277198d9d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 21 17:38:37.689422 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:37.689387    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8558d-predictor-5594879d74-cksjh" podUID="0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 21 17:38:46.531368 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:46.531330    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l"
Jul 21 17:38:46.684940 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:46.684897    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm" podUID="9de3561f-aecd-4d01-afa8-60277198d9d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 21 17:38:47.688835 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:47.688797    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8558d-predictor-5594879d74-cksjh" podUID="0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 21 17:38:56.684442 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:56.684401    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm" podUID="9de3561f-aecd-4d01-afa8-60277198d9d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 21 17:38:57.688633 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:38:57.688583    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8558d-predictor-5594879d74-cksjh" podUID="0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 21 17:39:06.685890 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:39:06.685861    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm"
Jul 21 17:39:07.690079 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:39:07.690044    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8558d-predictor-5594879d74-cksjh"
Jul 21 17:47:03.402384 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.402304    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6"]
Jul 21 17:47:03.402877 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.402559    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6" podUID="aeeeed18-d5cb-44dc-98d8-e896038dee1b" containerName="kserve-container" containerID="cri-o://7452dbc7929fea9bda0fd1f58e072e7bd11e89eaaf2631a1ad86999c16539f9a" gracePeriod=30
Jul 21 17:47:03.402877 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.402612    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6" podUID="aeeeed18-d5cb-44dc-98d8-e896038dee1b" containerName="kube-rbac-proxy" containerID="cri-o://3d2f7a01abcb50f5228bfb1aca202d30140d7ce4bb45386193a9cc46910537b8" gracePeriod=30
Jul 21 17:47:03.524574 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.524547    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l"]
Jul 21 17:47:03.524785 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.524765    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l" podUID="ef4ef854-bdcb-419b-b4a7-ae0f4caedc69" containerName="kserve-container" containerID="cri-o://244a840e829c5d11f54b697ef871553652af84ba2102dc15597cfd47ae486faf" gracePeriod=30
Jul 21 17:47:03.524888 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.524862    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l" podUID="ef4ef854-bdcb-419b-b4a7-ae0f4caedc69" containerName="kube-rbac-proxy" containerID="cri-o://5c511b41ad09bf8a6fd450fb0ac8b0bf4865195f94e08c5fbd61021ccbf44465" gracePeriod=30
Jul 21 17:47:03.538391 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.538368    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f64bd-predictor-7764f488c-c4ttx"]
Jul 21 17:47:03.538688 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.538676    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e8d920d-8b7d-4536-bae9-593c6439d608" containerName="storage-initializer"
Jul 21 17:47:03.538729 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.538690    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e8d920d-8b7d-4536-bae9-593c6439d608" containerName="storage-initializer"
Jul 21 17:47:03.538729 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.538705    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69e0a43b-e6af-4859-9a8f-872ada531c9d" containerName="kserve-container"
Jul 21 17:47:03.538729 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.538713    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="69e0a43b-e6af-4859-9a8f-872ada531c9d" containerName="kserve-container"
Jul 21 17:47:03.538729 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.538719    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e8d920d-8b7d-4536-bae9-593c6439d608" containerName="kserve-container"
Jul 21 17:47:03.538729 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.538724    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e8d920d-8b7d-4536-bae9-593c6439d608" containerName="kserve-container"
Jul 21 17:47:03.538876 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.538733    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69e0a43b-e6af-4859-9a8f-872ada531c9d" containerName="storage-initializer"
Jul 21 17:47:03.538876 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.538738    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="69e0a43b-e6af-4859-9a8f-872ada531c9d" containerName="storage-initializer"
Jul 21 17:47:03.538876 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.538746    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10" containerName="storage-initializer"
Jul 21 17:47:03.538876 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.538751    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10" containerName="storage-initializer"
Jul 21 17:47:03.538876 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.538759    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10" containerName="kserve-container"
Jul 21 17:47:03.538876 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.538764    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10" containerName="kserve-container"
Jul 21 17:47:03.538876 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.538771    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69e0a43b-e6af-4859-9a8f-872ada531c9d" containerName="kube-rbac-proxy"
Jul 21 17:47:03.538876 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.538776    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="69e0a43b-e6af-4859-9a8f-872ada531c9d" containerName="kube-rbac-proxy"
Jul 21 17:47:03.538876 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.538781    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10" containerName="kube-rbac-proxy"
Jul 21 17:47:03.538876 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.538786    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10" containerName="kube-rbac-proxy"
Jul 21 17:47:03.538876 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.538796    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e8d920d-8b7d-4536-bae9-593c6439d608" containerName="kube-rbac-proxy"
Jul 21 17:47:03.538876 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.538801    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e8d920d-8b7d-4536-bae9-593c6439d608" containerName="kube-rbac-proxy"
Jul 21 17:47:03.538876 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.538849    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10" containerName="kube-rbac-proxy"
Jul 21 17:47:03.538876 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.538858    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="69e0a43b-e6af-4859-9a8f-872ada531c9d" containerName="kserve-container"
Jul 21 17:47:03.538876 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.538863    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e8d920d-8b7d-4536-bae9-593c6439d608" containerName="kube-rbac-proxy"
Jul 21 17:47:03.538876 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.538869    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c5e3cb0-172b-4f1a-8e76-028bbc1c8f10" containerName="kserve-container"
Jul 21 17:47:03.538876 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.538874    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="69e0a43b-e6af-4859-9a8f-872ada531c9d" containerName="kube-rbac-proxy"
Jul 21 17:47:03.538876 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.538880    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e8d920d-8b7d-4536-bae9-593c6439d608" containerName="kserve-container"
Jul 21 17:47:03.541786 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.541768    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f64bd-predictor-7764f488c-c4ttx"
Jul 21 17:47:03.544296 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.544263    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f64bd-kube-rbac-proxy-sar-config\""
Jul 21 17:47:03.544420 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.544379    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f64bd-predictor-serving-cert\""
Jul 21 17:47:03.551690 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.551663    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f64bd-predictor-7764f488c-c4ttx"]
Jul 21 17:47:03.570200 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.570154    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80b3aae2-8538-4e58-8249-c708ca4fde57-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f64bd-predictor-7764f488c-c4ttx\" (UID: \"80b3aae2-8538-4e58-8249-c708ca4fde57\") " pod="kserve-ci-e2e-test/success-200-isvc-f64bd-predictor-7764f488c-c4ttx"
Jul 21 17:47:03.570200 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.570198    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ptkz\" (UniqueName: \"kubernetes.io/projected/80b3aae2-8538-4e58-8249-c708ca4fde57-kube-api-access-2ptkz\") pod \"success-200-isvc-f64bd-predictor-7764f488c-c4ttx\" (UID: \"80b3aae2-8538-4e58-8249-c708ca4fde57\") " pod="kserve-ci-e2e-test/success-200-isvc-f64bd-predictor-7764f488c-c4ttx"
Jul 21 17:47:03.570392 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.570322    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80b3aae2-8538-4e58-8249-c708ca4fde57-proxy-tls\") pod \"success-200-isvc-f64bd-predictor-7764f488c-c4ttx\" (UID: \"80b3aae2-8538-4e58-8249-c708ca4fde57\") " pod="kserve-ci-e2e-test/success-200-isvc-f64bd-predictor-7764f488c-c4ttx"
Jul 21 17:47:03.612495 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.612457    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7"]
Jul 21 17:47:03.615655 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.615639    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7"
Jul 21 17:47:03.618232 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.618213    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f64bd-predictor-serving-cert\""
Jul 21 17:47:03.618315 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.618222    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f64bd-kube-rbac-proxy-sar-config\""
Jul 21 17:47:03.624835 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.624814    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7"]
Jul 21 17:47:03.670847 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.670759    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80b3aae2-8538-4e58-8249-c708ca4fde57-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f64bd-predictor-7764f488c-c4ttx\" (UID: \"80b3aae2-8538-4e58-8249-c708ca4fde57\") " pod="kserve-ci-e2e-test/success-200-isvc-f64bd-predictor-7764f488c-c4ttx"
Jul 21 17:47:03.670847 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.670796    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ptkz\" (UniqueName: \"kubernetes.io/projected/80b3aae2-8538-4e58-8249-c708ca4fde57-kube-api-access-2ptkz\") pod \"success-200-isvc-f64bd-predictor-7764f488c-c4ttx\" (UID: \"80b3aae2-8538-4e58-8249-c708ca4fde57\") " pod="kserve-ci-e2e-test/success-200-isvc-f64bd-predictor-7764f488c-c4ttx"
Jul 21 17:47:03.670847 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.670823    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq2c4\" (UniqueName: \"kubernetes.io/projected/26f7402a-85bc-48c6-b8aa-2d192f06bd3c-kube-api-access-pq2c4\") pod \"error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7\" (UID: \"26f7402a-85bc-48c6-b8aa-2d192f06bd3c\") " pod="kserve-ci-e2e-test/error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7"
Jul 21 17:47:03.671113 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.670870    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26f7402a-85bc-48c6-b8aa-2d192f06bd3c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7\" (UID: \"26f7402a-85bc-48c6-b8aa-2d192f06bd3c\") " pod="kserve-ci-e2e-test/error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7"
Jul 21 17:47:03.671113 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.670889    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80b3aae2-8538-4e58-8249-c708ca4fde57-proxy-tls\") pod \"success-200-isvc-f64bd-predictor-7764f488c-c4ttx\" (UID: \"80b3aae2-8538-4e58-8249-c708ca4fde57\") " pod="kserve-ci-e2e-test/success-200-isvc-f64bd-predictor-7764f488c-c4ttx"
Jul 21 17:47:03.671113 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.670921    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26f7402a-85bc-48c6-b8aa-2d192f06bd3c-proxy-tls\") pod \"error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7\" (UID: \"26f7402a-85bc-48c6-b8aa-2d192f06bd3c\") " pod="kserve-ci-e2e-test/error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7"
Jul 21 17:47:03.671113 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:47:03.671006    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-f64bd-predictor-serving-cert: secret "success-200-isvc-f64bd-predictor-serving-cert" not found
Jul 21 17:47:03.671113 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:47:03.671066    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80b3aae2-8538-4e58-8249-c708ca4fde57-proxy-tls podName:80b3aae2-8538-4e58-8249-c708ca4fde57 nodeName:}" failed. No retries permitted until 2026-07-21 17:47:04.171044898 +0000 UTC m=+1251.933184480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/80b3aae2-8538-4e58-8249-c708ca4fde57-proxy-tls") pod "success-200-isvc-f64bd-predictor-7764f488c-c4ttx" (UID: "80b3aae2-8538-4e58-8249-c708ca4fde57") : secret "success-200-isvc-f64bd-predictor-serving-cert" not found
Jul 21 17:47:03.671574 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.671550    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80b3aae2-8538-4e58-8249-c708ca4fde57-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f64bd-predictor-7764f488c-c4ttx\" (UID: \"80b3aae2-8538-4e58-8249-c708ca4fde57\") " pod="kserve-ci-e2e-test/success-200-isvc-f64bd-predictor-7764f488c-c4ttx"
Jul 21 17:47:03.682368 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.682330    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ptkz\" (UniqueName: \"kubernetes.io/projected/80b3aae2-8538-4e58-8249-c708ca4fde57-kube-api-access-2ptkz\") pod \"success-200-isvc-f64bd-predictor-7764f488c-c4ttx\" (UID: \"80b3aae2-8538-4e58-8249-c708ca4fde57\") " pod="kserve-ci-e2e-test/success-200-isvc-f64bd-predictor-7764f488c-c4ttx"
Jul 21 17:47:03.711095 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.711067    2563 generic.go:356] "Generic (PLEG): container finished" podID="ef4ef854-bdcb-419b-b4a7-ae0f4caedc69" containerID="5c511b41ad09bf8a6fd450fb0ac8b0bf4865195f94e08c5fbd61021ccbf44465" exitCode=2
Jul 21 17:47:03.711250 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.711158    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l" event={"ID":"ef4ef854-bdcb-419b-b4a7-ae0f4caedc69","Type":"ContainerDied","Data":"5c511b41ad09bf8a6fd450fb0ac8b0bf4865195f94e08c5fbd61021ccbf44465"}
Jul 21 17:47:03.712867 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.712847    2563 generic.go:356] "Generic (PLEG): container finished" podID="aeeeed18-d5cb-44dc-98d8-e896038dee1b" containerID="3d2f7a01abcb50f5228bfb1aca202d30140d7ce4bb45386193a9cc46910537b8" exitCode=2
Jul 21 17:47:03.712943 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.712899    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6" event={"ID":"aeeeed18-d5cb-44dc-98d8-e896038dee1b","Type":"ContainerDied","Data":"3d2f7a01abcb50f5228bfb1aca202d30140d7ce4bb45386193a9cc46910537b8"}
Jul 21 17:47:03.777424 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.772700    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26f7402a-85bc-48c6-b8aa-2d192f06bd3c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7\" (UID: \"26f7402a-85bc-48c6-b8aa-2d192f06bd3c\") " pod="kserve-ci-e2e-test/error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7"
Jul 21 17:47:03.777424 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.772811    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26f7402a-85bc-48c6-b8aa-2d192f06bd3c-proxy-tls\") pod \"error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7\" (UID: \"26f7402a-85bc-48c6-b8aa-2d192f06bd3c\") " pod="kserve-ci-e2e-test/error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7"
Jul 21 17:47:03.777424 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.772912    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq2c4\" (UniqueName: \"kubernetes.io/projected/26f7402a-85bc-48c6-b8aa-2d192f06bd3c-kube-api-access-pq2c4\") pod \"error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7\" (UID: \"26f7402a-85bc-48c6-b8aa-2d192f06bd3c\") " pod="kserve-ci-e2e-test/error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7"
Jul 21 17:47:03.777424 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:47:03.773507    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-f64bd-predictor-serving-cert: secret "error-404-isvc-f64bd-predictor-serving-cert" not found
Jul 21 17:47:03.777424 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:47:03.773585    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26f7402a-85bc-48c6-b8aa-2d192f06bd3c-proxy-tls podName:26f7402a-85bc-48c6-b8aa-2d192f06bd3c nodeName:}" failed. No retries permitted until 2026-07-21 17:47:04.2735663 +0000 UTC m=+1252.035705881 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/26f7402a-85bc-48c6-b8aa-2d192f06bd3c-proxy-tls") pod "error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7" (UID: "26f7402a-85bc-48c6-b8aa-2d192f06bd3c") : secret "error-404-isvc-f64bd-predictor-serving-cert" not found
Jul 21 17:47:03.777424 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.773964    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26f7402a-85bc-48c6-b8aa-2d192f06bd3c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7\" (UID: \"26f7402a-85bc-48c6-b8aa-2d192f06bd3c\") " pod="kserve-ci-e2e-test/error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7"
Jul 21 17:47:03.781699 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:03.781675    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq2c4\" (UniqueName: \"kubernetes.io/projected/26f7402a-85bc-48c6-b8aa-2d192f06bd3c-kube-api-access-pq2c4\") pod \"error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7\" (UID: \"26f7402a-85bc-48c6-b8aa-2d192f06bd3c\") " pod="kserve-ci-e2e-test/error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7"
Jul 21 17:47:04.176017 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:04.175976    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80b3aae2-8538-4e58-8249-c708ca4fde57-proxy-tls\") pod \"success-200-isvc-f64bd-predictor-7764f488c-c4ttx\" (UID: \"80b3aae2-8538-4e58-8249-c708ca4fde57\") " pod="kserve-ci-e2e-test/success-200-isvc-f64bd-predictor-7764f488c-c4ttx"
Jul 21 17:47:04.178547 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:04.178521    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80b3aae2-8538-4e58-8249-c708ca4fde57-proxy-tls\") pod \"success-200-isvc-f64bd-predictor-7764f488c-c4ttx\" (UID: \"80b3aae2-8538-4e58-8249-c708ca4fde57\") " pod="kserve-ci-e2e-test/success-200-isvc-f64bd-predictor-7764f488c-c4ttx"
Jul 21 17:47:04.277157 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:04.277101    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26f7402a-85bc-48c6-b8aa-2d192f06bd3c-proxy-tls\") pod \"error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7\" (UID: \"26f7402a-85bc-48c6-b8aa-2d192f06bd3c\") " pod="kserve-ci-e2e-test/error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7"
Jul 21 17:47:04.279593 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:04.279573    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26f7402a-85bc-48c6-b8aa-2d192f06bd3c-proxy-tls\") pod \"error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7\" (UID: \"26f7402a-85bc-48c6-b8aa-2d192f06bd3c\") " pod="kserve-ci-e2e-test/error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7"
Jul 21 17:47:04.453334 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:04.453221    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f64bd-predictor-7764f488c-c4ttx"
Jul 21 17:47:04.526695 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:04.526660    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7"
Jul 21 17:47:04.597546 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:04.597517    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f64bd-predictor-7764f488c-c4ttx"]
Jul 21 17:47:04.598819 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:47:04.598792    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80b3aae2_8538_4e58_8249_c708ca4fde57.slice/crio-5377e91eaaf5a227bf6a46bc42d5c5ed0afeb0c95ac95a45c0c1ab3232abc5a7 WatchSource:0}: Error finding container 5377e91eaaf5a227bf6a46bc42d5c5ed0afeb0c95ac95a45c0c1ab3232abc5a7: Status 404 returned error can't find the container with id 5377e91eaaf5a227bf6a46bc42d5c5ed0afeb0c95ac95a45c0c1ab3232abc5a7
Jul 21 17:47:04.670043 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:04.669712    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7"]
Jul 21 17:47:04.672054 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:47:04.672026    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26f7402a_85bc_48c6_b8aa_2d192f06bd3c.slice/crio-65ab87c0de8bf91dce765e034e800f6c0efc87d6991bf9b2edaf9c082515e38f WatchSource:0}: Error finding container 65ab87c0de8bf91dce765e034e800f6c0efc87d6991bf9b2edaf9c082515e38f: Status 404 returned error can't find the container with id 65ab87c0de8bf91dce765e034e800f6c0efc87d6991bf9b2edaf9c082515e38f
Jul 21 17:47:04.719195 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:04.719165    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7" event={"ID":"26f7402a-85bc-48c6-b8aa-2d192f06bd3c","Type":"ContainerStarted","Data":"65ab87c0de8bf91dce765e034e800f6c0efc87d6991bf9b2edaf9c082515e38f"}
Jul 21 17:47:04.720598 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:04.720576    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f64bd-predictor-7764f488c-c4ttx" event={"ID":"80b3aae2-8538-4e58-8249-c708ca4fde57","Type":"ContainerStarted","Data":"744c692918959eab78fa52227bbe74fe029a960aff6d49a324fccfd304bda384"}
Jul 21 17:47:04.720678 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:04.720623    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f64bd-predictor-7764f488c-c4ttx" event={"ID":"80b3aae2-8538-4e58-8249-c708ca4fde57","Type":"ContainerStarted","Data":"5377e91eaaf5a227bf6a46bc42d5c5ed0afeb0c95ac95a45c0c1ab3232abc5a7"}
Jul 21 17:47:05.521433 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:05.521390    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6" podUID="aeeeed18-d5cb-44dc-98d8-e896038dee1b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Jul 21 17:47:05.531397 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:05.531349    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6" podUID="aeeeed18-d5cb-44dc-98d8-e896038dee1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 21 17:47:05.726340 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:05.726301    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7" event={"ID":"26f7402a-85bc-48c6-b8aa-2d192f06bd3c","Type":"ContainerStarted","Data":"3a6ebe019aca80f430745d2da2e88b0f05468bf34c060f86780436e792bf5e62"}
Jul 21 17:47:05.726555 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:05.726376    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7"
Jul 21 17:47:05.726555 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:05.726504    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7"
Jul 21 17:47:05.726555 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:05.726536    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7" event={"ID":"26f7402a-85bc-48c6-b8aa-2d192f06bd3c","Type":"ContainerStarted","Data":"6d523e03444ae1a1a609b35bd052fe16cafbd38434655e868421bb9a16e7b8b1"}
Jul 21 17:47:05.727833 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:05.727801    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7" podUID="26f7402a-85bc-48c6-b8aa-2d192f06bd3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 21 17:47:05.728265 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:05.728242    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f64bd-predictor-7764f488c-c4ttx" event={"ID":"80b3aae2-8538-4e58-8249-c708ca4fde57","Type":"ContainerStarted","Data":"c61f19bd973ee0986a8fc94772f309bda529c7a6b4eb2960af40737924463c6d"}
Jul 21 17:47:05.728384 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:05.728370    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f64bd-predictor-7764f488c-c4ttx"
Jul 21 17:47:05.728427 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:05.728409    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f64bd-predictor-7764f488c-c4ttx"
Jul 21 17:47:05.729218 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:05.729197    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f64bd-predictor-7764f488c-c4ttx" podUID="80b3aae2-8538-4e58-8249-c708ca4fde57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 21 17:47:05.744639 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:05.744600    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7" podStartSLOduration=2.74458579 podStartE2EDuration="2.74458579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:47:05.742952352 +0000 UTC m=+1253.505091957" watchObservedRunningTime="2026-07-21 17:47:05.74458579 +0000 UTC m=+1253.506725443"
Jul 21 17:47:05.761428 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:05.761374    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-f64bd-predictor-7764f488c-c4ttx" podStartSLOduration=2.76136061 podStartE2EDuration="2.76136061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:47:05.759926725 +0000 UTC m=+1253.522066330" watchObservedRunningTime="2026-07-21 17:47:05.76136061 +0000 UTC m=+1253.523500213"
Jul 21 17:47:06.526047 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:06.526010    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l" podUID="ef4ef854-bdcb-419b-b4a7-ae0f4caedc69" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Jul 21 17:47:06.530433 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:06.530402    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l" podUID="ef4ef854-bdcb-419b-b4a7-ae0f4caedc69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 21 17:47:06.734642 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:06.734609    2563 generic.go:356] "Generic (PLEG): container finished" podID="aeeeed18-d5cb-44dc-98d8-e896038dee1b" containerID="7452dbc7929fea9bda0fd1f58e072e7bd11e89eaaf2631a1ad86999c16539f9a" exitCode=0
Jul 21 17:47:06.734784 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:06.734649    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6" event={"ID":"aeeeed18-d5cb-44dc-98d8-e896038dee1b","Type":"ContainerDied","Data":"7452dbc7929fea9bda0fd1f58e072e7bd11e89eaaf2631a1ad86999c16539f9a"}
Jul 21 17:47:06.735332 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:06.734936    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7" podUID="26f7402a-85bc-48c6-b8aa-2d192f06bd3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 21 17:47:06.735332 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:06.735030    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f64bd-predictor-7764f488c-c4ttx" podUID="80b3aae2-8538-4e58-8249-c708ca4fde57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 21 17:47:06.863047 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:06.863025    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6"
Jul 21 17:47:06.899578 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:06.899557    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aeeeed18-d5cb-44dc-98d8-e896038dee1b-kube-rbac-proxy-sar-config\") pod \"aeeeed18-d5cb-44dc-98d8-e896038dee1b\" (UID: \"aeeeed18-d5cb-44dc-98d8-e896038dee1b\") "
Jul 21 17:47:06.899673 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:06.899631    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aeeeed18-d5cb-44dc-98d8-e896038dee1b-proxy-tls\") pod \"aeeeed18-d5cb-44dc-98d8-e896038dee1b\" (UID: \"aeeeed18-d5cb-44dc-98d8-e896038dee1b\") "
Jul 21 17:47:06.899673 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:06.899653    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smrs6\" (UniqueName: \"kubernetes.io/projected/aeeeed18-d5cb-44dc-98d8-e896038dee1b-kube-api-access-smrs6\") pod \"aeeeed18-d5cb-44dc-98d8-e896038dee1b\" (UID: \"aeeeed18-d5cb-44dc-98d8-e896038dee1b\") "
Jul 21 17:47:06.900002 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:06.899975    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeeeed18-d5cb-44dc-98d8-e896038dee1b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "aeeeed18-d5cb-44dc-98d8-e896038dee1b" (UID: "aeeeed18-d5cb-44dc-98d8-e896038dee1b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:47:06.902580 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:06.902552    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeeeed18-d5cb-44dc-98d8-e896038dee1b-kube-api-access-smrs6" (OuterVolumeSpecName: "kube-api-access-smrs6") pod "aeeeed18-d5cb-44dc-98d8-e896038dee1b" (UID: "aeeeed18-d5cb-44dc-98d8-e896038dee1b"). InnerVolumeSpecName "kube-api-access-smrs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:47:06.902708 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:06.902604    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aeeeed18-d5cb-44dc-98d8-e896038dee1b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "aeeeed18-d5cb-44dc-98d8-e896038dee1b" (UID: "aeeeed18-d5cb-44dc-98d8-e896038dee1b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:47:06.965600 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:06.965570    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l"
Jul 21 17:47:07.000818 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:07.000791    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef4ef854-bdcb-419b-b4a7-ae0f4caedc69-kube-rbac-proxy-sar-config\") pod \"ef4ef854-bdcb-419b-b4a7-ae0f4caedc69\" (UID: \"ef4ef854-bdcb-419b-b4a7-ae0f4caedc69\") "
Jul 21 17:47:07.000973 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:07.000893    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28mnb\" (UniqueName: \"kubernetes.io/projected/ef4ef854-bdcb-419b-b4a7-ae0f4caedc69-kube-api-access-28mnb\") pod \"ef4ef854-bdcb-419b-b4a7-ae0f4caedc69\" (UID: \"ef4ef854-bdcb-419b-b4a7-ae0f4caedc69\") "
Jul 21 17:47:07.000973 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:07.000950    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef4ef854-bdcb-419b-b4a7-ae0f4caedc69-proxy-tls\") pod \"ef4ef854-bdcb-419b-b4a7-ae0f4caedc69\" (UID: \"ef4ef854-bdcb-419b-b4a7-ae0f4caedc69\") "
Jul 21 17:47:07.001220 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:07.001204    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aeeeed18-d5cb-44dc-98d8-e896038dee1b-proxy-tls\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:47:07.001262 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:07.001231    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smrs6\" (UniqueName: \"kubernetes.io/projected/aeeeed18-d5cb-44dc-98d8-e896038dee1b-kube-api-access-smrs6\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:47:07.001262 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:07.001244    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aeeeed18-d5cb-44dc-98d8-e896038dee1b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:47:07.001262 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:07.001203    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef4ef854-bdcb-419b-b4a7-ae0f4caedc69-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ef4ef854-bdcb-419b-b4a7-ae0f4caedc69" (UID: "ef4ef854-bdcb-419b-b4a7-ae0f4caedc69"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:47:07.003241 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:07.003216    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef4ef854-bdcb-419b-b4a7-ae0f4caedc69-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ef4ef854-bdcb-419b-b4a7-ae0f4caedc69" (UID: "ef4ef854-bdcb-419b-b4a7-ae0f4caedc69"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:47:07.003335 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:07.003214    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef4ef854-bdcb-419b-b4a7-ae0f4caedc69-kube-api-access-28mnb" (OuterVolumeSpecName: "kube-api-access-28mnb") pod "ef4ef854-bdcb-419b-b4a7-ae0f4caedc69" (UID: "ef4ef854-bdcb-419b-b4a7-ae0f4caedc69"). InnerVolumeSpecName "kube-api-access-28mnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:47:07.102507 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:07.102438    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef4ef854-bdcb-419b-b4a7-ae0f4caedc69-proxy-tls\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:47:07.102507 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:07.102462    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef4ef854-bdcb-419b-b4a7-ae0f4caedc69-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:47:07.102507 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:07.102471    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28mnb\" (UniqueName: \"kubernetes.io/projected/ef4ef854-bdcb-419b-b4a7-ae0f4caedc69-kube-api-access-28mnb\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:47:07.739604 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:07.739576    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6"
Jul 21 17:47:07.739604 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:07.739598    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6" event={"ID":"aeeeed18-d5cb-44dc-98d8-e896038dee1b","Type":"ContainerDied","Data":"1c17c9a54f749a2ce9b820778c1b467b0e65ac7455e4674721dfa4044c73a8f0"}
Jul 21 17:47:07.740112 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:07.739676    2563 scope.go:117] "RemoveContainer" containerID="3d2f7a01abcb50f5228bfb1aca202d30140d7ce4bb45386193a9cc46910537b8"
Jul 21 17:47:07.741181 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:07.741155    2563 generic.go:356] "Generic (PLEG): container finished" podID="ef4ef854-bdcb-419b-b4a7-ae0f4caedc69" containerID="244a840e829c5d11f54b697ef871553652af84ba2102dc15597cfd47ae486faf" exitCode=0
Jul 21 17:47:07.741312 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:07.741195    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l"
Jul 21 17:47:07.741312 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:07.741229    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l" event={"ID":"ef4ef854-bdcb-419b-b4a7-ae0f4caedc69","Type":"ContainerDied","Data":"244a840e829c5d11f54b697ef871553652af84ba2102dc15597cfd47ae486faf"}
Jul 21 17:47:07.741312 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:07.741283    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l" event={"ID":"ef4ef854-bdcb-419b-b4a7-ae0f4caedc69","Type":"ContainerDied","Data":"041d53034e1e26a85cf3a6748fc2d64931da6119aa4d34294b433dafca0a16c1"}
Jul 21 17:47:07.755895 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:07.749987    2563 scope.go:117] "RemoveContainer" containerID="7452dbc7929fea9bda0fd1f58e072e7bd11e89eaaf2631a1ad86999c16539f9a"
Jul 21 17:47:07.764658 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:07.764635    2563 scope.go:117] "RemoveContainer" containerID="5c511b41ad09bf8a6fd450fb0ac8b0bf4865195f94e08c5fbd61021ccbf44465"
Jul 21 17:47:07.765533 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:07.765514    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6"]
Jul 21 17:47:07.769188 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:07.769154    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5bf12-predictor-5dd8fcc8cf-qn9n6"]
Jul 21 17:47:07.772946 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:07.772931    2563 scope.go:117] "RemoveContainer" containerID="244a840e829c5d11f54b697ef871553652af84ba2102dc15597cfd47ae486faf"
Jul 21 17:47:07.780427 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:07.780410    2563 scope.go:117] "RemoveContainer" containerID="5c511b41ad09bf8a6fd450fb0ac8b0bf4865195f94e08c5fbd61021ccbf44465"
Jul 21 17:47:07.780712 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:47:07.780690    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c511b41ad09bf8a6fd450fb0ac8b0bf4865195f94e08c5fbd61021ccbf44465\": container with ID starting with 5c511b41ad09bf8a6fd450fb0ac8b0bf4865195f94e08c5fbd61021ccbf44465 not found: ID does not exist" containerID="5c511b41ad09bf8a6fd450fb0ac8b0bf4865195f94e08c5fbd61021ccbf44465"
Jul 21 17:47:07.780780 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:07.780719    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c511b41ad09bf8a6fd450fb0ac8b0bf4865195f94e08c5fbd61021ccbf44465"} err="failed to get container status \"5c511b41ad09bf8a6fd450fb0ac8b0bf4865195f94e08c5fbd61021ccbf44465\": rpc error: code = NotFound desc = could not find container \"5c511b41ad09bf8a6fd450fb0ac8b0bf4865195f94e08c5fbd61021ccbf44465\": container with ID starting with 5c511b41ad09bf8a6fd450fb0ac8b0bf4865195f94e08c5fbd61021ccbf44465 not found: ID does not exist"
Jul 21 17:47:07.780780 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:07.780761    2563 scope.go:117] "RemoveContainer" containerID="244a840e829c5d11f54b697ef871553652af84ba2102dc15597cfd47ae486faf"
Jul 21 17:47:07.781001 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:47:07.780983    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"244a840e829c5d11f54b697ef871553652af84ba2102dc15597cfd47ae486faf\": container with ID starting with 244a840e829c5d11f54b697ef871553652af84ba2102dc15597cfd47ae486faf not found: ID does not exist" containerID="244a840e829c5d11f54b697ef871553652af84ba2102dc15597cfd47ae486faf"
Jul 21 17:47:07.781057 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:07.781010    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"244a840e829c5d11f54b697ef871553652af84ba2102dc15597cfd47ae486faf"} err="failed to get container status \"244a840e829c5d11f54b697ef871553652af84ba2102dc15597cfd47ae486faf\": rpc error: code = NotFound desc = could not find container \"244a840e829c5d11f54b697ef871553652af84ba2102dc15597cfd47ae486faf\": container with ID starting with 244a840e829c5d11f54b697ef871553652af84ba2102dc15597cfd47ae486faf not found: ID does not exist"
Jul 21 17:47:07.781456 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:07.781438    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l"]
Jul 21 17:47:07.785288 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:07.785263    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5bf12-predictor-598dcbff9d-2wx7l"]
Jul 21 17:47:08.764531 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:08.764499    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aeeeed18-d5cb-44dc-98d8-e896038dee1b" path="/var/lib/kubelet/pods/aeeeed18-d5cb-44dc-98d8-e896038dee1b/volumes"
Jul 21 17:47:08.764898 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:08.764891    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef4ef854-bdcb-419b-b4a7-ae0f4caedc69" path="/var/lib/kubelet/pods/ef4ef854-bdcb-419b-b4a7-ae0f4caedc69/volumes"
Jul 21 17:47:11.739624 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:11.739595    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f64bd-predictor-7764f488c-c4ttx"
Jul 21 17:47:11.740107 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:11.739844    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7"
Jul 21 17:47:11.740107 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:11.740052    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f64bd-predictor-7764f488c-c4ttx" podUID="80b3aae2-8538-4e58-8249-c708ca4fde57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 21 17:47:11.740471 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:11.740452    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7" podUID="26f7402a-85bc-48c6-b8aa-2d192f06bd3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 21 17:47:21.740195 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:21.740154    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f64bd-predictor-7764f488c-c4ttx" podUID="80b3aae2-8538-4e58-8249-c708ca4fde57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 21 17:47:21.740649 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:21.740401    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7" podUID="26f7402a-85bc-48c6-b8aa-2d192f06bd3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 21 17:47:31.740267 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:31.740226    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f64bd-predictor-7764f488c-c4ttx" podUID="80b3aae2-8538-4e58-8249-c708ca4fde57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 21 17:47:31.740634 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:31.740464    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7" podUID="26f7402a-85bc-48c6-b8aa-2d192f06bd3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 21 17:47:33.475697 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.475664    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm"]
Jul 21 17:47:33.476206 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.475941    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm" podUID="9de3561f-aecd-4d01-afa8-60277198d9d1" containerName="kserve-container" containerID="cri-o://e4f6d8721fa2f9540446e5a6614dbdf7c035bab1f481fa35a611d0edefacd25a" gracePeriod=30
Jul 21 17:47:33.476350 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.476325    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm" podUID="9de3561f-aecd-4d01-afa8-60277198d9d1" containerName="kube-rbac-proxy" containerID="cri-o://2fda7a820f11725cfdc128e121170e26c6a967bba17cbf36c18b19f038d55f1a" gracePeriod=30
Jul 21 17:47:33.526312 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.526281    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8558d-predictor-5594879d74-cksjh"]
Jul 21 17:47:33.526524 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.526496    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8558d-predictor-5594879d74-cksjh" podUID="0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914" containerName="kserve-container" containerID="cri-o://b010d842ec367d0665206db19eb7144c4bebd42c902e3bfc40639c9a3e835090" gracePeriod=30
Jul 21 17:47:33.526524 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.526519    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8558d-predictor-5594879d74-cksjh" podUID="0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914" containerName="kube-rbac-proxy" containerID="cri-o://c233fcf6344213c2b6df031801e140b614e17f8d34da857bf5e3bff4d3574f01" gracePeriod=30
Jul 21 17:47:33.542436 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.542413    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b64c9-predictor-765f6b789f-47rtc"]
Jul 21 17:47:33.542743 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.542730    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeeeed18-d5cb-44dc-98d8-e896038dee1b" containerName="kserve-container"
Jul 21 17:47:33.542784 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.542745    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeeeed18-d5cb-44dc-98d8-e896038dee1b" containerName="kserve-container"
Jul 21 17:47:33.542784 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.542760    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeeeed18-d5cb-44dc-98d8-e896038dee1b" containerName="kube-rbac-proxy"
Jul 21 17:47:33.542784 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.542765    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeeeed18-d5cb-44dc-98d8-e896038dee1b" containerName="kube-rbac-proxy"
Jul 21 17:47:33.542784 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.542775    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef4ef854-bdcb-419b-b4a7-ae0f4caedc69" containerName="kube-rbac-proxy"
Jul 21 17:47:33.542784 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.542780    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef4ef854-bdcb-419b-b4a7-ae0f4caedc69" containerName="kube-rbac-proxy"
Jul 21 17:47:33.542933 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.542789    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef4ef854-bdcb-419b-b4a7-ae0f4caedc69" containerName="kserve-container"
Jul 21 17:47:33.542933 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.542795    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef4ef854-bdcb-419b-b4a7-ae0f4caedc69" containerName="kserve-container"
Jul 21 17:47:33.542933 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.542846    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef4ef854-bdcb-419b-b4a7-ae0f4caedc69" containerName="kube-rbac-proxy"
Jul 21 17:47:33.542933 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.542855    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef4ef854-bdcb-419b-b4a7-ae0f4caedc69" containerName="kserve-container"
Jul 21 17:47:33.542933 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.542863    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeeeed18-d5cb-44dc-98d8-e896038dee1b" containerName="kserve-container"
Jul 21 17:47:33.542933 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.542869    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeeeed18-d5cb-44dc-98d8-e896038dee1b" containerName="kube-rbac-proxy"
Jul 21 17:47:33.547561 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.547539    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b64c9-predictor-765f6b789f-47rtc"
Jul 21 17:47:33.549815 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.549789    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-b64c9-kube-rbac-proxy-sar-config\""
Jul 21 17:47:33.549914 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.549862    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-b64c9-predictor-serving-cert\""
Jul 21 17:47:33.554404 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.554205    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b64c9-predictor-765f6b789f-47rtc"]
Jul 21 17:47:33.634554 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.634515    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07fc8a39-1f80-4327-bf72-32bb289206e5-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b64c9-predictor-765f6b789f-47rtc\" (UID: \"07fc8a39-1f80-4327-bf72-32bb289206e5\") " pod="kserve-ci-e2e-test/success-200-isvc-b64c9-predictor-765f6b789f-47rtc"
Jul 21 17:47:33.634712 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.634568    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07fc8a39-1f80-4327-bf72-32bb289206e5-proxy-tls\") pod \"success-200-isvc-b64c9-predictor-765f6b789f-47rtc\" (UID: \"07fc8a39-1f80-4327-bf72-32bb289206e5\") " pod="kserve-ci-e2e-test/success-200-isvc-b64c9-predictor-765f6b789f-47rtc"
Jul 21 17:47:33.634712 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.634681    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhwgp\" (UniqueName: \"kubernetes.io/projected/07fc8a39-1f80-4327-bf72-32bb289206e5-kube-api-access-rhwgp\") pod \"success-200-isvc-b64c9-predictor-765f6b789f-47rtc\" (UID: \"07fc8a39-1f80-4327-bf72-32bb289206e5\") " pod="kserve-ci-e2e-test/success-200-isvc-b64c9-predictor-765f6b789f-47rtc"
Jul 21 17:47:33.641935 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.641907    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b64c9-predictor-f779895b6-4rwsl"]
Jul 21 17:47:33.645201 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.645184    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b64c9-predictor-f779895b6-4rwsl"
Jul 21 17:47:33.648900 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.648704    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-b64c9-predictor-serving-cert\""
Jul 21 17:47:33.648900 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.648803    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-b64c9-kube-rbac-proxy-sar-config\""
Jul 21 17:47:33.657192 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.655786    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b64c9-predictor-f779895b6-4rwsl"]
Jul 21 17:47:33.736207 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.736119    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26970d94-f960-4da3-a5fe-adfa5d370484-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b64c9-predictor-f779895b6-4rwsl\" (UID: \"26970d94-f960-4da3-a5fe-adfa5d370484\") " pod="kserve-ci-e2e-test/error-404-isvc-b64c9-predictor-f779895b6-4rwsl"
Jul 21 17:47:33.736356 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.736240    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07fc8a39-1f80-4327-bf72-32bb289206e5-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b64c9-predictor-765f6b789f-47rtc\" (UID: \"07fc8a39-1f80-4327-bf72-32bb289206e5\") " pod="kserve-ci-e2e-test/success-200-isvc-b64c9-predictor-765f6b789f-47rtc"
Jul 21 17:47:33.736356 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.736275    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26970d94-f960-4da3-a5fe-adfa5d370484-proxy-tls\") pod \"error-404-isvc-b64c9-predictor-f779895b6-4rwsl\" (UID: \"26970d94-f960-4da3-a5fe-adfa5d370484\") " pod="kserve-ci-e2e-test/error-404-isvc-b64c9-predictor-f779895b6-4rwsl"
Jul 21 17:47:33.736356 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.736295    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8j4h\" (UniqueName: \"kubernetes.io/projected/26970d94-f960-4da3-a5fe-adfa5d370484-kube-api-access-b8j4h\") pod \"error-404-isvc-b64c9-predictor-f779895b6-4rwsl\" (UID: \"26970d94-f960-4da3-a5fe-adfa5d370484\") " pod="kserve-ci-e2e-test/error-404-isvc-b64c9-predictor-f779895b6-4rwsl"
Jul 21 17:47:33.736356 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.736313    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07fc8a39-1f80-4327-bf72-32bb289206e5-proxy-tls\") pod \"success-200-isvc-b64c9-predictor-765f6b789f-47rtc\" (UID: \"07fc8a39-1f80-4327-bf72-32bb289206e5\") " pod="kserve-ci-e2e-test/success-200-isvc-b64c9-predictor-765f6b789f-47rtc"
Jul 21 17:47:33.736356 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.736348    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhwgp\" (UniqueName: \"kubernetes.io/projected/07fc8a39-1f80-4327-bf72-32bb289206e5-kube-api-access-rhwgp\") pod \"success-200-isvc-b64c9-predictor-765f6b789f-47rtc\" (UID: \"07fc8a39-1f80-4327-bf72-32bb289206e5\") " pod="kserve-ci-e2e-test/success-200-isvc-b64c9-predictor-765f6b789f-47rtc"
Jul 21 17:47:33.741774 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:47:33.736925    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-b64c9-predictor-serving-cert: secret "success-200-isvc-b64c9-predictor-serving-cert" not found
Jul 21 17:47:33.741774 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:47:33.737011    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07fc8a39-1f80-4327-bf72-32bb289206e5-proxy-tls podName:07fc8a39-1f80-4327-bf72-32bb289206e5 nodeName:}" failed. No retries permitted until 2026-07-21 17:47:34.236991018 +0000 UTC m=+1281.999130600 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/07fc8a39-1f80-4327-bf72-32bb289206e5-proxy-tls") pod "success-200-isvc-b64c9-predictor-765f6b789f-47rtc" (UID: "07fc8a39-1f80-4327-bf72-32bb289206e5") : secret "success-200-isvc-b64c9-predictor-serving-cert" not found
Jul 21 17:47:33.741774 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.737347    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07fc8a39-1f80-4327-bf72-32bb289206e5-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b64c9-predictor-765f6b789f-47rtc\" (UID: \"07fc8a39-1f80-4327-bf72-32bb289206e5\") " pod="kserve-ci-e2e-test/success-200-isvc-b64c9-predictor-765f6b789f-47rtc"
Jul 21 17:47:33.745369 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.745346    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhwgp\" (UniqueName: \"kubernetes.io/projected/07fc8a39-1f80-4327-bf72-32bb289206e5-kube-api-access-rhwgp\") pod \"success-200-isvc-b64c9-predictor-765f6b789f-47rtc\" (UID: \"07fc8a39-1f80-4327-bf72-32bb289206e5\") " pod="kserve-ci-e2e-test/success-200-isvc-b64c9-predictor-765f6b789f-47rtc"
Jul 21 17:47:33.837473 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.837435    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26970d94-f960-4da3-a5fe-adfa5d370484-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b64c9-predictor-f779895b6-4rwsl\" (UID: \"26970d94-f960-4da3-a5fe-adfa5d370484\") " pod="kserve-ci-e2e-test/error-404-isvc-b64c9-predictor-f779895b6-4rwsl"
Jul 21 17:47:33.837604 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.837508    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26970d94-f960-4da3-a5fe-adfa5d370484-proxy-tls\") pod \"error-404-isvc-b64c9-predictor-f779895b6-4rwsl\" (UID: \"26970d94-f960-4da3-a5fe-adfa5d370484\") " pod="kserve-ci-e2e-test/error-404-isvc-b64c9-predictor-f779895b6-4rwsl"
Jul 21 17:47:33.837604 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.837526    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8j4h\" (UniqueName: \"kubernetes.io/projected/26970d94-f960-4da3-a5fe-adfa5d370484-kube-api-access-b8j4h\") pod \"error-404-isvc-b64c9-predictor-f779895b6-4rwsl\" (UID: \"26970d94-f960-4da3-a5fe-adfa5d370484\") " pod="kserve-ci-e2e-test/error-404-isvc-b64c9-predictor-f779895b6-4rwsl"
Jul 21 17:47:33.837719 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:47:33.837668    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-b64c9-predictor-serving-cert: secret "error-404-isvc-b64c9-predictor-serving-cert" not found
Jul 21 17:47:33.837754 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:47:33.837724    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26970d94-f960-4da3-a5fe-adfa5d370484-proxy-tls podName:26970d94-f960-4da3-a5fe-adfa5d370484 nodeName:}" failed. No retries permitted until 2026-07-21 17:47:34.337709096 +0000 UTC m=+1282.099848678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/26970d94-f960-4da3-a5fe-adfa5d370484-proxy-tls") pod "error-404-isvc-b64c9-predictor-f779895b6-4rwsl" (UID: "26970d94-f960-4da3-a5fe-adfa5d370484") : secret "error-404-isvc-b64c9-predictor-serving-cert" not found
Jul 21 17:47:33.838062 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.838044    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26970d94-f960-4da3-a5fe-adfa5d370484-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b64c9-predictor-f779895b6-4rwsl\" (UID: \"26970d94-f960-4da3-a5fe-adfa5d370484\") " pod="kserve-ci-e2e-test/error-404-isvc-b64c9-predictor-f779895b6-4rwsl"
Jul 21 17:47:33.844893 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.844868    2563 generic.go:356] "Generic (PLEG): container finished" podID="9de3561f-aecd-4d01-afa8-60277198d9d1" containerID="2fda7a820f11725cfdc128e121170e26c6a967bba17cbf36c18b19f038d55f1a" exitCode=2
Jul 21 17:47:33.845006 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.844945    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm" event={"ID":"9de3561f-aecd-4d01-afa8-60277198d9d1","Type":"ContainerDied","Data":"2fda7a820f11725cfdc128e121170e26c6a967bba17cbf36c18b19f038d55f1a"}
Jul 21 17:47:33.846651 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.846631    2563 generic.go:356] "Generic (PLEG): container finished" podID="0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914" containerID="c233fcf6344213c2b6df031801e140b614e17f8d34da857bf5e3bff4d3574f01" exitCode=2
Jul 21 17:47:33.846734 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.846662    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8558d-predictor-5594879d74-cksjh" event={"ID":"0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914","Type":"ContainerDied","Data":"c233fcf6344213c2b6df031801e140b614e17f8d34da857bf5e3bff4d3574f01"}
Jul 21 17:47:33.848176 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:33.848151    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8j4h\" (UniqueName: \"kubernetes.io/projected/26970d94-f960-4da3-a5fe-adfa5d370484-kube-api-access-b8j4h\") pod \"error-404-isvc-b64c9-predictor-f779895b6-4rwsl\" (UID: \"26970d94-f960-4da3-a5fe-adfa5d370484\") " pod="kserve-ci-e2e-test/error-404-isvc-b64c9-predictor-f779895b6-4rwsl"
Jul 21 17:47:34.240479 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:34.240447    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07fc8a39-1f80-4327-bf72-32bb289206e5-proxy-tls\") pod \"success-200-isvc-b64c9-predictor-765f6b789f-47rtc\" (UID: \"07fc8a39-1f80-4327-bf72-32bb289206e5\") " pod="kserve-ci-e2e-test/success-200-isvc-b64c9-predictor-765f6b789f-47rtc"
Jul 21 17:47:34.242967 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:34.242939    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07fc8a39-1f80-4327-bf72-32bb289206e5-proxy-tls\") pod \"success-200-isvc-b64c9-predictor-765f6b789f-47rtc\" (UID: \"07fc8a39-1f80-4327-bf72-32bb289206e5\") " pod="kserve-ci-e2e-test/success-200-isvc-b64c9-predictor-765f6b789f-47rtc"
Jul 21 17:47:34.341324 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:34.341288    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26970d94-f960-4da3-a5fe-adfa5d370484-proxy-tls\") pod \"error-404-isvc-b64c9-predictor-f779895b6-4rwsl\" (UID: \"26970d94-f960-4da3-a5fe-adfa5d370484\") " pod="kserve-ci-e2e-test/error-404-isvc-b64c9-predictor-f779895b6-4rwsl"
Jul 21 17:47:34.343753 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:34.343730    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26970d94-f960-4da3-a5fe-adfa5d370484-proxy-tls\") pod \"error-404-isvc-b64c9-predictor-f779895b6-4rwsl\" (UID: \"26970d94-f960-4da3-a5fe-adfa5d370484\") " pod="kserve-ci-e2e-test/error-404-isvc-b64c9-predictor-f779895b6-4rwsl"
Jul 21 17:47:34.459561 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:34.459532    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b64c9-predictor-765f6b789f-47rtc"
Jul 21 17:47:34.557583 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:34.557321    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b64c9-predictor-f779895b6-4rwsl"
Jul 21 17:47:34.590822 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:34.590798    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b64c9-predictor-765f6b789f-47rtc"]
Jul 21 17:47:34.593475 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:47:34.593446    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07fc8a39_1f80_4327_bf72_32bb289206e5.slice/crio-3c92600805771262427cd2fa38f9cbfc6dadd724f6364fe5a18174b0a9c663bc WatchSource:0}: Error finding container 3c92600805771262427cd2fa38f9cbfc6dadd724f6364fe5a18174b0a9c663bc: Status 404 returned error can't find the container with id 3c92600805771262427cd2fa38f9cbfc6dadd724f6364fe5a18174b0a9c663bc
Jul 21 17:47:34.703308 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:34.703285    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b64c9-predictor-f779895b6-4rwsl"]
Jul 21 17:47:34.704770 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:47:34.704748    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26970d94_f960_4da3_a5fe_adfa5d370484.slice/crio-371f723d155fdc0777dd5fc68279c17ec74f30a5fbaeaa6b01ee2ead472f34b8 WatchSource:0}: Error finding container 371f723d155fdc0777dd5fc68279c17ec74f30a5fbaeaa6b01ee2ead472f34b8: Status 404 returned error can't find the container with id 371f723d155fdc0777dd5fc68279c17ec74f30a5fbaeaa6b01ee2ead472f34b8
Jul 21 17:47:34.855642 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:34.854629    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b64c9-predictor-765f6b789f-47rtc" event={"ID":"07fc8a39-1f80-4327-bf72-32bb289206e5","Type":"ContainerStarted","Data":"fbcbd37ad94553c2d170eecec3545ae69ab8fedbe6233c58acbcd6efebf2a44e"}
Jul 21 17:47:34.855642 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:34.854701    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b64c9-predictor-765f6b789f-47rtc" event={"ID":"07fc8a39-1f80-4327-bf72-32bb289206e5","Type":"ContainerStarted","Data":"e9da7d49714f27df5ba6e9daba22919a7574caf8c836eb5248152495861af925"}
Jul 21 17:47:34.855642 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:34.854731    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b64c9-predictor-765f6b789f-47rtc" event={"ID":"07fc8a39-1f80-4327-bf72-32bb289206e5","Type":"ContainerStarted","Data":"3c92600805771262427cd2fa38f9cbfc6dadd724f6364fe5a18174b0a9c663bc"}
Jul 21 17:47:34.855642 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:34.855421    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b64c9-predictor-765f6b789f-47rtc"
Jul 21 17:47:34.855642 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:34.855504    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b64c9-predictor-765f6b789f-47rtc"
Jul 21 17:47:34.857122 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:34.857061    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b64c9-predictor-765f6b789f-47rtc" podUID="07fc8a39-1f80-4327-bf72-32bb289206e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 21 17:47:34.858215 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:34.858119    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b64c9-predictor-f779895b6-4rwsl" event={"ID":"26970d94-f960-4da3-a5fe-adfa5d370484","Type":"ContainerStarted","Data":"cd19330596b1be5239337196757330fd5e6df488d278aca33d2d4e02876a5c74"}
Jul 21 17:47:34.858332 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:34.858235    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b64c9-predictor-f779895b6-4rwsl" event={"ID":"26970d94-f960-4da3-a5fe-adfa5d370484","Type":"ContainerStarted","Data":"487a55beb06d1ff3a5f9bdddd2a310a22f82f98cdd09a7c56662e4cde9f5755d"}
Jul 21 17:47:34.858332 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:34.858274    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b64c9-predictor-f779895b6-4rwsl" event={"ID":"26970d94-f960-4da3-a5fe-adfa5d370484","Type":"ContainerStarted","Data":"371f723d155fdc0777dd5fc68279c17ec74f30a5fbaeaa6b01ee2ead472f34b8"}
Jul 21 17:47:34.859343 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:34.859320    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-b64c9-predictor-f779895b6-4rwsl"
Jul 21 17:47:34.872587 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:34.872542    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-b64c9-predictor-765f6b789f-47rtc" podStartSLOduration=1.872530035 podStartE2EDuration="1.872530035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:47:34.871408599 +0000 UTC m=+1282.633548202" watchObservedRunningTime="2026-07-21 17:47:34.872530035 +0000 UTC m=+1282.634669639"
Jul 21 17:47:34.888742 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:34.888702    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-b64c9-predictor-f779895b6-4rwsl" podStartSLOduration=1.888688328 podStartE2EDuration="1.888688328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:47:34.887491606 +0000 UTC m=+1282.649631209" watchObservedRunningTime="2026-07-21 17:47:34.888688328 +0000 UTC m=+1282.650827931"
Jul 21 17:47:35.861800 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:35.861762    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b64c9-predictor-765f6b789f-47rtc" podUID="07fc8a39-1f80-4327-bf72-32bb289206e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 21 17:47:35.862258 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:35.861875    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-b64c9-predictor-f779895b6-4rwsl"
Jul 21 17:47:35.862972 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:35.862946    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b64c9-predictor-f779895b6-4rwsl" podUID="26970d94-f960-4da3-a5fe-adfa5d370484" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 21 17:47:36.679099 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:36.679055    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm" podUID="9de3561f-aecd-4d01-afa8-60277198d9d1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Jul 21 17:47:36.684511 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:36.684473    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm" podUID="9de3561f-aecd-4d01-afa8-60277198d9d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 21 17:47:36.865998 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:36.865959    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b64c9-predictor-765f6b789f-47rtc" podUID="07fc8a39-1f80-4327-bf72-32bb289206e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 21 17:47:36.866461 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:36.866053    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b64c9-predictor-f779895b6-4rwsl" podUID="26970d94-f960-4da3-a5fe-adfa5d370484" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 21 17:47:37.308384 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.308320    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8558d-predictor-5594879d74-cksjh"
Jul 21 17:47:37.424301 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.424276    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm"
Jul 21 17:47:37.469337 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.469304    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914-kube-rbac-proxy-sar-config\") pod \"0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914\" (UID: \"0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914\") "
Jul 21 17:47:37.469527 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.469386    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6xt6\" (UniqueName: \"kubernetes.io/projected/0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914-kube-api-access-v6xt6\") pod \"0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914\" (UID: \"0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914\") "
Jul 21 17:47:37.469527 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.469467    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914-proxy-tls\") pod \"0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914\" (UID: \"0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914\") "
Jul 21 17:47:37.469660 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.469636    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914" (UID: "0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:47:37.471558 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.471526    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914" (UID: "0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:47:37.471656 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.471604    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914-kube-api-access-v6xt6" (OuterVolumeSpecName: "kube-api-access-v6xt6") pod "0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914" (UID: "0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914"). InnerVolumeSpecName "kube-api-access-v6xt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:47:37.570636 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.570544    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wjbt\" (UniqueName: \"kubernetes.io/projected/9de3561f-aecd-4d01-afa8-60277198d9d1-kube-api-access-7wjbt\") pod \"9de3561f-aecd-4d01-afa8-60277198d9d1\" (UID: \"9de3561f-aecd-4d01-afa8-60277198d9d1\") "
Jul 21 17:47:37.570636 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.570600    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9de3561f-aecd-4d01-afa8-60277198d9d1-proxy-tls\") pod \"9de3561f-aecd-4d01-afa8-60277198d9d1\" (UID: \"9de3561f-aecd-4d01-afa8-60277198d9d1\") "
Jul 21 17:47:37.570855 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.570739    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9de3561f-aecd-4d01-afa8-60277198d9d1-kube-rbac-proxy-sar-config\") pod \"9de3561f-aecd-4d01-afa8-60277198d9d1\" (UID: \"9de3561f-aecd-4d01-afa8-60277198d9d1\") "
Jul 21 17:47:37.570994 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.570977    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6xt6\" (UniqueName: \"kubernetes.io/projected/0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914-kube-api-access-v6xt6\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:47:37.571058 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.570999    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914-proxy-tls\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:47:37.571058 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.571018    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:47:37.571058 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.571019    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9de3561f-aecd-4d01-afa8-60277198d9d1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9de3561f-aecd-4d01-afa8-60277198d9d1" (UID: "9de3561f-aecd-4d01-afa8-60277198d9d1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:47:37.572776 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.572756    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9de3561f-aecd-4d01-afa8-60277198d9d1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9de3561f-aecd-4d01-afa8-60277198d9d1" (UID: "9de3561f-aecd-4d01-afa8-60277198d9d1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:47:37.572927 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.572908    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9de3561f-aecd-4d01-afa8-60277198d9d1-kube-api-access-7wjbt" (OuterVolumeSpecName: "kube-api-access-7wjbt") pod "9de3561f-aecd-4d01-afa8-60277198d9d1" (UID: "9de3561f-aecd-4d01-afa8-60277198d9d1"). InnerVolumeSpecName "kube-api-access-7wjbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:47:37.671540 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.671505    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wjbt\" (UniqueName: \"kubernetes.io/projected/9de3561f-aecd-4d01-afa8-60277198d9d1-kube-api-access-7wjbt\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:47:37.671540 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.671537    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9de3561f-aecd-4d01-afa8-60277198d9d1-proxy-tls\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:47:37.671540 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.671547    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9de3561f-aecd-4d01-afa8-60277198d9d1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:47:37.870842 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.870804    2563 generic.go:356] "Generic (PLEG): container finished" podID="0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914" containerID="b010d842ec367d0665206db19eb7144c4bebd42c902e3bfc40639c9a3e835090" exitCode=0
Jul 21 17:47:37.871299 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.870857    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8558d-predictor-5594879d74-cksjh"
Jul 21 17:47:37.871299 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.870892    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8558d-predictor-5594879d74-cksjh" event={"ID":"0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914","Type":"ContainerDied","Data":"b010d842ec367d0665206db19eb7144c4bebd42c902e3bfc40639c9a3e835090"}
Jul 21 17:47:37.871299 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.870948    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8558d-predictor-5594879d74-cksjh" event={"ID":"0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914","Type":"ContainerDied","Data":"242623dd85bc26f070e9db14487bba00e1d7f7fdb616bb6eb594120c37894186"}
Jul 21 17:47:37.871299 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.870975    2563 scope.go:117] "RemoveContainer" containerID="c233fcf6344213c2b6df031801e140b614e17f8d34da857bf5e3bff4d3574f01"
Jul 21 17:47:37.872708 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.872686    2563 generic.go:356] "Generic (PLEG): container finished" podID="9de3561f-aecd-4d01-afa8-60277198d9d1" containerID="e4f6d8721fa2f9540446e5a6614dbdf7c035bab1f481fa35a611d0edefacd25a" exitCode=0
Jul 21 17:47:37.872808 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.872754    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm" event={"ID":"9de3561f-aecd-4d01-afa8-60277198d9d1","Type":"ContainerDied","Data":"e4f6d8721fa2f9540446e5a6614dbdf7c035bab1f481fa35a611d0edefacd25a"}
Jul 21 17:47:37.872808 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.872804    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm" event={"ID":"9de3561f-aecd-4d01-afa8-60277198d9d1","Type":"ContainerDied","Data":"cfcd10935550433bd7389fd18d0d315d71712d0057eea965b460c917c66773de"}
Jul 21 17:47:37.872884 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.872821    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm"
Jul 21 17:47:37.886871 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.886847    2563 scope.go:117] "RemoveContainer" containerID="b010d842ec367d0665206db19eb7144c4bebd42c902e3bfc40639c9a3e835090"
Jul 21 17:47:37.894432 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.894416    2563 scope.go:117] "RemoveContainer" containerID="c233fcf6344213c2b6df031801e140b614e17f8d34da857bf5e3bff4d3574f01"
Jul 21 17:47:37.894704 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:47:37.894686    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c233fcf6344213c2b6df031801e140b614e17f8d34da857bf5e3bff4d3574f01\": container with ID starting with c233fcf6344213c2b6df031801e140b614e17f8d34da857bf5e3bff4d3574f01 not found: ID does not exist" containerID="c233fcf6344213c2b6df031801e140b614e17f8d34da857bf5e3bff4d3574f01"
Jul 21 17:47:37.894747 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.894712    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c233fcf6344213c2b6df031801e140b614e17f8d34da857bf5e3bff4d3574f01"} err="failed to get container status \"c233fcf6344213c2b6df031801e140b614e17f8d34da857bf5e3bff4d3574f01\": rpc error: code = NotFound desc = could not find container \"c233fcf6344213c2b6df031801e140b614e17f8d34da857bf5e3bff4d3574f01\": container with ID starting with c233fcf6344213c2b6df031801e140b614e17f8d34da857bf5e3bff4d3574f01 not found: ID does not exist"
Jul 21 17:47:37.894747 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.894731    2563 scope.go:117] "RemoveContainer" containerID="b010d842ec367d0665206db19eb7144c4bebd42c902e3bfc40639c9a3e835090"
Jul 21 17:47:37.894977 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:47:37.894960    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b010d842ec367d0665206db19eb7144c4bebd42c902e3bfc40639c9a3e835090\": container with ID starting with b010d842ec367d0665206db19eb7144c4bebd42c902e3bfc40639c9a3e835090 not found: ID does not exist" containerID="b010d842ec367d0665206db19eb7144c4bebd42c902e3bfc40639c9a3e835090"
Jul 21 17:47:37.895026 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.894982    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b010d842ec367d0665206db19eb7144c4bebd42c902e3bfc40639c9a3e835090"} err="failed to get container status \"b010d842ec367d0665206db19eb7144c4bebd42c902e3bfc40639c9a3e835090\": rpc error: code = NotFound desc = could not find container \"b010d842ec367d0665206db19eb7144c4bebd42c902e3bfc40639c9a3e835090\": container with ID starting with b010d842ec367d0665206db19eb7144c4bebd42c902e3bfc40639c9a3e835090 not found: ID does not exist"
Jul 21 17:47:37.895026 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.894996    2563 scope.go:117] "RemoveContainer" containerID="2fda7a820f11725cfdc128e121170e26c6a967bba17cbf36c18b19f038d55f1a"
Jul 21 17:47:37.898275 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.898253    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8558d-predictor-5594879d74-cksjh"]
Jul 21 17:47:37.902103 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.902084    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8558d-predictor-5594879d74-cksjh"]
Jul 21 17:47:37.903290 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.903267    2563 scope.go:117] "RemoveContainer" containerID="e4f6d8721fa2f9540446e5a6614dbdf7c035bab1f481fa35a611d0edefacd25a"
Jul 21 17:47:37.910482 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.910464    2563 scope.go:117] "RemoveContainer" containerID="2fda7a820f11725cfdc128e121170e26c6a967bba17cbf36c18b19f038d55f1a"
Jul 21 17:47:37.910756 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:47:37.910738    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fda7a820f11725cfdc128e121170e26c6a967bba17cbf36c18b19f038d55f1a\": container with ID starting with 2fda7a820f11725cfdc128e121170e26c6a967bba17cbf36c18b19f038d55f1a not found: ID does not exist" containerID="2fda7a820f11725cfdc128e121170e26c6a967bba17cbf36c18b19f038d55f1a"
Jul 21 17:47:37.910798 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.910765    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fda7a820f11725cfdc128e121170e26c6a967bba17cbf36c18b19f038d55f1a"} err="failed to get container status \"2fda7a820f11725cfdc128e121170e26c6a967bba17cbf36c18b19f038d55f1a\": rpc error: code = NotFound desc = could not find container \"2fda7a820f11725cfdc128e121170e26c6a967bba17cbf36c18b19f038d55f1a\": container with ID starting with 2fda7a820f11725cfdc128e121170e26c6a967bba17cbf36c18b19f038d55f1a not found: ID does not exist"
Jul 21 17:47:37.910798 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.910783    2563 scope.go:117] "RemoveContainer" containerID="e4f6d8721fa2f9540446e5a6614dbdf7c035bab1f481fa35a611d0edefacd25a"
Jul 21 17:47:37.911017 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:47:37.910997    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4f6d8721fa2f9540446e5a6614dbdf7c035bab1f481fa35a611d0edefacd25a\": container with ID starting with e4f6d8721fa2f9540446e5a6614dbdf7c035bab1f481fa35a611d0edefacd25a not found: ID does not exist" containerID="e4f6d8721fa2f9540446e5a6614dbdf7c035bab1f481fa35a611d0edefacd25a"
Jul 21 17:47:37.911070 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.911022    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4f6d8721fa2f9540446e5a6614dbdf7c035bab1f481fa35a611d0edefacd25a"} err="failed to get container status \"e4f6d8721fa2f9540446e5a6614dbdf7c035bab1f481fa35a611d0edefacd25a\": rpc error: code = NotFound desc = could not find container \"e4f6d8721fa2f9540446e5a6614dbdf7c035bab1f481fa35a611d0edefacd25a\": container with ID starting with e4f6d8721fa2f9540446e5a6614dbdf7c035bab1f481fa35a611d0edefacd25a not found: ID does not exist"
Jul 21 17:47:37.913663 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.913640    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm"]
Jul 21 17:47:37.917453 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:37.917431    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8558d-predictor-6df9dd9cf5-bn5nm"]
Jul 21 17:47:38.764279 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:38.764247    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914" path="/var/lib/kubelet/pods/0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914/volumes"
Jul 21 17:47:38.764665 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:38.764653    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9de3561f-aecd-4d01-afa8-60277198d9d1" path="/var/lib/kubelet/pods/9de3561f-aecd-4d01-afa8-60277198d9d1/volumes"
Jul 21 17:47:41.740261 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:41.740228    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f64bd-predictor-7764f488c-c4ttx" podUID="80b3aae2-8538-4e58-8249-c708ca4fde57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 21 17:47:41.740708 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:41.740503    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7" podUID="26f7402a-85bc-48c6-b8aa-2d192f06bd3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 21 17:47:41.869929 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:41.869904    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b64c9-predictor-765f6b789f-47rtc"
Jul 21 17:47:41.870093 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:41.869986    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-b64c9-predictor-f779895b6-4rwsl"
Jul 21 17:47:41.870415 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:41.870386    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b64c9-predictor-765f6b789f-47rtc" podUID="07fc8a39-1f80-4327-bf72-32bb289206e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 21 17:47:41.870490 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:41.870408    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b64c9-predictor-f779895b6-4rwsl" podUID="26970d94-f960-4da3-a5fe-adfa5d370484" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 21 17:47:51.741312 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:51.741274    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f64bd-predictor-7764f488c-c4ttx"
Jul 21 17:47:51.743449 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:51.741384    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7"
Jul 21 17:47:51.870408 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:51.870374    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b64c9-predictor-f779895b6-4rwsl" podUID="26970d94-f960-4da3-a5fe-adfa5d370484" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 21 17:47:51.870588 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:47:51.870370    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b64c9-predictor-765f6b789f-47rtc" podUID="07fc8a39-1f80-4327-bf72-32bb289206e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 21 17:48:01.870764 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:01.870696    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b64c9-predictor-f779895b6-4rwsl" podUID="26970d94-f960-4da3-a5fe-adfa5d370484" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 21 17:48:01.871276 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:01.871181    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b64c9-predictor-765f6b789f-47rtc" podUID="07fc8a39-1f80-4327-bf72-32bb289206e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 21 17:48:11.870537 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:11.870490    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b64c9-predictor-765f6b789f-47rtc" podUID="07fc8a39-1f80-4327-bf72-32bb289206e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 21 17:48:11.871069 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:11.870490    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b64c9-predictor-f779895b6-4rwsl" podUID="26970d94-f960-4da3-a5fe-adfa5d370484" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 21 17:48:13.709738 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:13.709708    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7"]
Jul 21 17:48:13.710223 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:13.709934    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7" podUID="26f7402a-85bc-48c6-b8aa-2d192f06bd3c" containerName="kserve-container" containerID="cri-o://6d523e03444ae1a1a609b35bd052fe16cafbd38434655e868421bb9a16e7b8b1" gracePeriod=30
Jul 21 17:48:13.710223 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:13.709998    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7" podUID="26f7402a-85bc-48c6-b8aa-2d192f06bd3c" containerName="kube-rbac-proxy" containerID="cri-o://3a6ebe019aca80f430745d2da2e88b0f05468bf34c060f86780436e792bf5e62" gracePeriod=30
Jul 21 17:48:13.771650 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:13.771622    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f64bd-predictor-7764f488c-c4ttx"]
Jul 21 17:48:13.771876 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:13.771850    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f64bd-predictor-7764f488c-c4ttx" podUID="80b3aae2-8538-4e58-8249-c708ca4fde57" containerName="kserve-container" containerID="cri-o://744c692918959eab78fa52227bbe74fe029a960aff6d49a324fccfd304bda384" gracePeriod=30
Jul 21 17:48:13.771925 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:13.771871    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f64bd-predictor-7764f488c-c4ttx" podUID="80b3aae2-8538-4e58-8249-c708ca4fde57" containerName="kube-rbac-proxy" containerID="cri-o://c61f19bd973ee0986a8fc94772f309bda529c7a6b4eb2960af40737924463c6d" gracePeriod=30
Jul 21 17:48:13.829774 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:13.829740    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg"]
Jul 21 17:48:13.830087 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:13.830072    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914" containerName="kserve-container"
Jul 21 17:48:13.830087 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:13.830087    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914" containerName="kserve-container"
Jul 21 17:48:13.830252 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:13.830096    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914" containerName="kube-rbac-proxy"
Jul 21 17:48:13.830252 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:13.830102    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914" containerName="kube-rbac-proxy"
Jul 21 17:48:13.830252 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:13.830111    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9de3561f-aecd-4d01-afa8-60277198d9d1" containerName="kserve-container"
Jul 21 17:48:13.830252 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:13.830116    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9de3561f-aecd-4d01-afa8-60277198d9d1" containerName="kserve-container"
Jul 21 17:48:13.830252 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:13.830125    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9de3561f-aecd-4d01-afa8-60277198d9d1" containerName="kube-rbac-proxy"
Jul 21 17:48:13.830252 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:13.830149    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9de3561f-aecd-4d01-afa8-60277198d9d1" containerName="kube-rbac-proxy"
Jul 21 17:48:13.830252 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:13.830214    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914" containerName="kserve-container"
Jul 21 17:48:13.830252 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:13.830222    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9de3561f-aecd-4d01-afa8-60277198d9d1" containerName="kube-rbac-proxy"
Jul 21 17:48:13.830252 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:13.830228    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d3e5f2a-fb6e-48d6-bb0c-d7c9eaf25914" containerName="kube-rbac-proxy"
Jul 21 17:48:13.830252 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:13.830233    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9de3561f-aecd-4d01-afa8-60277198d9d1" containerName="kserve-container"
Jul 21 17:48:13.833168 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:13.833122    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg"
Jul 21 17:48:13.835390 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:13.835372    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4c9b0-predictor-serving-cert\""
Jul 21 17:48:13.835495 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:13.835421    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4c9b0-kube-rbac-proxy-sar-config\""
Jul 21 17:48:13.843675 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:13.843652    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg"]
Jul 21 17:48:13.929345 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:13.929313    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr"]
Jul 21 17:48:13.932449 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:13.932431    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr"
Jul 21 17:48:13.934717 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:13.934699    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4c9b0-predictor-serving-cert\""
Jul 21 17:48:13.934899 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:13.934881    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4c9b0-kube-rbac-proxy-sar-config\""
Jul 21 17:48:13.942713 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:13.942693    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr"]
Jul 21 17:48:13.976860 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:13.976779    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/584fd906-8440-4595-9371-2482e6946cdb-proxy-tls\") pod \"success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg\" (UID: \"584fd906-8440-4595-9371-2482e6946cdb\") " pod="kserve-ci-e2e-test/success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg"
Jul 21 17:48:13.976860 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:13.976838    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/584fd906-8440-4595-9371-2482e6946cdb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg\" (UID: \"584fd906-8440-4595-9371-2482e6946cdb\") " pod="kserve-ci-e2e-test/success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg"
Jul 21 17:48:13.977053 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:13.976932    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88r7w\" (UniqueName: \"kubernetes.io/projected/584fd906-8440-4595-9371-2482e6946cdb-kube-api-access-88r7w\") pod \"success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg\" (UID: \"584fd906-8440-4595-9371-2482e6946cdb\") " pod="kserve-ci-e2e-test/success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg"
Jul 21 17:48:14.018471 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:14.018444    2563 generic.go:356] "Generic (PLEG): container finished" podID="80b3aae2-8538-4e58-8249-c708ca4fde57" containerID="c61f19bd973ee0986a8fc94772f309bda529c7a6b4eb2960af40737924463c6d" exitCode=2
Jul 21 17:48:14.018613 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:14.018514    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f64bd-predictor-7764f488c-c4ttx" event={"ID":"80b3aae2-8538-4e58-8249-c708ca4fde57","Type":"ContainerDied","Data":"c61f19bd973ee0986a8fc94772f309bda529c7a6b4eb2960af40737924463c6d"}
Jul 21 17:48:14.020322 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:14.020298    2563 generic.go:356] "Generic (PLEG): container finished" podID="26f7402a-85bc-48c6-b8aa-2d192f06bd3c" containerID="3a6ebe019aca80f430745d2da2e88b0f05468bf34c060f86780436e792bf5e62" exitCode=2
Jul 21 17:48:14.020414 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:14.020331    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7" event={"ID":"26f7402a-85bc-48c6-b8aa-2d192f06bd3c","Type":"ContainerDied","Data":"3a6ebe019aca80f430745d2da2e88b0f05468bf34c060f86780436e792bf5e62"}
Jul 21 17:48:14.078104 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:14.078070    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1633ad8f-0036-4ccf-893f-61ffdbb15b56-proxy-tls\") pod \"error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr\" (UID: \"1633ad8f-0036-4ccf-893f-61ffdbb15b56\") " pod="kserve-ci-e2e-test/error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr"
Jul 21 17:48:14.078273 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:14.078122    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88r7w\" (UniqueName: \"kubernetes.io/projected/584fd906-8440-4595-9371-2482e6946cdb-kube-api-access-88r7w\") pod \"success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg\" (UID: \"584fd906-8440-4595-9371-2482e6946cdb\") " pod="kserve-ci-e2e-test/success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg"
Jul 21 17:48:14.078273 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:14.078175    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1633ad8f-0036-4ccf-893f-61ffdbb15b56-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr\" (UID: \"1633ad8f-0036-4ccf-893f-61ffdbb15b56\") " pod="kserve-ci-e2e-test/error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr"
Jul 21 17:48:14.078590 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:14.078314    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/584fd906-8440-4595-9371-2482e6946cdb-proxy-tls\") pod \"success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg\" (UID: \"584fd906-8440-4595-9371-2482e6946cdb\") " pod="kserve-ci-e2e-test/success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg"
Jul 21 17:48:14.078590 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:14.078343    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gklh2\" (UniqueName: \"kubernetes.io/projected/1633ad8f-0036-4ccf-893f-61ffdbb15b56-kube-api-access-gklh2\") pod \"error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr\" (UID: \"1633ad8f-0036-4ccf-893f-61ffdbb15b56\") " pod="kserve-ci-e2e-test/error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr"
Jul 21 17:48:14.078590 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:14.078394    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/584fd906-8440-4595-9371-2482e6946cdb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg\" (UID: \"584fd906-8440-4595-9371-2482e6946cdb\") " pod="kserve-ci-e2e-test/success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg"
Jul 21 17:48:14.079013 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:14.078990    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/584fd906-8440-4595-9371-2482e6946cdb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg\" (UID: \"584fd906-8440-4595-9371-2482e6946cdb\") " pod="kserve-ci-e2e-test/success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg"
Jul 21 17:48:14.080801 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:14.080780    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/584fd906-8440-4595-9371-2482e6946cdb-proxy-tls\") pod \"success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg\" (UID: \"584fd906-8440-4595-9371-2482e6946cdb\") " pod="kserve-ci-e2e-test/success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg"
Jul 21 17:48:14.086108 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:14.086085    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88r7w\" (UniqueName: \"kubernetes.io/projected/584fd906-8440-4595-9371-2482e6946cdb-kube-api-access-88r7w\") pod \"success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg\" (UID: \"584fd906-8440-4595-9371-2482e6946cdb\") " pod="kserve-ci-e2e-test/success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg"
Jul 21 17:48:14.144290 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:14.144260    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg"
Jul 21 17:48:14.179509 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:14.179470    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1633ad8f-0036-4ccf-893f-61ffdbb15b56-proxy-tls\") pod \"error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr\" (UID: \"1633ad8f-0036-4ccf-893f-61ffdbb15b56\") " pod="kserve-ci-e2e-test/error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr"
Jul 21 17:48:14.179685 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:14.179545    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1633ad8f-0036-4ccf-893f-61ffdbb15b56-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr\" (UID: \"1633ad8f-0036-4ccf-893f-61ffdbb15b56\") " pod="kserve-ci-e2e-test/error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr"
Jul 21 17:48:14.179685 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:14.179589    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gklh2\" (UniqueName: \"kubernetes.io/projected/1633ad8f-0036-4ccf-893f-61ffdbb15b56-kube-api-access-gklh2\") pod \"error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr\" (UID: \"1633ad8f-0036-4ccf-893f-61ffdbb15b56\") " pod="kserve-ci-e2e-test/error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr"
Jul 21 17:48:14.180357 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:14.180237    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1633ad8f-0036-4ccf-893f-61ffdbb15b56-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr\" (UID: \"1633ad8f-0036-4ccf-893f-61ffdbb15b56\") " pod="kserve-ci-e2e-test/error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr"
Jul 21 17:48:14.182096 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:14.182070    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1633ad8f-0036-4ccf-893f-61ffdbb15b56-proxy-tls\") pod \"error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr\" (UID: \"1633ad8f-0036-4ccf-893f-61ffdbb15b56\") " pod="kserve-ci-e2e-test/error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr"
Jul 21 17:48:14.188187 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:14.188164    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gklh2\" (UniqueName: \"kubernetes.io/projected/1633ad8f-0036-4ccf-893f-61ffdbb15b56-kube-api-access-gklh2\") pod \"error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr\" (UID: \"1633ad8f-0036-4ccf-893f-61ffdbb15b56\") " pod="kserve-ci-e2e-test/error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr"
Jul 21 17:48:14.242420 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:14.242380    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr"
Jul 21 17:48:14.289321 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:14.289293    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg"]
Jul 21 17:48:14.291658 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:48:14.291633    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod584fd906_8440_4595_9371_2482e6946cdb.slice/crio-dc273eed8a07e47d8a916025d73ae56077fc9ae46ac9a0967af77b09888e0884 WatchSource:0}: Error finding container dc273eed8a07e47d8a916025d73ae56077fc9ae46ac9a0967af77b09888e0884: Status 404 returned error can't find the container with id dc273eed8a07e47d8a916025d73ae56077fc9ae46ac9a0967af77b09888e0884
Jul 21 17:48:14.390172 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:14.390150    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr"]
Jul 21 17:48:14.392256 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:48:14.392228    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1633ad8f_0036_4ccf_893f_61ffdbb15b56.slice/crio-f7c003e35c5b249a5394242d1a57622bbcf2aa79ebc99fb8074288243cd79366 WatchSource:0}: Error finding container f7c003e35c5b249a5394242d1a57622bbcf2aa79ebc99fb8074288243cd79366: Status 404 returned error can't find the container with id f7c003e35c5b249a5394242d1a57622bbcf2aa79ebc99fb8074288243cd79366
Jul 21 17:48:15.026003 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:15.025963    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg" event={"ID":"584fd906-8440-4595-9371-2482e6946cdb","Type":"ContainerStarted","Data":"60d4ef1e42cdc343024bf803ccbf215943a5b4ff4009de89a264909d16ea1223"}
Jul 21 17:48:15.026492 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:15.026034    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg" event={"ID":"584fd906-8440-4595-9371-2482e6946cdb","Type":"ContainerStarted","Data":"33487f3732c3fd0cd559abee39dcfdfc4b47b828059617cf2d4219f14653c87b"}
Jul 21 17:48:15.026492 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:15.026065    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg" event={"ID":"584fd906-8440-4595-9371-2482e6946cdb","Type":"ContainerStarted","Data":"dc273eed8a07e47d8a916025d73ae56077fc9ae46ac9a0967af77b09888e0884"}
Jul 21 17:48:15.026492 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:15.026227    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg"
Jul 21 17:48:15.026492 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:15.026283    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg"
Jul 21 17:48:15.027586 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:15.027551    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg" podUID="584fd906-8440-4595-9371-2482e6946cdb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 21 17:48:15.027919 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:15.027901    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr" event={"ID":"1633ad8f-0036-4ccf-893f-61ffdbb15b56","Type":"ContainerStarted","Data":"ae3e8dd7ff1eee25d0f9cf106cb45fe5c80ebd31a9de74f33027038360562a95"}
Jul 21 17:48:15.028006 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:15.027939    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr" event={"ID":"1633ad8f-0036-4ccf-893f-61ffdbb15b56","Type":"ContainerStarted","Data":"c4c635e0ae90f9d0075ae239a908cf2c4226c47e543b6fdf3cc055a5fd8ad5aa"}
Jul 21 17:48:15.028006 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:15.027957    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr" event={"ID":"1633ad8f-0036-4ccf-893f-61ffdbb15b56","Type":"ContainerStarted","Data":"f7c003e35c5b249a5394242d1a57622bbcf2aa79ebc99fb8074288243cd79366"}
Jul 21 17:48:15.028148 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:15.028113    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr"
Jul 21 17:48:15.028207 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:15.028175    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr"
Jul 21 17:48:15.029023 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:15.029001    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr" podUID="1633ad8f-0036-4ccf-893f-61ffdbb15b56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 21 17:48:15.044563 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:15.044524    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg" podStartSLOduration=2.044510979 podStartE2EDuration="2.044510979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:48:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:48:15.043292702 +0000 UTC m=+1322.805432322" watchObservedRunningTime="2026-07-21 17:48:15.044510979 +0000 UTC m=+1322.806650583"
Jul 21 17:48:15.061268 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:15.061220    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr" podStartSLOduration=2.061205802 podStartE2EDuration="2.061205802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:48:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:48:15.060315373 +0000 UTC m=+1322.822454996" watchObservedRunningTime="2026-07-21 17:48:15.061205802 +0000 UTC m=+1322.823345406"
Jul 21 17:48:16.031774 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:16.031725    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr" podUID="1633ad8f-0036-4ccf-893f-61ffdbb15b56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 21 17:48:16.031774 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:16.031751    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg" podUID="584fd906-8440-4595-9371-2482e6946cdb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 21 17:48:16.735764 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:16.735724    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7" podUID="26f7402a-85bc-48c6-b8aa-2d192f06bd3c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 21 17:48:16.735949 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:16.735724    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f64bd-predictor-7764f488c-c4ttx" podUID="80b3aae2-8538-4e58-8249-c708ca4fde57" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Jul 21 17:48:17.215933 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:17.215912    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f64bd-predictor-7764f488c-c4ttx"
Jul 21 17:48:17.307023 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:17.306929    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ptkz\" (UniqueName: \"kubernetes.io/projected/80b3aae2-8538-4e58-8249-c708ca4fde57-kube-api-access-2ptkz\") pod \"80b3aae2-8538-4e58-8249-c708ca4fde57\" (UID: \"80b3aae2-8538-4e58-8249-c708ca4fde57\") "
Jul 21 17:48:17.307023 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:17.306980    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80b3aae2-8538-4e58-8249-c708ca4fde57-kube-rbac-proxy-sar-config\") pod \"80b3aae2-8538-4e58-8249-c708ca4fde57\" (UID: \"80b3aae2-8538-4e58-8249-c708ca4fde57\") "
Jul 21 17:48:17.307023 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:17.307010    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80b3aae2-8538-4e58-8249-c708ca4fde57-proxy-tls\") pod \"80b3aae2-8538-4e58-8249-c708ca4fde57\" (UID: \"80b3aae2-8538-4e58-8249-c708ca4fde57\") "
Jul 21 17:48:17.307377 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:17.307353    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80b3aae2-8538-4e58-8249-c708ca4fde57-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "80b3aae2-8538-4e58-8249-c708ca4fde57" (UID: "80b3aae2-8538-4e58-8249-c708ca4fde57"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:48:17.309226 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:17.309197    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80b3aae2-8538-4e58-8249-c708ca4fde57-kube-api-access-2ptkz" (OuterVolumeSpecName: "kube-api-access-2ptkz") pod "80b3aae2-8538-4e58-8249-c708ca4fde57" (UID: "80b3aae2-8538-4e58-8249-c708ca4fde57"). InnerVolumeSpecName "kube-api-access-2ptkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:48:17.309347 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:17.309280    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80b3aae2-8538-4e58-8249-c708ca4fde57-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "80b3aae2-8538-4e58-8249-c708ca4fde57" (UID: "80b3aae2-8538-4e58-8249-c708ca4fde57"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:48:17.408119 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:17.408088    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ptkz\" (UniqueName: \"kubernetes.io/projected/80b3aae2-8538-4e58-8249-c708ca4fde57-kube-api-access-2ptkz\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:48:17.408119 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:17.408114    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80b3aae2-8538-4e58-8249-c708ca4fde57-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:48:17.408119 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:17.408124    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80b3aae2-8538-4e58-8249-c708ca4fde57-proxy-tls\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:48:17.943650 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:17.943628    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7"
Jul 21 17:48:18.013373 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:18.013296    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pq2c4\" (UniqueName: \"kubernetes.io/projected/26f7402a-85bc-48c6-b8aa-2d192f06bd3c-kube-api-access-pq2c4\") pod \"26f7402a-85bc-48c6-b8aa-2d192f06bd3c\" (UID: \"26f7402a-85bc-48c6-b8aa-2d192f06bd3c\") "
Jul 21 17:48:18.013373 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:18.013336    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26f7402a-85bc-48c6-b8aa-2d192f06bd3c-proxy-tls\") pod \"26f7402a-85bc-48c6-b8aa-2d192f06bd3c\" (UID: \"26f7402a-85bc-48c6-b8aa-2d192f06bd3c\") "
Jul 21 17:48:18.013544 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:18.013409    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26f7402a-85bc-48c6-b8aa-2d192f06bd3c-kube-rbac-proxy-sar-config\") pod \"26f7402a-85bc-48c6-b8aa-2d192f06bd3c\" (UID: \"26f7402a-85bc-48c6-b8aa-2d192f06bd3c\") "
Jul 21 17:48:18.013803 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:18.013771    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26f7402a-85bc-48c6-b8aa-2d192f06bd3c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "26f7402a-85bc-48c6-b8aa-2d192f06bd3c" (UID: "26f7402a-85bc-48c6-b8aa-2d192f06bd3c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:48:18.015441 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:18.015416    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26f7402a-85bc-48c6-b8aa-2d192f06bd3c-kube-api-access-pq2c4" (OuterVolumeSpecName: "kube-api-access-pq2c4") pod "26f7402a-85bc-48c6-b8aa-2d192f06bd3c" (UID: "26f7402a-85bc-48c6-b8aa-2d192f06bd3c"). InnerVolumeSpecName "kube-api-access-pq2c4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:48:18.015544 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:18.015482    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26f7402a-85bc-48c6-b8aa-2d192f06bd3c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "26f7402a-85bc-48c6-b8aa-2d192f06bd3c" (UID: "26f7402a-85bc-48c6-b8aa-2d192f06bd3c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:48:18.040268 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:18.040232    2563 generic.go:356] "Generic (PLEG): container finished" podID="80b3aae2-8538-4e58-8249-c708ca4fde57" containerID="744c692918959eab78fa52227bbe74fe029a960aff6d49a324fccfd304bda384" exitCode=0
Jul 21 17:48:18.040379 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:18.040281    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f64bd-predictor-7764f488c-c4ttx"
Jul 21 17:48:18.040379 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:18.040287    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f64bd-predictor-7764f488c-c4ttx" event={"ID":"80b3aae2-8538-4e58-8249-c708ca4fde57","Type":"ContainerDied","Data":"744c692918959eab78fa52227bbe74fe029a960aff6d49a324fccfd304bda384"}
Jul 21 17:48:18.040379 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:18.040340    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f64bd-predictor-7764f488c-c4ttx" event={"ID":"80b3aae2-8538-4e58-8249-c708ca4fde57","Type":"ContainerDied","Data":"5377e91eaaf5a227bf6a46bc42d5c5ed0afeb0c95ac95a45c0c1ab3232abc5a7"}
Jul 21 17:48:18.040379 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:18.040372    2563 scope.go:117] "RemoveContainer" containerID="c61f19bd973ee0986a8fc94772f309bda529c7a6b4eb2960af40737924463c6d"
Jul 21 17:48:18.042224 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:18.042114    2563 generic.go:356] "Generic (PLEG): container finished" podID="26f7402a-85bc-48c6-b8aa-2d192f06bd3c" containerID="6d523e03444ae1a1a609b35bd052fe16cafbd38434655e868421bb9a16e7b8b1" exitCode=0
Jul 21 17:48:18.042297 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:18.042163    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7" event={"ID":"26f7402a-85bc-48c6-b8aa-2d192f06bd3c","Type":"ContainerDied","Data":"6d523e03444ae1a1a609b35bd052fe16cafbd38434655e868421bb9a16e7b8b1"}
Jul 21 17:48:18.042297 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:18.042283    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7" event={"ID":"26f7402a-85bc-48c6-b8aa-2d192f06bd3c","Type":"ContainerDied","Data":"65ab87c0de8bf91dce765e034e800f6c0efc87d6991bf9b2edaf9c082515e38f"}
Jul 21 17:48:18.042383 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:18.042183    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7"
Jul 21 17:48:18.056288 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:18.056262    2563 scope.go:117] "RemoveContainer" containerID="744c692918959eab78fa52227bbe74fe029a960aff6d49a324fccfd304bda384"
Jul 21 17:48:18.070832 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:18.070811    2563 scope.go:117] "RemoveContainer" containerID="c61f19bd973ee0986a8fc94772f309bda529c7a6b4eb2960af40737924463c6d"
Jul 21 17:48:18.071113 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:48:18.071093    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c61f19bd973ee0986a8fc94772f309bda529c7a6b4eb2960af40737924463c6d\": container with ID starting with c61f19bd973ee0986a8fc94772f309bda529c7a6b4eb2960af40737924463c6d not found: ID does not exist" containerID="c61f19bd973ee0986a8fc94772f309bda529c7a6b4eb2960af40737924463c6d"
Jul 21 17:48:18.071213 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:18.071120    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c61f19bd973ee0986a8fc94772f309bda529c7a6b4eb2960af40737924463c6d"} err="failed to get container status \"c61f19bd973ee0986a8fc94772f309bda529c7a6b4eb2960af40737924463c6d\": rpc error: code = NotFound desc = could not find container \"c61f19bd973ee0986a8fc94772f309bda529c7a6b4eb2960af40737924463c6d\": container with ID starting with c61f19bd973ee0986a8fc94772f309bda529c7a6b4eb2960af40737924463c6d not found: ID does not exist"
Jul 21 17:48:18.071213 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:18.071157    2563 scope.go:117] "RemoveContainer" containerID="744c692918959eab78fa52227bbe74fe029a960aff6d49a324fccfd304bda384"
Jul 21 17:48:18.071392 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:48:18.071378    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"744c692918959eab78fa52227bbe74fe029a960aff6d49a324fccfd304bda384\": container with ID starting with 744c692918959eab78fa52227bbe74fe029a960aff6d49a324fccfd304bda384 not found: ID does not exist" containerID="744c692918959eab78fa52227bbe74fe029a960aff6d49a324fccfd304bda384"
Jul 21 17:48:18.071436 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:18.071396    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"744c692918959eab78fa52227bbe74fe029a960aff6d49a324fccfd304bda384"} err="failed to get container status \"744c692918959eab78fa52227bbe74fe029a960aff6d49a324fccfd304bda384\": rpc error: code = NotFound desc = could not find container \"744c692918959eab78fa52227bbe74fe029a960aff6d49a324fccfd304bda384\": container with ID starting with 744c692918959eab78fa52227bbe74fe029a960aff6d49a324fccfd304bda384 not found: ID does not exist"
Jul 21 17:48:18.071436 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:18.071410    2563 scope.go:117] "RemoveContainer" containerID="3a6ebe019aca80f430745d2da2e88b0f05468bf34c060f86780436e792bf5e62"
Jul 21 17:48:18.071976 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:18.071956    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f64bd-predictor-7764f488c-c4ttx"]
Jul 21 17:48:18.076183 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:18.076163    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f64bd-predictor-7764f488c-c4ttx"]
Jul 21 17:48:18.079236 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:18.079218    2563 scope.go:117] "RemoveContainer" containerID="6d523e03444ae1a1a609b35bd052fe16cafbd38434655e868421bb9a16e7b8b1"
Jul 21 17:48:18.086296 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:18.086278    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7"]
Jul 21 17:48:18.087013 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:18.086870    2563 scope.go:117] "RemoveContainer" containerID="3a6ebe019aca80f430745d2da2e88b0f05468bf34c060f86780436e792bf5e62"
Jul 21 17:48:18.087173 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:48:18.087153    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a6ebe019aca80f430745d2da2e88b0f05468bf34c060f86780436e792bf5e62\": container with ID starting with 3a6ebe019aca80f430745d2da2e88b0f05468bf34c060f86780436e792bf5e62 not found: ID does not exist" containerID="3a6ebe019aca80f430745d2da2e88b0f05468bf34c060f86780436e792bf5e62"
Jul 21 17:48:18.087255 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:18.087177    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a6ebe019aca80f430745d2da2e88b0f05468bf34c060f86780436e792bf5e62"} err="failed to get container status \"3a6ebe019aca80f430745d2da2e88b0f05468bf34c060f86780436e792bf5e62\": rpc error: code = NotFound desc = could not find container \"3a6ebe019aca80f430745d2da2e88b0f05468bf34c060f86780436e792bf5e62\": container with ID starting with 3a6ebe019aca80f430745d2da2e88b0f05468bf34c060f86780436e792bf5e62 not found: ID does not exist"
Jul 21 17:48:18.087255 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:18.087193    2563 scope.go:117] "RemoveContainer" containerID="6d523e03444ae1a1a609b35bd052fe16cafbd38434655e868421bb9a16e7b8b1"
Jul 21 17:48:18.087462 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:48:18.087446    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d523e03444ae1a1a609b35bd052fe16cafbd38434655e868421bb9a16e7b8b1\": container with ID starting with 6d523e03444ae1a1a609b35bd052fe16cafbd38434655e868421bb9a16e7b8b1 not found: ID does not exist" containerID="6d523e03444ae1a1a609b35bd052fe16cafbd38434655e868421bb9a16e7b8b1"
Jul 21 17:48:18.087511 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:18.087468    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d523e03444ae1a1a609b35bd052fe16cafbd38434655e868421bb9a16e7b8b1"} err="failed to get container status \"6d523e03444ae1a1a609b35bd052fe16cafbd38434655e868421bb9a16e7b8b1\": rpc error: code = NotFound desc = could not find container \"6d523e03444ae1a1a609b35bd052fe16cafbd38434655e868421bb9a16e7b8b1\": container with ID starting with 6d523e03444ae1a1a609b35bd052fe16cafbd38434655e868421bb9a16e7b8b1 not found: ID does not exist"
Jul 21 17:48:18.093334 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:18.093314    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f64bd-predictor-76c89fcd66-2mzm7"]
Jul 21 17:48:18.114381 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:18.114356    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26f7402a-85bc-48c6-b8aa-2d192f06bd3c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:48:18.114381 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:18.114375    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq2c4\" (UniqueName: \"kubernetes.io/projected/26f7402a-85bc-48c6-b8aa-2d192f06bd3c-kube-api-access-pq2c4\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:48:18.114503 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:18.114386    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26f7402a-85bc-48c6-b8aa-2d192f06bd3c-proxy-tls\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:48:18.764177 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:18.764121    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26f7402a-85bc-48c6-b8aa-2d192f06bd3c" path="/var/lib/kubelet/pods/26f7402a-85bc-48c6-b8aa-2d192f06bd3c/volumes"
Jul 21 17:48:18.764556 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:18.764542    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80b3aae2-8538-4e58-8249-c708ca4fde57" path="/var/lib/kubelet/pods/80b3aae2-8538-4e58-8249-c708ca4fde57/volumes"
Jul 21 17:48:21.036013 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:21.035985    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr"
Jul 21 17:48:21.036492 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:21.036358    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg"
Jul 21 17:48:21.036492 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:21.036435    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr" podUID="1633ad8f-0036-4ccf-893f-61ffdbb15b56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 21 17:48:21.036736 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:21.036705    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg" podUID="584fd906-8440-4595-9371-2482e6946cdb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 21 17:48:21.871613 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:21.871586    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b64c9-predictor-765f6b789f-47rtc"
Jul 21 17:48:21.871760 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:21.871670    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-b64c9-predictor-f779895b6-4rwsl"
Jul 21 17:48:31.036423 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:31.036381    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr" podUID="1633ad8f-0036-4ccf-893f-61ffdbb15b56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 21 17:48:31.036808 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:31.036792    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg" podUID="584fd906-8440-4595-9371-2482e6946cdb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 21 17:48:41.037356 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:41.037277    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr" podUID="1633ad8f-0036-4ccf-893f-61ffdbb15b56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 21 17:48:41.037709 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:41.037276    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg" podUID="584fd906-8440-4595-9371-2482e6946cdb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 21 17:48:43.775358 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:43.775315    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b64c9-predictor-765f6b789f-47rtc"]
Jul 21 17:48:43.775971 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:43.775651    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b64c9-predictor-765f6b789f-47rtc" podUID="07fc8a39-1f80-4327-bf72-32bb289206e5" containerName="kserve-container" containerID="cri-o://e9da7d49714f27df5ba6e9daba22919a7574caf8c836eb5248152495861af925" gracePeriod=30
Jul 21 17:48:43.775971 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:43.775704    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b64c9-predictor-765f6b789f-47rtc" podUID="07fc8a39-1f80-4327-bf72-32bb289206e5" containerName="kube-rbac-proxy" containerID="cri-o://fbcbd37ad94553c2d170eecec3545ae69ab8fedbe6233c58acbcd6efebf2a44e" gracePeriod=30
Jul 21 17:48:43.830845 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:43.830812    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b64c9-predictor-f779895b6-4rwsl"]
Jul 21 17:48:43.831096 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:43.831041    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-b64c9-predictor-f779895b6-4rwsl" podUID="26970d94-f960-4da3-a5fe-adfa5d370484" containerName="kserve-container" containerID="cri-o://487a55beb06d1ff3a5f9bdddd2a310a22f82f98cdd09a7c56662e4cde9f5755d" gracePeriod=30
Jul 21 17:48:43.831245 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:43.831085    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-b64c9-predictor-f779895b6-4rwsl" podUID="26970d94-f960-4da3-a5fe-adfa5d370484" containerName="kube-rbac-proxy" containerID="cri-o://cd19330596b1be5239337196757330fd5e6df488d278aca33d2d4e02876a5c74" gracePeriod=30
Jul 21 17:48:43.851046 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:43.851018    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8e991-predictor-799cbc5d5f-vzztz"]
Jul 21 17:48:43.851415 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:43.851398    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80b3aae2-8538-4e58-8249-c708ca4fde57" containerName="kserve-container"
Jul 21 17:48:43.851415 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:43.851414    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="80b3aae2-8538-4e58-8249-c708ca4fde57" containerName="kserve-container"
Jul 21 17:48:43.851554 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:43.851427    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26f7402a-85bc-48c6-b8aa-2d192f06bd3c" containerName="kube-rbac-proxy"
Jul 21 17:48:43.851554 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:43.851432    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="26f7402a-85bc-48c6-b8aa-2d192f06bd3c" containerName="kube-rbac-proxy"
Jul 21 17:48:43.851554 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:43.851448    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26f7402a-85bc-48c6-b8aa-2d192f06bd3c" containerName="kserve-container"
Jul 21 17:48:43.851554 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:43.851456    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="26f7402a-85bc-48c6-b8aa-2d192f06bd3c" containerName="kserve-container"
Jul 21 17:48:43.851554 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:43.851471    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80b3aae2-8538-4e58-8249-c708ca4fde57" containerName="kube-rbac-proxy"
Jul 21 17:48:43.851554 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:43.851477    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="80b3aae2-8538-4e58-8249-c708ca4fde57" containerName="kube-rbac-proxy"
Jul 21 17:48:43.851554 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:43.851536    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="80b3aae2-8538-4e58-8249-c708ca4fde57" containerName="kube-rbac-proxy"
Jul 21 17:48:43.851554 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:43.851548    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="80b3aae2-8538-4e58-8249-c708ca4fde57" containerName="kserve-container"
Jul 21 17:48:43.851554 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:43.851554    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="26f7402a-85bc-48c6-b8aa-2d192f06bd3c" containerName="kserve-container"
Jul 21 17:48:43.851824 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:43.851561    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="26f7402a-85bc-48c6-b8aa-2d192f06bd3c" containerName="kube-rbac-proxy"
Jul 21 17:48:43.854833 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:43.854813    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8e991-predictor-799cbc5d5f-vzztz"
Jul 21 17:48:43.857191 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:43.857163    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8e991-kube-rbac-proxy-sar-config\""
Jul 21 17:48:43.857289 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:43.857230    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8e991-predictor-serving-cert\""
Jul 21 17:48:43.866071 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:43.866050    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8e991-predictor-799cbc5d5f-vzztz"]
Jul 21 17:48:43.926920 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:43.926886    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mffqv\" (UniqueName: \"kubernetes.io/projected/9622b1bb-2668-4e94-a02f-e1c605d3a187-kube-api-access-mffqv\") pod \"success-200-isvc-8e991-predictor-799cbc5d5f-vzztz\" (UID: \"9622b1bb-2668-4e94-a02f-e1c605d3a187\") " pod="kserve-ci-e2e-test/success-200-isvc-8e991-predictor-799cbc5d5f-vzztz"
Jul 21 17:48:43.927080 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:43.926934    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9622b1bb-2668-4e94-a02f-e1c605d3a187-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8e991-predictor-799cbc5d5f-vzztz\" (UID: \"9622b1bb-2668-4e94-a02f-e1c605d3a187\") " pod="kserve-ci-e2e-test/success-200-isvc-8e991-predictor-799cbc5d5f-vzztz"
Jul 21 17:48:43.927080 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:43.926971    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9622b1bb-2668-4e94-a02f-e1c605d3a187-proxy-tls\") pod \"success-200-isvc-8e991-predictor-799cbc5d5f-vzztz\" (UID: \"9622b1bb-2668-4e94-a02f-e1c605d3a187\") " pod="kserve-ci-e2e-test/success-200-isvc-8e991-predictor-799cbc5d5f-vzztz"
Jul 21 17:48:43.940930 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:43.940904    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8e991-predictor-5749cbc5db-9qmzg"]
Jul 21 17:48:43.944326 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:43.944306    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8e991-predictor-5749cbc5db-9qmzg"
Jul 21 17:48:43.946536 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:43.946517    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8e991-predictor-serving-cert\""
Jul 21 17:48:43.946643 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:43.946526    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8e991-kube-rbac-proxy-sar-config\""
Jul 21 17:48:43.955031 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:43.955004    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8e991-predictor-5749cbc5db-9qmzg"]
Jul 21 17:48:44.028399 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:44.028316    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9622b1bb-2668-4e94-a02f-e1c605d3a187-proxy-tls\") pod \"success-200-isvc-8e991-predictor-799cbc5d5f-vzztz\" (UID: \"9622b1bb-2668-4e94-a02f-e1c605d3a187\") " pod="kserve-ci-e2e-test/success-200-isvc-8e991-predictor-799cbc5d5f-vzztz"
Jul 21 17:48:44.028553 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:44.028409    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7a5c652-026f-4a13-9316-fde065088fb1-proxy-tls\") pod \"error-404-isvc-8e991-predictor-5749cbc5db-9qmzg\" (UID: \"d7a5c652-026f-4a13-9316-fde065088fb1\") " pod="kserve-ci-e2e-test/error-404-isvc-8e991-predictor-5749cbc5db-9qmzg"
Jul 21 17:48:44.028553 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:44.028444    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mffqv\" (UniqueName: \"kubernetes.io/projected/9622b1bb-2668-4e94-a02f-e1c605d3a187-kube-api-access-mffqv\") pod \"success-200-isvc-8e991-predictor-799cbc5d5f-vzztz\" (UID: \"9622b1bb-2668-4e94-a02f-e1c605d3a187\") " pod="kserve-ci-e2e-test/success-200-isvc-8e991-predictor-799cbc5d5f-vzztz"
Jul 21 17:48:44.028553 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:44.028470    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bttxq\" (UniqueName: \"kubernetes.io/projected/d7a5c652-026f-4a13-9316-fde065088fb1-kube-api-access-bttxq\") pod \"error-404-isvc-8e991-predictor-5749cbc5db-9qmzg\" (UID: \"d7a5c652-026f-4a13-9316-fde065088fb1\") " pod="kserve-ci-e2e-test/error-404-isvc-8e991-predictor-5749cbc5db-9qmzg"
Jul 21 17:48:44.028553 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:44.028517    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7a5c652-026f-4a13-9316-fde065088fb1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8e991-predictor-5749cbc5db-9qmzg\" (UID: \"d7a5c652-026f-4a13-9316-fde065088fb1\") " pod="kserve-ci-e2e-test/error-404-isvc-8e991-predictor-5749cbc5db-9qmzg"
Jul 21 17:48:44.028745 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:44.028594    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9622b1bb-2668-4e94-a02f-e1c605d3a187-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8e991-predictor-799cbc5d5f-vzztz\" (UID: \"9622b1bb-2668-4e94-a02f-e1c605d3a187\") " pod="kserve-ci-e2e-test/success-200-isvc-8e991-predictor-799cbc5d5f-vzztz"
Jul 21 17:48:44.029227 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:44.029209    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9622b1bb-2668-4e94-a02f-e1c605d3a187-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8e991-predictor-799cbc5d5f-vzztz\" (UID: \"9622b1bb-2668-4e94-a02f-e1c605d3a187\") " pod="kserve-ci-e2e-test/success-200-isvc-8e991-predictor-799cbc5d5f-vzztz"
Jul 21 17:48:44.030871 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:44.030853    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9622b1bb-2668-4e94-a02f-e1c605d3a187-proxy-tls\") pod \"success-200-isvc-8e991-predictor-799cbc5d5f-vzztz\" (UID: \"9622b1bb-2668-4e94-a02f-e1c605d3a187\") " pod="kserve-ci-e2e-test/success-200-isvc-8e991-predictor-799cbc5d5f-vzztz"
Jul 21 17:48:44.036868 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:44.036844    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mffqv\" (UniqueName: \"kubernetes.io/projected/9622b1bb-2668-4e94-a02f-e1c605d3a187-kube-api-access-mffqv\") pod \"success-200-isvc-8e991-predictor-799cbc5d5f-vzztz\" (UID: \"9622b1bb-2668-4e94-a02f-e1c605d3a187\") " pod="kserve-ci-e2e-test/success-200-isvc-8e991-predictor-799cbc5d5f-vzztz"
Jul 21 17:48:44.129024 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:44.128985    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7a5c652-026f-4a13-9316-fde065088fb1-proxy-tls\") pod \"error-404-isvc-8e991-predictor-5749cbc5db-9qmzg\" (UID: \"d7a5c652-026f-4a13-9316-fde065088fb1\") " pod="kserve-ci-e2e-test/error-404-isvc-8e991-predictor-5749cbc5db-9qmzg"
Jul 21 17:48:44.129264 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:44.129032    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bttxq\" (UniqueName: \"kubernetes.io/projected/d7a5c652-026f-4a13-9316-fde065088fb1-kube-api-access-bttxq\") pod \"error-404-isvc-8e991-predictor-5749cbc5db-9qmzg\" (UID: \"d7a5c652-026f-4a13-9316-fde065088fb1\") " pod="kserve-ci-e2e-test/error-404-isvc-8e991-predictor-5749cbc5db-9qmzg"
Jul 21 17:48:44.129264 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:44.129055    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7a5c652-026f-4a13-9316-fde065088fb1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8e991-predictor-5749cbc5db-9qmzg\" (UID: \"d7a5c652-026f-4a13-9316-fde065088fb1\") " pod="kserve-ci-e2e-test/error-404-isvc-8e991-predictor-5749cbc5db-9qmzg"
Jul 21 17:48:44.129711 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:44.129689    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7a5c652-026f-4a13-9316-fde065088fb1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8e991-predictor-5749cbc5db-9qmzg\" (UID: \"d7a5c652-026f-4a13-9316-fde065088fb1\") " pod="kserve-ci-e2e-test/error-404-isvc-8e991-predictor-5749cbc5db-9qmzg"
Jul 21 17:48:44.131592 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:44.131574    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7a5c652-026f-4a13-9316-fde065088fb1-proxy-tls\") pod \"error-404-isvc-8e991-predictor-5749cbc5db-9qmzg\" (UID: \"d7a5c652-026f-4a13-9316-fde065088fb1\") " pod="kserve-ci-e2e-test/error-404-isvc-8e991-predictor-5749cbc5db-9qmzg"
Jul 21 17:48:44.137978 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:44.137950    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bttxq\" (UniqueName: \"kubernetes.io/projected/d7a5c652-026f-4a13-9316-fde065088fb1-kube-api-access-bttxq\") pod \"error-404-isvc-8e991-predictor-5749cbc5db-9qmzg\" (UID: \"d7a5c652-026f-4a13-9316-fde065088fb1\") " pod="kserve-ci-e2e-test/error-404-isvc-8e991-predictor-5749cbc5db-9qmzg"
Jul 21 17:48:44.150932 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:44.150890    2563 generic.go:356] "Generic (PLEG): container finished" podID="07fc8a39-1f80-4327-bf72-32bb289206e5" containerID="fbcbd37ad94553c2d170eecec3545ae69ab8fedbe6233c58acbcd6efebf2a44e" exitCode=2
Jul 21 17:48:44.151182 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:44.151125    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b64c9-predictor-765f6b789f-47rtc" event={"ID":"07fc8a39-1f80-4327-bf72-32bb289206e5","Type":"ContainerDied","Data":"fbcbd37ad94553c2d170eecec3545ae69ab8fedbe6233c58acbcd6efebf2a44e"}
Jul 21 17:48:44.153394 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:44.153366    2563 generic.go:356] "Generic (PLEG): container finished" podID="26970d94-f960-4da3-a5fe-adfa5d370484" containerID="cd19330596b1be5239337196757330fd5e6df488d278aca33d2d4e02876a5c74" exitCode=2
Jul 21 17:48:44.153527 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:44.153447    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b64c9-predictor-f779895b6-4rwsl" event={"ID":"26970d94-f960-4da3-a5fe-adfa5d370484","Type":"ContainerDied","Data":"cd19330596b1be5239337196757330fd5e6df488d278aca33d2d4e02876a5c74"}
Jul 21 17:48:44.164601 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:44.164577    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8e991-predictor-799cbc5d5f-vzztz"
Jul 21 17:48:44.256036 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:44.255782    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8e991-predictor-5749cbc5db-9qmzg"
Jul 21 17:48:44.306380 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:44.306348    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8e991-predictor-799cbc5d5f-vzztz"]
Jul 21 17:48:44.309063 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:48:44.309033    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9622b1bb_2668_4e94_a02f_e1c605d3a187.slice/crio-18388bed19944a52da90009308bef174047b30f2114cef2ad9b1ff819c19ed82 WatchSource:0}: Error finding container 18388bed19944a52da90009308bef174047b30f2114cef2ad9b1ff819c19ed82: Status 404 returned error can't find the container with id 18388bed19944a52da90009308bef174047b30f2114cef2ad9b1ff819c19ed82
Jul 21 17:48:44.399219 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:44.399198    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8e991-predictor-5749cbc5db-9qmzg"]
Jul 21 17:48:44.401713 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:48:44.401691    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7a5c652_026f_4a13_9316_fde065088fb1.slice/crio-00c2ce538b2a25e176c4f6afed49308b456c84443af0db95f80e36887306d049 WatchSource:0}: Error finding container 00c2ce538b2a25e176c4f6afed49308b456c84443af0db95f80e36887306d049: Status 404 returned error can't find the container with id 00c2ce538b2a25e176c4f6afed49308b456c84443af0db95f80e36887306d049
Jul 21 17:48:45.160671 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:45.160637    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8e991-predictor-799cbc5d5f-vzztz" event={"ID":"9622b1bb-2668-4e94-a02f-e1c605d3a187","Type":"ContainerStarted","Data":"570ae0d7b305cccc4a671d55016c2e47861f3fd2beb36a1c8533a3e4488ea419"}
Jul 21 17:48:45.161153 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:45.160714    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8e991-predictor-799cbc5d5f-vzztz"
Jul 21 17:48:45.161153 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:45.160839    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8e991-predictor-799cbc5d5f-vzztz"
Jul 21 17:48:45.161153 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:45.160880    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8e991-predictor-799cbc5d5f-vzztz" event={"ID":"9622b1bb-2668-4e94-a02f-e1c605d3a187","Type":"ContainerStarted","Data":"5ae44d193f106e37bd84bdc83aa606d5bd37cf6176e2e4207837cb2e8dc4c785"}
Jul 21 17:48:45.161153 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:45.160908    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8e991-predictor-799cbc5d5f-vzztz" event={"ID":"9622b1bb-2668-4e94-a02f-e1c605d3a187","Type":"ContainerStarted","Data":"18388bed19944a52da90009308bef174047b30f2114cef2ad9b1ff819c19ed82"}
Jul 21 17:48:45.162439 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:45.162404    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8e991-predictor-799cbc5d5f-vzztz" podUID="9622b1bb-2668-4e94-a02f-e1c605d3a187" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 21 17:48:45.162582 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:45.162557    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8e991-predictor-5749cbc5db-9qmzg" event={"ID":"d7a5c652-026f-4a13-9316-fde065088fb1","Type":"ContainerStarted","Data":"a29f5b5e0eaef97591f29cc4da7163683d3dfe71b648db64d5cb4cd46d386445"}
Jul 21 17:48:45.162716 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:45.162610    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8e991-predictor-5749cbc5db-9qmzg" event={"ID":"d7a5c652-026f-4a13-9316-fde065088fb1","Type":"ContainerStarted","Data":"f3ef3f03e0e4116bc607f81f35f16c01436c2e622abc3c3f53ce7d79378f482d"}
Jul 21 17:48:45.162716 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:45.162642    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8e991-predictor-5749cbc5db-9qmzg" event={"ID":"d7a5c652-026f-4a13-9316-fde065088fb1","Type":"ContainerStarted","Data":"00c2ce538b2a25e176c4f6afed49308b456c84443af0db95f80e36887306d049"}
Jul 21 17:48:45.162716 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:45.162694    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8e991-predictor-5749cbc5db-9qmzg"
Jul 21 17:48:45.178966 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:45.178919    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-8e991-predictor-799cbc5d5f-vzztz" podStartSLOduration=2.17890651 podStartE2EDuration="2.17890651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:48:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:48:45.178288331 +0000 UTC m=+1352.940427948" watchObservedRunningTime="2026-07-21 17:48:45.17890651 +0000 UTC m=+1352.941046113"
Jul 21 17:48:45.197985 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:45.197939    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-8e991-predictor-5749cbc5db-9qmzg" podStartSLOduration=2.197923797 podStartE2EDuration="2.197923797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:48:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:48:45.196055396 +0000 UTC m=+1352.958195001" watchObservedRunningTime="2026-07-21 17:48:45.197923797 +0000 UTC m=+1352.960063476"
Jul 21 17:48:46.167073 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:46.167040    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8e991-predictor-5749cbc5db-9qmzg"
Jul 21 17:48:46.167574 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:46.167075    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8e991-predictor-799cbc5d5f-vzztz" podUID="9622b1bb-2668-4e94-a02f-e1c605d3a187" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 21 17:48:46.168226 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:46.168201    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8e991-predictor-5749cbc5db-9qmzg" podUID="d7a5c652-026f-4a13-9316-fde065088fb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 21 17:48:46.866624 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:46.866582    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b64c9-predictor-765f6b789f-47rtc" podUID="07fc8a39-1f80-4327-bf72-32bb289206e5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 21 17:48:46.866810 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:46.866584    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b64c9-predictor-f779895b6-4rwsl" podUID="26970d94-f960-4da3-a5fe-adfa5d370484" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 21 17:48:47.170833 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:47.170792    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8e991-predictor-5749cbc5db-9qmzg" podUID="d7a5c652-026f-4a13-9316-fde065088fb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 21 17:48:47.854652 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:47.854628    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b64c9-predictor-765f6b789f-47rtc"
Jul 21 17:48:47.961551 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:47.961530    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07fc8a39-1f80-4327-bf72-32bb289206e5-kube-rbac-proxy-sar-config\") pod \"07fc8a39-1f80-4327-bf72-32bb289206e5\" (UID: \"07fc8a39-1f80-4327-bf72-32bb289206e5\") "
Jul 21 17:48:47.961699 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:47.961590    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhwgp\" (UniqueName: \"kubernetes.io/projected/07fc8a39-1f80-4327-bf72-32bb289206e5-kube-api-access-rhwgp\") pod \"07fc8a39-1f80-4327-bf72-32bb289206e5\" (UID: \"07fc8a39-1f80-4327-bf72-32bb289206e5\") "
Jul 21 17:48:47.961699 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:47.961634    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07fc8a39-1f80-4327-bf72-32bb289206e5-proxy-tls\") pod \"07fc8a39-1f80-4327-bf72-32bb289206e5\" (UID: \"07fc8a39-1f80-4327-bf72-32bb289206e5\") "
Jul 21 17:48:47.961911 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:47.961882    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07fc8a39-1f80-4327-bf72-32bb289206e5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "07fc8a39-1f80-4327-bf72-32bb289206e5" (UID: "07fc8a39-1f80-4327-bf72-32bb289206e5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:48:47.963969 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:47.963942    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07fc8a39-1f80-4327-bf72-32bb289206e5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "07fc8a39-1f80-4327-bf72-32bb289206e5" (UID: "07fc8a39-1f80-4327-bf72-32bb289206e5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:48:47.963969 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:47.963955    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07fc8a39-1f80-4327-bf72-32bb289206e5-kube-api-access-rhwgp" (OuterVolumeSpecName: "kube-api-access-rhwgp") pod "07fc8a39-1f80-4327-bf72-32bb289206e5" (UID: "07fc8a39-1f80-4327-bf72-32bb289206e5"). InnerVolumeSpecName "kube-api-access-rhwgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:48:47.966160 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:47.966126    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b64c9-predictor-f779895b6-4rwsl"
Jul 21 17:48:48.062085 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:48.062051    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26970d94-f960-4da3-a5fe-adfa5d370484-kube-rbac-proxy-sar-config\") pod \"26970d94-f960-4da3-a5fe-adfa5d370484\" (UID: \"26970d94-f960-4da3-a5fe-adfa5d370484\") "
Jul 21 17:48:48.062259 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:48.062156    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26970d94-f960-4da3-a5fe-adfa5d370484-proxy-tls\") pod \"26970d94-f960-4da3-a5fe-adfa5d370484\" (UID: \"26970d94-f960-4da3-a5fe-adfa5d370484\") "
Jul 21 17:48:48.062259 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:48.062205    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8j4h\" (UniqueName: \"kubernetes.io/projected/26970d94-f960-4da3-a5fe-adfa5d370484-kube-api-access-b8j4h\") pod \"26970d94-f960-4da3-a5fe-adfa5d370484\" (UID: \"26970d94-f960-4da3-a5fe-adfa5d370484\") "
Jul 21 17:48:48.062432 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:48.062415    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhwgp\" (UniqueName: \"kubernetes.io/projected/07fc8a39-1f80-4327-bf72-32bb289206e5-kube-api-access-rhwgp\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:48:48.062478 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:48.062440    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07fc8a39-1f80-4327-bf72-32bb289206e5-proxy-tls\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:48:48.062478 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:48.062455    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07fc8a39-1f80-4327-bf72-32bb289206e5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:48:48.062538 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:48.062468    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26970d94-f960-4da3-a5fe-adfa5d370484-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "26970d94-f960-4da3-a5fe-adfa5d370484" (UID: "26970d94-f960-4da3-a5fe-adfa5d370484"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:48:48.064283 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:48.064262    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26970d94-f960-4da3-a5fe-adfa5d370484-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "26970d94-f960-4da3-a5fe-adfa5d370484" (UID: "26970d94-f960-4da3-a5fe-adfa5d370484"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:48:48.064378 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:48.064358    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26970d94-f960-4da3-a5fe-adfa5d370484-kube-api-access-b8j4h" (OuterVolumeSpecName: "kube-api-access-b8j4h") pod "26970d94-f960-4da3-a5fe-adfa5d370484" (UID: "26970d94-f960-4da3-a5fe-adfa5d370484"). InnerVolumeSpecName "kube-api-access-b8j4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:48:48.163485 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:48.163457    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26970d94-f960-4da3-a5fe-adfa5d370484-proxy-tls\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:48:48.163485 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:48.163485    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8j4h\" (UniqueName: \"kubernetes.io/projected/26970d94-f960-4da3-a5fe-adfa5d370484-kube-api-access-b8j4h\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:48:48.163684 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:48.163504    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26970d94-f960-4da3-a5fe-adfa5d370484-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:48:48.176097 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:48.176063    2563 generic.go:356] "Generic (PLEG): container finished" podID="07fc8a39-1f80-4327-bf72-32bb289206e5" containerID="e9da7d49714f27df5ba6e9daba22919a7574caf8c836eb5248152495861af925" exitCode=0
Jul 21 17:48:48.176508 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:48.176119    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b64c9-predictor-765f6b789f-47rtc"
Jul 21 17:48:48.176508 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:48.176171    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b64c9-predictor-765f6b789f-47rtc" event={"ID":"07fc8a39-1f80-4327-bf72-32bb289206e5","Type":"ContainerDied","Data":"e9da7d49714f27df5ba6e9daba22919a7574caf8c836eb5248152495861af925"}
Jul 21 17:48:48.176508 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:48.176232    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b64c9-predictor-765f6b789f-47rtc" event={"ID":"07fc8a39-1f80-4327-bf72-32bb289206e5","Type":"ContainerDied","Data":"3c92600805771262427cd2fa38f9cbfc6dadd724f6364fe5a18174b0a9c663bc"}
Jul 21 17:48:48.176508 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:48.176271    2563 scope.go:117] "RemoveContainer" containerID="fbcbd37ad94553c2d170eecec3545ae69ab8fedbe6233c58acbcd6efebf2a44e"
Jul 21 17:48:48.177891 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:48.177870    2563 generic.go:356] "Generic (PLEG): container finished" podID="26970d94-f960-4da3-a5fe-adfa5d370484" containerID="487a55beb06d1ff3a5f9bdddd2a310a22f82f98cdd09a7c56662e4cde9f5755d" exitCode=0
Jul 21 17:48:48.178029 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:48.177927    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b64c9-predictor-f779895b6-4rwsl"
Jul 21 17:48:48.178029 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:48.177961    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b64c9-predictor-f779895b6-4rwsl" event={"ID":"26970d94-f960-4da3-a5fe-adfa5d370484","Type":"ContainerDied","Data":"487a55beb06d1ff3a5f9bdddd2a310a22f82f98cdd09a7c56662e4cde9f5755d"}
Jul 21 17:48:48.178029 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:48.178004    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b64c9-predictor-f779895b6-4rwsl" event={"ID":"26970d94-f960-4da3-a5fe-adfa5d370484","Type":"ContainerDied","Data":"371f723d155fdc0777dd5fc68279c17ec74f30a5fbaeaa6b01ee2ead472f34b8"}
Jul 21 17:48:48.187697 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:48.187678    2563 scope.go:117] "RemoveContainer" containerID="e9da7d49714f27df5ba6e9daba22919a7574caf8c836eb5248152495861af925"
Jul 21 17:48:48.195598 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:48.195581    2563 scope.go:117] "RemoveContainer" containerID="fbcbd37ad94553c2d170eecec3545ae69ab8fedbe6233c58acbcd6efebf2a44e"
Jul 21 17:48:48.195828 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:48:48.195811    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbcbd37ad94553c2d170eecec3545ae69ab8fedbe6233c58acbcd6efebf2a44e\": container with ID starting with fbcbd37ad94553c2d170eecec3545ae69ab8fedbe6233c58acbcd6efebf2a44e not found: ID does not exist" containerID="fbcbd37ad94553c2d170eecec3545ae69ab8fedbe6233c58acbcd6efebf2a44e"
Jul 21 17:48:48.195888 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:48.195835    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbcbd37ad94553c2d170eecec3545ae69ab8fedbe6233c58acbcd6efebf2a44e"} err="failed to get container status \"fbcbd37ad94553c2d170eecec3545ae69ab8fedbe6233c58acbcd6efebf2a44e\": rpc error: code = NotFound desc = could not find container \"fbcbd37ad94553c2d170eecec3545ae69ab8fedbe6233c58acbcd6efebf2a44e\": container with ID starting with fbcbd37ad94553c2d170eecec3545ae69ab8fedbe6233c58acbcd6efebf2a44e not found: ID does not exist"
Jul 21 17:48:48.195888 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:48.195852    2563 scope.go:117] "RemoveContainer" containerID="e9da7d49714f27df5ba6e9daba22919a7574caf8c836eb5248152495861af925"
Jul 21 17:48:48.196068 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:48:48.196051    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9da7d49714f27df5ba6e9daba22919a7574caf8c836eb5248152495861af925\": container with ID starting with e9da7d49714f27df5ba6e9daba22919a7574caf8c836eb5248152495861af925 not found: ID does not exist" containerID="e9da7d49714f27df5ba6e9daba22919a7574caf8c836eb5248152495861af925"
Jul 21 17:48:48.196108 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:48.196074    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9da7d49714f27df5ba6e9daba22919a7574caf8c836eb5248152495861af925"} err="failed to get container status \"e9da7d49714f27df5ba6e9daba22919a7574caf8c836eb5248152495861af925\": rpc error: code = NotFound desc = could not find container \"e9da7d49714f27df5ba6e9daba22919a7574caf8c836eb5248152495861af925\": container with ID starting with e9da7d49714f27df5ba6e9daba22919a7574caf8c836eb5248152495861af925 not found: ID does not exist"
Jul 21 17:48:48.196108 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:48.196087    2563 scope.go:117] "RemoveContainer" containerID="cd19330596b1be5239337196757330fd5e6df488d278aca33d2d4e02876a5c74"
Jul 21 17:48:48.199325 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:48.199305    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b64c9-predictor-765f6b789f-47rtc"]
Jul 21 17:48:48.202939 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:48.202917    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b64c9-predictor-765f6b789f-47rtc"]
Jul 21 17:48:48.203516 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:48.203501    2563 scope.go:117] "RemoveContainer" containerID="487a55beb06d1ff3a5f9bdddd2a310a22f82f98cdd09a7c56662e4cde9f5755d"
Jul 21 17:48:48.210571 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:48.210546    2563 scope.go:117] "RemoveContainer" containerID="cd19330596b1be5239337196757330fd5e6df488d278aca33d2d4e02876a5c74"
Jul 21 17:48:48.210810 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:48:48.210786    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd19330596b1be5239337196757330fd5e6df488d278aca33d2d4e02876a5c74\": container with ID starting with cd19330596b1be5239337196757330fd5e6df488d278aca33d2d4e02876a5c74 not found: ID does not exist" containerID="cd19330596b1be5239337196757330fd5e6df488d278aca33d2d4e02876a5c74"
Jul 21 17:48:48.210901 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:48.210816    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd19330596b1be5239337196757330fd5e6df488d278aca33d2d4e02876a5c74"} err="failed to get container status \"cd19330596b1be5239337196757330fd5e6df488d278aca33d2d4e02876a5c74\": rpc error: code = NotFound desc = could not find container \"cd19330596b1be5239337196757330fd5e6df488d278aca33d2d4e02876a5c74\": container with ID starting with cd19330596b1be5239337196757330fd5e6df488d278aca33d2d4e02876a5c74 not found: ID does not exist"
Jul 21 17:48:48.210901 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:48.210834    2563 scope.go:117] "RemoveContainer" containerID="487a55beb06d1ff3a5f9bdddd2a310a22f82f98cdd09a7c56662e4cde9f5755d"
Jul 21 17:48:48.211200 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:48:48.211176    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"487a55beb06d1ff3a5f9bdddd2a310a22f82f98cdd09a7c56662e4cde9f5755d\": container with ID starting with 487a55beb06d1ff3a5f9bdddd2a310a22f82f98cdd09a7c56662e4cde9f5755d not found: ID does not exist" containerID="487a55beb06d1ff3a5f9bdddd2a310a22f82f98cdd09a7c56662e4cde9f5755d"
Jul 21 17:48:48.211310 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:48.211224    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"487a55beb06d1ff3a5f9bdddd2a310a22f82f98cdd09a7c56662e4cde9f5755d"} err="failed to get container status \"487a55beb06d1ff3a5f9bdddd2a310a22f82f98cdd09a7c56662e4cde9f5755d\": rpc error: code = NotFound desc = could not find container \"487a55beb06d1ff3a5f9bdddd2a310a22f82f98cdd09a7c56662e4cde9f5755d\": container with ID starting with 487a55beb06d1ff3a5f9bdddd2a310a22f82f98cdd09a7c56662e4cde9f5755d not found: ID does not exist"
Jul 21 17:48:48.212804 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:48.212785    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b64c9-predictor-f779895b6-4rwsl"]
Jul 21 17:48:48.220009 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:48.219990    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b64c9-predictor-f779895b6-4rwsl"]
Jul 21 17:48:48.764227 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:48.764197    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07fc8a39-1f80-4327-bf72-32bb289206e5" path="/var/lib/kubelet/pods/07fc8a39-1f80-4327-bf72-32bb289206e5/volumes"
Jul 21 17:48:48.764621 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:48.764607    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26970d94-f960-4da3-a5fe-adfa5d370484" path="/var/lib/kubelet/pods/26970d94-f960-4da3-a5fe-adfa5d370484/volumes"
Jul 21 17:48:51.036445 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:51.036406    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr" podUID="1633ad8f-0036-4ccf-893f-61ffdbb15b56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 21 17:48:51.036875 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:51.036709    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg" podUID="584fd906-8440-4595-9371-2482e6946cdb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 21 17:48:51.171754 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:51.171717    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8e991-predictor-799cbc5d5f-vzztz"
Jul 21 17:48:51.172338 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:51.172312    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8e991-predictor-799cbc5d5f-vzztz" podUID="9622b1bb-2668-4e94-a02f-e1c605d3a187" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 21 17:48:52.175199 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:52.175173    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8e991-predictor-5749cbc5db-9qmzg"
Jul 21 17:48:52.175715 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:48:52.175687    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8e991-predictor-5749cbc5db-9qmzg" podUID="d7a5c652-026f-4a13-9316-fde065088fb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 21 17:49:01.036944 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:49:01.036905    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr" podUID="1633ad8f-0036-4ccf-893f-61ffdbb15b56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 21 17:49:01.037353 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:49:01.036905    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg" podUID="584fd906-8440-4595-9371-2482e6946cdb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 21 17:49:01.173338 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:49:01.173291    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8e991-predictor-799cbc5d5f-vzztz" podUID="9622b1bb-2668-4e94-a02f-e1c605d3a187" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 21 17:49:02.175682 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:49:02.175638    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8e991-predictor-5749cbc5db-9qmzg" podUID="d7a5c652-026f-4a13-9316-fde065088fb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 21 17:49:11.036966 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:49:11.036935    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr"
Jul 21 17:49:11.037691 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:49:11.037674    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg"
Jul 21 17:49:11.172692 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:49:11.172656    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8e991-predictor-799cbc5d5f-vzztz" podUID="9622b1bb-2668-4e94-a02f-e1c605d3a187" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 21 17:49:12.175724 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:49:12.175685    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8e991-predictor-5749cbc5db-9qmzg" podUID="d7a5c652-026f-4a13-9316-fde065088fb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 21 17:49:21.172477 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:49:21.172431    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8e991-predictor-799cbc5d5f-vzztz" podUID="9622b1bb-2668-4e94-a02f-e1c605d3a187" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 21 17:49:22.176293 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:49:22.176256    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8e991-predictor-5749cbc5db-9qmzg" podUID="d7a5c652-026f-4a13-9316-fde065088fb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 21 17:49:31.173738 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:49:31.173706    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8e991-predictor-799cbc5d5f-vzztz"
Jul 21 17:49:32.176883 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:49:32.176852    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8e991-predictor-5749cbc5db-9qmzg"
Jul 21 17:57:38.767061 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:38.766982    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg"]
Jul 21 17:57:38.767619 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:38.767206    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg" podUID="584fd906-8440-4595-9371-2482e6946cdb" containerName="kserve-container" containerID="cri-o://33487f3732c3fd0cd559abee39dcfdfc4b47b828059617cf2d4219f14653c87b" gracePeriod=30
Jul 21 17:57:38.767619 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:38.767276    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg" podUID="584fd906-8440-4595-9371-2482e6946cdb" containerName="kube-rbac-proxy" containerID="cri-o://60d4ef1e42cdc343024bf803ccbf215943a5b4ff4009de89a264909d16ea1223" gracePeriod=30
Jul 21 17:57:38.839909 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:38.839879    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p"]
Jul 21 17:57:38.840264 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:38.840245    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26970d94-f960-4da3-a5fe-adfa5d370484" containerName="kserve-container"
Jul 21 17:57:38.840380 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:38.840265    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="26970d94-f960-4da3-a5fe-adfa5d370484" containerName="kserve-container"
Jul 21 17:57:38.840380 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:38.840283    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07fc8a39-1f80-4327-bf72-32bb289206e5" containerName="kube-rbac-proxy"
Jul 21 17:57:38.840380 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:38.840291    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="07fc8a39-1f80-4327-bf72-32bb289206e5" containerName="kube-rbac-proxy"
Jul 21 17:57:38.840380 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:38.840318    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07fc8a39-1f80-4327-bf72-32bb289206e5" containerName="kserve-container"
Jul 21 17:57:38.840380 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:38.840327    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="07fc8a39-1f80-4327-bf72-32bb289206e5" containerName="kserve-container"
Jul 21 17:57:38.840380 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:38.840337    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26970d94-f960-4da3-a5fe-adfa5d370484" containerName="kube-rbac-proxy"
Jul 21 17:57:38.840380 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:38.840346    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="26970d94-f960-4da3-a5fe-adfa5d370484" containerName="kube-rbac-proxy"
Jul 21 17:57:38.840623 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:38.840448    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="26970d94-f960-4da3-a5fe-adfa5d370484" containerName="kube-rbac-proxy"
Jul 21 17:57:38.840623 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:38.840461    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="07fc8a39-1f80-4327-bf72-32bb289206e5" containerName="kserve-container"
Jul 21 17:57:38.840623 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:38.840470    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="26970d94-f960-4da3-a5fe-adfa5d370484" containerName="kserve-container"
Jul 21 17:57:38.840623 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:38.840482    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="07fc8a39-1f80-4327-bf72-32bb289206e5" containerName="kube-rbac-proxy"
Jul 21 17:57:38.843722 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:38.843708    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p"
Jul 21 17:57:38.847973 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:38.847941    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-edfd2-predictor-serving-cert\""
Jul 21 17:57:38.848178 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:38.847958    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-edfd2-kube-rbac-proxy-sar-config\""
Jul 21 17:57:38.856069 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:38.856026    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p"]
Jul 21 17:57:38.869889 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:38.869855    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7f959bd-b166-4bf3-9d39-061839b2f5fa-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p\" (UID: \"e7f959bd-b166-4bf3-9d39-061839b2f5fa\") " pod="kserve-ci-e2e-test/success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p"
Jul 21 17:57:38.869996 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:38.869915    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7f959bd-b166-4bf3-9d39-061839b2f5fa-proxy-tls\") pod \"success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p\" (UID: \"e7f959bd-b166-4bf3-9d39-061839b2f5fa\") " pod="kserve-ci-e2e-test/success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p"
Jul 21 17:57:38.869996 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:38.869954    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwn72\" (UniqueName: \"kubernetes.io/projected/e7f959bd-b166-4bf3-9d39-061839b2f5fa-kube-api-access-kwn72\") pod \"success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p\" (UID: \"e7f959bd-b166-4bf3-9d39-061839b2f5fa\") " pod="kserve-ci-e2e-test/success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p"
Jul 21 17:57:38.881411 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:38.881384    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr"]
Jul 21 17:57:38.881666 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:38.881640    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr" podUID="1633ad8f-0036-4ccf-893f-61ffdbb15b56" containerName="kserve-container" containerID="cri-o://c4c635e0ae90f9d0075ae239a908cf2c4226c47e543b6fdf3cc055a5fd8ad5aa" gracePeriod=30
Jul 21 17:57:38.881748 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:38.881681    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr" podUID="1633ad8f-0036-4ccf-893f-61ffdbb15b56" containerName="kube-rbac-proxy" containerID="cri-o://ae3e8dd7ff1eee25d0f9cf106cb45fe5c80ebd31a9de74f33027038360562a95" gracePeriod=30
Jul 21 17:57:38.912684 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:38.912659    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-edfd2-predictor-84887cddcb-x27z7"]
Jul 21 17:57:38.916310 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:38.916291    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-edfd2-predictor-84887cddcb-x27z7"
Jul 21 17:57:38.918891 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:38.918870    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-edfd2-predictor-serving-cert\""
Jul 21 17:57:38.918978 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:38.918900    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-edfd2-kube-rbac-proxy-sar-config\""
Jul 21 17:57:38.927282 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:38.927226    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-edfd2-predictor-84887cddcb-x27z7"]
Jul 21 17:57:38.971123 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:38.971092    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7f959bd-b166-4bf3-9d39-061839b2f5fa-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p\" (UID: \"e7f959bd-b166-4bf3-9d39-061839b2f5fa\") " pod="kserve-ci-e2e-test/success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p"
Jul 21 17:57:38.971332 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:38.971174    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7f959bd-b166-4bf3-9d39-061839b2f5fa-proxy-tls\") pod \"success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p\" (UID: \"e7f959bd-b166-4bf3-9d39-061839b2f5fa\") " pod="kserve-ci-e2e-test/success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p"
Jul 21 17:57:38.971332 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:38.971214    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwn72\" (UniqueName: \"kubernetes.io/projected/e7f959bd-b166-4bf3-9d39-061839b2f5fa-kube-api-access-kwn72\") pod \"success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p\" (UID: \"e7f959bd-b166-4bf3-9d39-061839b2f5fa\") " pod="kserve-ci-e2e-test/success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p"
Jul 21 17:57:38.971332 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:38.971250    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06da2dde-96ec-4bbe-9eac-69de21849e6a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-edfd2-predictor-84887cddcb-x27z7\" (UID: \"06da2dde-96ec-4bbe-9eac-69de21849e6a\") " pod="kserve-ci-e2e-test/error-404-isvc-edfd2-predictor-84887cddcb-x27z7"
Jul 21 17:57:38.971332 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:38.971308    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06da2dde-96ec-4bbe-9eac-69de21849e6a-proxy-tls\") pod \"error-404-isvc-edfd2-predictor-84887cddcb-x27z7\" (UID: \"06da2dde-96ec-4bbe-9eac-69de21849e6a\") " pod="kserve-ci-e2e-test/error-404-isvc-edfd2-predictor-84887cddcb-x27z7"
Jul 21 17:57:38.971549 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:57:38.971326    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-edfd2-predictor-serving-cert: secret "success-200-isvc-edfd2-predictor-serving-cert" not found
Jul 21 17:57:38.971549 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:38.971336    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mkdz\" (UniqueName: \"kubernetes.io/projected/06da2dde-96ec-4bbe-9eac-69de21849e6a-kube-api-access-7mkdz\") pod \"error-404-isvc-edfd2-predictor-84887cddcb-x27z7\" (UID: \"06da2dde-96ec-4bbe-9eac-69de21849e6a\") " pod="kserve-ci-e2e-test/error-404-isvc-edfd2-predictor-84887cddcb-x27z7"
Jul 21 17:57:38.971549 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:57:38.971390    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7f959bd-b166-4bf3-9d39-061839b2f5fa-proxy-tls podName:e7f959bd-b166-4bf3-9d39-061839b2f5fa nodeName:}" failed. No retries permitted until 2026-07-21 17:57:39.471371172 +0000 UTC m=+1887.233510757 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e7f959bd-b166-4bf3-9d39-061839b2f5fa-proxy-tls") pod "success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p" (UID: "e7f959bd-b166-4bf3-9d39-061839b2f5fa") : secret "success-200-isvc-edfd2-predictor-serving-cert" not found
Jul 21 17:57:38.976877 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:38.972476    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7f959bd-b166-4bf3-9d39-061839b2f5fa-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p\" (UID: \"e7f959bd-b166-4bf3-9d39-061839b2f5fa\") " pod="kserve-ci-e2e-test/success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p"
Jul 21 17:57:38.980614 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:38.980585    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwn72\" (UniqueName: \"kubernetes.io/projected/e7f959bd-b166-4bf3-9d39-061839b2f5fa-kube-api-access-kwn72\") pod \"success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p\" (UID: \"e7f959bd-b166-4bf3-9d39-061839b2f5fa\") " pod="kserve-ci-e2e-test/success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p"
Jul 21 17:57:39.072503 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:39.072418    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mkdz\" (UniqueName: \"kubernetes.io/projected/06da2dde-96ec-4bbe-9eac-69de21849e6a-kube-api-access-7mkdz\") pod \"error-404-isvc-edfd2-predictor-84887cddcb-x27z7\" (UID: \"06da2dde-96ec-4bbe-9eac-69de21849e6a\") " pod="kserve-ci-e2e-test/error-404-isvc-edfd2-predictor-84887cddcb-x27z7"
Jul 21 17:57:39.072665 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:39.072508    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06da2dde-96ec-4bbe-9eac-69de21849e6a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-edfd2-predictor-84887cddcb-x27z7\" (UID: \"06da2dde-96ec-4bbe-9eac-69de21849e6a\") " pod="kserve-ci-e2e-test/error-404-isvc-edfd2-predictor-84887cddcb-x27z7"
Jul 21 17:57:39.072665 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:39.072540    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06da2dde-96ec-4bbe-9eac-69de21849e6a-proxy-tls\") pod \"error-404-isvc-edfd2-predictor-84887cddcb-x27z7\" (UID: \"06da2dde-96ec-4bbe-9eac-69de21849e6a\") " pod="kserve-ci-e2e-test/error-404-isvc-edfd2-predictor-84887cddcb-x27z7"
Jul 21 17:57:39.072665 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:57:39.072620    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-edfd2-predictor-serving-cert: secret "error-404-isvc-edfd2-predictor-serving-cert" not found
Jul 21 17:57:39.072821 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:57:39.072673    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06da2dde-96ec-4bbe-9eac-69de21849e6a-proxy-tls podName:06da2dde-96ec-4bbe-9eac-69de21849e6a nodeName:}" failed. No retries permitted until 2026-07-21 17:57:39.572658215 +0000 UTC m=+1887.334797797 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/06da2dde-96ec-4bbe-9eac-69de21849e6a-proxy-tls") pod "error-404-isvc-edfd2-predictor-84887cddcb-x27z7" (UID: "06da2dde-96ec-4bbe-9eac-69de21849e6a") : secret "error-404-isvc-edfd2-predictor-serving-cert" not found
Jul 21 17:57:39.073119 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:39.073097    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06da2dde-96ec-4bbe-9eac-69de21849e6a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-edfd2-predictor-84887cddcb-x27z7\" (UID: \"06da2dde-96ec-4bbe-9eac-69de21849e6a\") " pod="kserve-ci-e2e-test/error-404-isvc-edfd2-predictor-84887cddcb-x27z7"
Jul 21 17:57:39.081833 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:39.081805    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mkdz\" (UniqueName: \"kubernetes.io/projected/06da2dde-96ec-4bbe-9eac-69de21849e6a-kube-api-access-7mkdz\") pod \"error-404-isvc-edfd2-predictor-84887cddcb-x27z7\" (UID: \"06da2dde-96ec-4bbe-9eac-69de21849e6a\") " pod="kserve-ci-e2e-test/error-404-isvc-edfd2-predictor-84887cddcb-x27z7"
Jul 21 17:57:39.177971 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:39.177937    2563 generic.go:356] "Generic (PLEG): container finished" podID="584fd906-8440-4595-9371-2482e6946cdb" containerID="60d4ef1e42cdc343024bf803ccbf215943a5b4ff4009de89a264909d16ea1223" exitCode=2
Jul 21 17:57:39.178149 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:39.178004    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg" event={"ID":"584fd906-8440-4595-9371-2482e6946cdb","Type":"ContainerDied","Data":"60d4ef1e42cdc343024bf803ccbf215943a5b4ff4009de89a264909d16ea1223"}
Jul 21 17:57:39.179593 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:39.179573    2563 generic.go:356] "Generic (PLEG): container finished" podID="1633ad8f-0036-4ccf-893f-61ffdbb15b56" containerID="ae3e8dd7ff1eee25d0f9cf106cb45fe5c80ebd31a9de74f33027038360562a95" exitCode=2
Jul 21 17:57:39.179693 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:39.179639    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr" event={"ID":"1633ad8f-0036-4ccf-893f-61ffdbb15b56","Type":"ContainerDied","Data":"ae3e8dd7ff1eee25d0f9cf106cb45fe5c80ebd31a9de74f33027038360562a95"}
Jul 21 17:57:39.476755 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:39.476713    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7f959bd-b166-4bf3-9d39-061839b2f5fa-proxy-tls\") pod \"success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p\" (UID: \"e7f959bd-b166-4bf3-9d39-061839b2f5fa\") " pod="kserve-ci-e2e-test/success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p"
Jul 21 17:57:39.479292 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:39.479261    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7f959bd-b166-4bf3-9d39-061839b2f5fa-proxy-tls\") pod \"success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p\" (UID: \"e7f959bd-b166-4bf3-9d39-061839b2f5fa\") " pod="kserve-ci-e2e-test/success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p"
Jul 21 17:57:39.577681 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:39.577643    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06da2dde-96ec-4bbe-9eac-69de21849e6a-proxy-tls\") pod \"error-404-isvc-edfd2-predictor-84887cddcb-x27z7\" (UID: \"06da2dde-96ec-4bbe-9eac-69de21849e6a\") " pod="kserve-ci-e2e-test/error-404-isvc-edfd2-predictor-84887cddcb-x27z7"
Jul 21 17:57:39.580076 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:39.580057    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06da2dde-96ec-4bbe-9eac-69de21849e6a-proxy-tls\") pod \"error-404-isvc-edfd2-predictor-84887cddcb-x27z7\" (UID: \"06da2dde-96ec-4bbe-9eac-69de21849e6a\") " pod="kserve-ci-e2e-test/error-404-isvc-edfd2-predictor-84887cddcb-x27z7"
Jul 21 17:57:39.757807 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:39.757722    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p"
Jul 21 17:57:39.826771 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:39.826739    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-edfd2-predictor-84887cddcb-x27z7"
Jul 21 17:57:39.892870 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:39.892842    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p"]
Jul 21 17:57:39.895835 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:57:39.895796    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f959bd_b166_4bf3_9d39_061839b2f5fa.slice/crio-dfa4d98eeed7079d9596bd0aec9bfd071e324a04d21d0eca884a1763c6eb128e WatchSource:0}: Error finding container dfa4d98eeed7079d9596bd0aec9bfd071e324a04d21d0eca884a1763c6eb128e: Status 404 returned error can't find the container with id dfa4d98eeed7079d9596bd0aec9bfd071e324a04d21d0eca884a1763c6eb128e
Jul 21 17:57:39.968578 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:39.968529    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-edfd2-predictor-84887cddcb-x27z7"]
Jul 21 17:57:39.970471 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:57:39.970443    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06da2dde_96ec_4bbe_9eac_69de21849e6a.slice/crio-7015f0e2671ae41290a1a1a0dc460b9d5108d78c5702d914e78196aef99a98c6 WatchSource:0}: Error finding container 7015f0e2671ae41290a1a1a0dc460b9d5108d78c5702d914e78196aef99a98c6: Status 404 returned error can't find the container with id 7015f0e2671ae41290a1a1a0dc460b9d5108d78c5702d914e78196aef99a98c6
Jul 21 17:57:40.185078 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:40.185048    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-edfd2-predictor-84887cddcb-x27z7" event={"ID":"06da2dde-96ec-4bbe-9eac-69de21849e6a","Type":"ContainerStarted","Data":"5877f5a317d6daa3fecc11f9430c7fe212a8614dbfb359e31a1250bf93d4a484"}
Jul 21 17:57:40.185269 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:40.185113    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-edfd2-predictor-84887cddcb-x27z7" event={"ID":"06da2dde-96ec-4bbe-9eac-69de21849e6a","Type":"ContainerStarted","Data":"47ea08b9889600f8af6095ea291e8448db2d004c230e371703f672767b5bc002"}
Jul 21 17:57:40.185269 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:40.185164    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-edfd2-predictor-84887cddcb-x27z7" event={"ID":"06da2dde-96ec-4bbe-9eac-69de21849e6a","Type":"ContainerStarted","Data":"7015f0e2671ae41290a1a1a0dc460b9d5108d78c5702d914e78196aef99a98c6"}
Jul 21 17:57:40.185269 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:40.185197    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-edfd2-predictor-84887cddcb-x27z7"
Jul 21 17:57:40.186922 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:40.186894    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p" event={"ID":"e7f959bd-b166-4bf3-9d39-061839b2f5fa","Type":"ContainerStarted","Data":"e7ec9e69edd1b1897e3a94f42ce85b04ad763b7c548db8d25a00200cf3aa838f"}
Jul 21 17:57:40.187073 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:40.186945    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p"
Jul 21 17:57:40.187073 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:40.186965    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p"
Jul 21 17:57:40.187073 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:40.186988    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p" event={"ID":"e7f959bd-b166-4bf3-9d39-061839b2f5fa","Type":"ContainerStarted","Data":"cc79644cc1ee7a1400ae29b8c07ec28affa8b322dcdfcb5749c1d3dbab31f52e"}
Jul 21 17:57:40.187073 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:40.187011    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p" event={"ID":"e7f959bd-b166-4bf3-9d39-061839b2f5fa","Type":"ContainerStarted","Data":"dfa4d98eeed7079d9596bd0aec9bfd071e324a04d21d0eca884a1763c6eb128e"}
Jul 21 17:57:40.188211 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:40.188186    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p" podUID="e7f959bd-b166-4bf3-9d39-061839b2f5fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 21 17:57:40.203878 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:40.203833    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-edfd2-predictor-84887cddcb-x27z7" podStartSLOduration=2.203818448 podStartE2EDuration="2.203818448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:57:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:57:40.202775742 +0000 UTC m=+1887.964915346" watchObservedRunningTime="2026-07-21 17:57:40.203818448 +0000 UTC m=+1887.965958052"
Jul 21 17:57:40.218978 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:40.218925    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p" podStartSLOduration=2.218907389 podStartE2EDuration="2.218907389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:57:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:57:40.218401182 +0000 UTC m=+1887.980540791" watchObservedRunningTime="2026-07-21 17:57:40.218907389 +0000 UTC m=+1887.981046994"
Jul 21 17:57:41.032601 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:41.032560    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr" podUID="1633ad8f-0036-4ccf-893f-61ffdbb15b56" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 21 17:57:41.032995 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:41.032569    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg" podUID="584fd906-8440-4595-9371-2482e6946cdb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 21 17:57:41.036822 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:41.036798    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr" podUID="1633ad8f-0036-4ccf-893f-61ffdbb15b56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 21 17:57:41.036882 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:41.036798    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg" podUID="584fd906-8440-4595-9371-2482e6946cdb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 21 17:57:41.191515 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:41.191478    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p" podUID="e7f959bd-b166-4bf3-9d39-061839b2f5fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 21 17:57:41.191708 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:41.191499    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-edfd2-predictor-84887cddcb-x27z7"
Jul 21 17:57:41.192568 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:41.192543    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-edfd2-predictor-84887cddcb-x27z7" podUID="06da2dde-96ec-4bbe-9eac-69de21849e6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 21 17:57:42.014569 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:42.014547    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg"
Jul 21 17:57:42.098826 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:42.098797    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/584fd906-8440-4595-9371-2482e6946cdb-kube-rbac-proxy-sar-config\") pod \"584fd906-8440-4595-9371-2482e6946cdb\" (UID: \"584fd906-8440-4595-9371-2482e6946cdb\") "
Jul 21 17:57:42.099219 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:42.098846    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88r7w\" (UniqueName: \"kubernetes.io/projected/584fd906-8440-4595-9371-2482e6946cdb-kube-api-access-88r7w\") pod \"584fd906-8440-4595-9371-2482e6946cdb\" (UID: \"584fd906-8440-4595-9371-2482e6946cdb\") "
Jul 21 17:57:42.099219 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:42.098874    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/584fd906-8440-4595-9371-2482e6946cdb-proxy-tls\") pod \"584fd906-8440-4595-9371-2482e6946cdb\" (UID: \"584fd906-8440-4595-9371-2482e6946cdb\") "
Jul 21 17:57:42.099219 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:42.099176    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/584fd906-8440-4595-9371-2482e6946cdb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "584fd906-8440-4595-9371-2482e6946cdb" (UID: "584fd906-8440-4595-9371-2482e6946cdb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:57:42.147566 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:42.147528    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/584fd906-8440-4595-9371-2482e6946cdb-kube-api-access-88r7w" (OuterVolumeSpecName: "kube-api-access-88r7w") pod "584fd906-8440-4595-9371-2482e6946cdb" (UID: "584fd906-8440-4595-9371-2482e6946cdb"). InnerVolumeSpecName "kube-api-access-88r7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:57:42.147682 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:42.147565    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/584fd906-8440-4595-9371-2482e6946cdb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "584fd906-8440-4595-9371-2482e6946cdb" (UID: "584fd906-8440-4595-9371-2482e6946cdb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:57:42.196191 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:42.196159    2563 generic.go:356] "Generic (PLEG): container finished" podID="584fd906-8440-4595-9371-2482e6946cdb" containerID="33487f3732c3fd0cd559abee39dcfdfc4b47b828059617cf2d4219f14653c87b" exitCode=0
Jul 21 17:57:42.196312 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:42.196206    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg"
Jul 21 17:57:42.196312 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:42.196239    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg" event={"ID":"584fd906-8440-4595-9371-2482e6946cdb","Type":"ContainerDied","Data":"33487f3732c3fd0cd559abee39dcfdfc4b47b828059617cf2d4219f14653c87b"}
Jul 21 17:57:42.196312 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:42.196297    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg" event={"ID":"584fd906-8440-4595-9371-2482e6946cdb","Type":"ContainerDied","Data":"dc273eed8a07e47d8a916025d73ae56077fc9ae46ac9a0967af77b09888e0884"}
Jul 21 17:57:42.196455 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:42.196331    2563 scope.go:117] "RemoveContainer" containerID="60d4ef1e42cdc343024bf803ccbf215943a5b4ff4009de89a264909d16ea1223"
Jul 21 17:57:42.198183 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:42.198161    2563 generic.go:356] "Generic (PLEG): container finished" podID="1633ad8f-0036-4ccf-893f-61ffdbb15b56" containerID="c4c635e0ae90f9d0075ae239a908cf2c4226c47e543b6fdf3cc055a5fd8ad5aa" exitCode=0
Jul 21 17:57:42.198286 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:42.198234    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr" event={"ID":"1633ad8f-0036-4ccf-893f-61ffdbb15b56","Type":"ContainerDied","Data":"c4c635e0ae90f9d0075ae239a908cf2c4226c47e543b6fdf3cc055a5fd8ad5aa"}
Jul 21 17:57:42.198482 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:42.198446    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-edfd2-predictor-84887cddcb-x27z7" podUID="06da2dde-96ec-4bbe-9eac-69de21849e6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 21 17:57:42.199420 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:42.199404    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88r7w\" (UniqueName: \"kubernetes.io/projected/584fd906-8440-4595-9371-2482e6946cdb-kube-api-access-88r7w\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:57:42.199520 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:42.199422    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/584fd906-8440-4595-9371-2482e6946cdb-proxy-tls\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:57:42.199520 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:42.199432    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/584fd906-8440-4595-9371-2482e6946cdb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:57:42.213937 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:42.213921    2563 scope.go:117] "RemoveContainer" containerID="33487f3732c3fd0cd559abee39dcfdfc4b47b828059617cf2d4219f14653c87b"
Jul 21 17:57:42.221251 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:42.221218    2563 scope.go:117] "RemoveContainer" containerID="60d4ef1e42cdc343024bf803ccbf215943a5b4ff4009de89a264909d16ea1223"
Jul 21 17:57:42.221472 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:57:42.221452    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60d4ef1e42cdc343024bf803ccbf215943a5b4ff4009de89a264909d16ea1223\": container with ID starting with 60d4ef1e42cdc343024bf803ccbf215943a5b4ff4009de89a264909d16ea1223 not found: ID does not exist" containerID="60d4ef1e42cdc343024bf803ccbf215943a5b4ff4009de89a264909d16ea1223"
Jul 21 17:57:42.221523 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:42.221480    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60d4ef1e42cdc343024bf803ccbf215943a5b4ff4009de89a264909d16ea1223"} err="failed to get container status \"60d4ef1e42cdc343024bf803ccbf215943a5b4ff4009de89a264909d16ea1223\": rpc error: code = NotFound desc = could not find container \"60d4ef1e42cdc343024bf803ccbf215943a5b4ff4009de89a264909d16ea1223\": container with ID starting with 60d4ef1e42cdc343024bf803ccbf215943a5b4ff4009de89a264909d16ea1223 not found: ID does not exist"
Jul 21 17:57:42.221523 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:42.221495    2563 scope.go:117] "RemoveContainer" containerID="33487f3732c3fd0cd559abee39dcfdfc4b47b828059617cf2d4219f14653c87b"
Jul 21 17:57:42.221759 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:57:42.221742    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33487f3732c3fd0cd559abee39dcfdfc4b47b828059617cf2d4219f14653c87b\": container with ID starting with 33487f3732c3fd0cd559abee39dcfdfc4b47b828059617cf2d4219f14653c87b not found: ID does not exist" containerID="33487f3732c3fd0cd559abee39dcfdfc4b47b828059617cf2d4219f14653c87b"
Jul 21 17:57:42.221810 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:42.221765    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33487f3732c3fd0cd559abee39dcfdfc4b47b828059617cf2d4219f14653c87b"} err="failed to get container status \"33487f3732c3fd0cd559abee39dcfdfc4b47b828059617cf2d4219f14653c87b\": rpc error: code = NotFound desc = could not find container \"33487f3732c3fd0cd559abee39dcfdfc4b47b828059617cf2d4219f14653c87b\": container with ID starting with 33487f3732c3fd0cd559abee39dcfdfc4b47b828059617cf2d4219f14653c87b not found: ID does not exist"
Jul 21 17:57:42.240496 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:42.240468    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg"]
Jul 21 17:57:42.253211 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:42.253191    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4c9b0-predictor-698bd5c5d7-z6hkg"]
Jul 21 17:57:42.618765 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:42.618743    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr"
Jul 21 17:57:42.703967 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:42.703934    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gklh2\" (UniqueName: \"kubernetes.io/projected/1633ad8f-0036-4ccf-893f-61ffdbb15b56-kube-api-access-gklh2\") pod \"1633ad8f-0036-4ccf-893f-61ffdbb15b56\" (UID: \"1633ad8f-0036-4ccf-893f-61ffdbb15b56\") "
Jul 21 17:57:42.704161 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:42.703989    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1633ad8f-0036-4ccf-893f-61ffdbb15b56-kube-rbac-proxy-sar-config\") pod \"1633ad8f-0036-4ccf-893f-61ffdbb15b56\" (UID: \"1633ad8f-0036-4ccf-893f-61ffdbb15b56\") "
Jul 21 17:57:42.704161 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:42.704016    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1633ad8f-0036-4ccf-893f-61ffdbb15b56-proxy-tls\") pod \"1633ad8f-0036-4ccf-893f-61ffdbb15b56\" (UID: \"1633ad8f-0036-4ccf-893f-61ffdbb15b56\") "
Jul 21 17:57:42.704421 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:42.704397    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1633ad8f-0036-4ccf-893f-61ffdbb15b56-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1633ad8f-0036-4ccf-893f-61ffdbb15b56" (UID: "1633ad8f-0036-4ccf-893f-61ffdbb15b56"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:57:42.706262 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:42.706240    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1633ad8f-0036-4ccf-893f-61ffdbb15b56-kube-api-access-gklh2" (OuterVolumeSpecName: "kube-api-access-gklh2") pod "1633ad8f-0036-4ccf-893f-61ffdbb15b56" (UID: "1633ad8f-0036-4ccf-893f-61ffdbb15b56"). InnerVolumeSpecName "kube-api-access-gklh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:57:42.706262 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:42.706258    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1633ad8f-0036-4ccf-893f-61ffdbb15b56-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1633ad8f-0036-4ccf-893f-61ffdbb15b56" (UID: "1633ad8f-0036-4ccf-893f-61ffdbb15b56"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:57:42.766379 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:42.763993    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="584fd906-8440-4595-9371-2482e6946cdb" path="/var/lib/kubelet/pods/584fd906-8440-4595-9371-2482e6946cdb/volumes"
Jul 21 17:57:42.804605 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:42.804568    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gklh2\" (UniqueName: \"kubernetes.io/projected/1633ad8f-0036-4ccf-893f-61ffdbb15b56-kube-api-access-gklh2\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:57:42.804605 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:42.804595    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1633ad8f-0036-4ccf-893f-61ffdbb15b56-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:57:42.804605 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:42.804605    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1633ad8f-0036-4ccf-893f-61ffdbb15b56-proxy-tls\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:57:43.204938 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:43.204897    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr" event={"ID":"1633ad8f-0036-4ccf-893f-61ffdbb15b56","Type":"ContainerDied","Data":"f7c003e35c5b249a5394242d1a57622bbcf2aa79ebc99fb8074288243cd79366"}
Jul 21 17:57:43.204938 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:43.204936    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr"
Jul 21 17:57:43.205432 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:43.204962    2563 scope.go:117] "RemoveContainer" containerID="ae3e8dd7ff1eee25d0f9cf106cb45fe5c80ebd31a9de74f33027038360562a95"
Jul 21 17:57:43.215882 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:43.215868    2563 scope.go:117] "RemoveContainer" containerID="c4c635e0ae90f9d0075ae239a908cf2c4226c47e543b6fdf3cc055a5fd8ad5aa"
Jul 21 17:57:43.222096 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:43.222072    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr"]
Jul 21 17:57:43.228343 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:43.228323    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4c9b0-predictor-7fc4f48954-k7lmr"]
Jul 21 17:57:44.763741 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:44.763713    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1633ad8f-0036-4ccf-893f-61ffdbb15b56" path="/var/lib/kubelet/pods/1633ad8f-0036-4ccf-893f-61ffdbb15b56/volumes"
Jul 21 17:57:46.197439 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:46.197410    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p"
Jul 21 17:57:46.197945 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:46.197922    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p" podUID="e7f959bd-b166-4bf3-9d39-061839b2f5fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 21 17:57:47.202449 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:47.202425    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-edfd2-predictor-84887cddcb-x27z7"
Jul 21 17:57:47.202889 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:47.202865    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-edfd2-predictor-84887cddcb-x27z7" podUID="06da2dde-96ec-4bbe-9eac-69de21849e6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 21 17:57:56.198179 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:56.198117    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p" podUID="e7f959bd-b166-4bf3-9d39-061839b2f5fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 21 17:57:57.202877 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:57.202840    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-edfd2-predictor-84887cddcb-x27z7" podUID="06da2dde-96ec-4bbe-9eac-69de21849e6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 21 17:57:58.598203 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:58.598170    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8e991-predictor-799cbc5d5f-vzztz"]
Jul 21 17:57:58.598606 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:58.598429    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8e991-predictor-799cbc5d5f-vzztz" podUID="9622b1bb-2668-4e94-a02f-e1c605d3a187" containerName="kserve-container" containerID="cri-o://5ae44d193f106e37bd84bdc83aa606d5bd37cf6176e2e4207837cb2e8dc4c785" gracePeriod=30
Jul 21 17:57:58.598606 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:58.598487    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8e991-predictor-799cbc5d5f-vzztz" podUID="9622b1bb-2668-4e94-a02f-e1c605d3a187" containerName="kube-rbac-proxy" containerID="cri-o://570ae0d7b305cccc4a671d55016c2e47861f3fd2beb36a1c8533a3e4488ea419" gracePeriod=30
Jul 21 17:57:58.645718 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:58.645685    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8e991-predictor-5749cbc5db-9qmzg"]
Jul 21 17:57:58.646034 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:58.645980    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8e991-predictor-5749cbc5db-9qmzg" podUID="d7a5c652-026f-4a13-9316-fde065088fb1" containerName="kserve-container" containerID="cri-o://f3ef3f03e0e4116bc607f81f35f16c01436c2e622abc3c3f53ce7d79378f482d" gracePeriod=30
Jul 21 17:57:58.646108 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:58.646057    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8e991-predictor-5749cbc5db-9qmzg" podUID="d7a5c652-026f-4a13-9316-fde065088fb1" containerName="kube-rbac-proxy" containerID="cri-o://a29f5b5e0eaef97591f29cc4da7163683d3dfe71b648db64d5cb4cd46d386445" gracePeriod=30
Jul 21 17:57:58.693168 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:58.693124    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9"]
Jul 21 17:57:58.693479 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:58.693466    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1633ad8f-0036-4ccf-893f-61ffdbb15b56" containerName="kserve-container"
Jul 21 17:57:58.693586 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:58.693481    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1633ad8f-0036-4ccf-893f-61ffdbb15b56" containerName="kserve-container"
Jul 21 17:57:58.693586 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:58.693496    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1633ad8f-0036-4ccf-893f-61ffdbb15b56" containerName="kube-rbac-proxy"
Jul 21 17:57:58.693586 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:58.693501    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1633ad8f-0036-4ccf-893f-61ffdbb15b56" containerName="kube-rbac-proxy"
Jul 21 17:57:58.693586 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:58.693509    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="584fd906-8440-4595-9371-2482e6946cdb" containerName="kserve-container"
Jul 21 17:57:58.693586 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:58.693515    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="584fd906-8440-4595-9371-2482e6946cdb" containerName="kserve-container"
Jul 21 17:57:58.693586 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:58.693521    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="584fd906-8440-4595-9371-2482e6946cdb" containerName="kube-rbac-proxy"
Jul 21 17:57:58.693586 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:58.693526    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="584fd906-8440-4595-9371-2482e6946cdb" containerName="kube-rbac-proxy"
Jul 21 17:57:58.693586 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:58.693573    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="584fd906-8440-4595-9371-2482e6946cdb" containerName="kube-rbac-proxy"
Jul 21 17:57:58.693586 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:58.693580    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1633ad8f-0036-4ccf-893f-61ffdbb15b56" containerName="kube-rbac-proxy"
Jul 21 17:57:58.693586 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:58.693589    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="584fd906-8440-4595-9371-2482e6946cdb" containerName="kserve-container"
Jul 21 17:57:58.693897 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:58.693595    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1633ad8f-0036-4ccf-893f-61ffdbb15b56" containerName="kserve-container"
Jul 21 17:57:58.698127 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:58.698110    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9"
Jul 21 17:57:58.700673 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:58.700646    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-984cf-predictor-serving-cert\""
Jul 21 17:57:58.700775 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:58.700689    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-984cf-kube-rbac-proxy-sar-config\""
Jul 21 17:57:58.707618 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:58.707585    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9"]
Jul 21 17:57:58.788776 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:58.788752    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h"]
Jul 21 17:57:58.792044 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:58.792027    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h"
Jul 21 17:57:58.794614 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:58.794594    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-984cf-predictor-serving-cert\""
Jul 21 17:57:58.794614 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:58.794603    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-984cf-kube-rbac-proxy-sar-config\""
Jul 21 17:57:58.802182 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:58.802161    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h"]
Jul 21 17:57:58.838963 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:58.838938    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59d6e22f-3560-457e-8d5e-95cd24157bb9-proxy-tls\") pod \"success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9\" (UID: \"59d6e22f-3560-457e-8d5e-95cd24157bb9\") " pod="kserve-ci-e2e-test/success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9"
Jul 21 17:57:58.839122 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:58.839101    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59d6e22f-3560-457e-8d5e-95cd24157bb9-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9\" (UID: \"59d6e22f-3560-457e-8d5e-95cd24157bb9\") " pod="kserve-ci-e2e-test/success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9"
Jul 21 17:57:58.839236 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:58.839216    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95l75\" (UniqueName: \"kubernetes.io/projected/59d6e22f-3560-457e-8d5e-95cd24157bb9-kube-api-access-95l75\") pod \"success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9\" (UID: \"59d6e22f-3560-457e-8d5e-95cd24157bb9\") " pod="kserve-ci-e2e-test/success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9"
Jul 21 17:57:58.940515 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:58.940485    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59d6e22f-3560-457e-8d5e-95cd24157bb9-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9\" (UID: \"59d6e22f-3560-457e-8d5e-95cd24157bb9\") " pod="kserve-ci-e2e-test/success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9"
Jul 21 17:57:58.940693 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:58.940525    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/724ce6ed-3557-4232-9b1c-1035c5c14be7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h\" (UID: \"724ce6ed-3557-4232-9b1c-1035c5c14be7\") " pod="kserve-ci-e2e-test/error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h"
Jul 21 17:57:58.940693 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:58.940554    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95l75\" (UniqueName: \"kubernetes.io/projected/59d6e22f-3560-457e-8d5e-95cd24157bb9-kube-api-access-95l75\") pod \"success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9\" (UID: \"59d6e22f-3560-457e-8d5e-95cd24157bb9\") " pod="kserve-ci-e2e-test/success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9"
Jul 21 17:57:58.940693 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:58.940627    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/724ce6ed-3557-4232-9b1c-1035c5c14be7-proxy-tls\") pod \"error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h\" (UID: \"724ce6ed-3557-4232-9b1c-1035c5c14be7\") " pod="kserve-ci-e2e-test/error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h"
Jul 21 17:57:58.940819 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:58.940751    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c4dg\" (UniqueName: \"kubernetes.io/projected/724ce6ed-3557-4232-9b1c-1035c5c14be7-kube-api-access-4c4dg\") pod \"error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h\" (UID: \"724ce6ed-3557-4232-9b1c-1035c5c14be7\") " pod="kserve-ci-e2e-test/error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h"
Jul 21 17:57:58.940867 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:58.940833    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59d6e22f-3560-457e-8d5e-95cd24157bb9-proxy-tls\") pod \"success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9\" (UID: \"59d6e22f-3560-457e-8d5e-95cd24157bb9\") " pod="kserve-ci-e2e-test/success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9"
Jul 21 17:57:58.940946 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:57:58.940933    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-984cf-predictor-serving-cert: secret "success-200-isvc-984cf-predictor-serving-cert" not found
Jul 21 17:57:58.940995 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:57:58.940985    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59d6e22f-3560-457e-8d5e-95cd24157bb9-proxy-tls podName:59d6e22f-3560-457e-8d5e-95cd24157bb9 nodeName:}" failed. No retries permitted until 2026-07-21 17:57:59.440971088 +0000 UTC m=+1907.203110670 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/59d6e22f-3560-457e-8d5e-95cd24157bb9-proxy-tls") pod "success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9" (UID: "59d6e22f-3560-457e-8d5e-95cd24157bb9") : secret "success-200-isvc-984cf-predictor-serving-cert" not found
Jul 21 17:57:58.941265 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:58.941224    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59d6e22f-3560-457e-8d5e-95cd24157bb9-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9\" (UID: \"59d6e22f-3560-457e-8d5e-95cd24157bb9\") " pod="kserve-ci-e2e-test/success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9"
Jul 21 17:57:58.951715 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:58.951681    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95l75\" (UniqueName: \"kubernetes.io/projected/59d6e22f-3560-457e-8d5e-95cd24157bb9-kube-api-access-95l75\") pod \"success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9\" (UID: \"59d6e22f-3560-457e-8d5e-95cd24157bb9\") " pod="kserve-ci-e2e-test/success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9"
Jul 21 17:57:59.041917 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:59.041885    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/724ce6ed-3557-4232-9b1c-1035c5c14be7-proxy-tls\") pod \"error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h\" (UID: \"724ce6ed-3557-4232-9b1c-1035c5c14be7\") " pod="kserve-ci-e2e-test/error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h"
Jul 21 17:57:59.041917 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:59.041919    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c4dg\" (UniqueName: \"kubernetes.io/projected/724ce6ed-3557-4232-9b1c-1035c5c14be7-kube-api-access-4c4dg\") pod \"error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h\" (UID: \"724ce6ed-3557-4232-9b1c-1035c5c14be7\") " pod="kserve-ci-e2e-test/error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h"
Jul 21 17:57:59.042120 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:59.041974    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/724ce6ed-3557-4232-9b1c-1035c5c14be7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h\" (UID: \"724ce6ed-3557-4232-9b1c-1035c5c14be7\") " pod="kserve-ci-e2e-test/error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h"
Jul 21 17:57:59.042635 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:59.042612    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/724ce6ed-3557-4232-9b1c-1035c5c14be7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h\" (UID: \"724ce6ed-3557-4232-9b1c-1035c5c14be7\") " pod="kserve-ci-e2e-test/error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h"
Jul 21 17:57:59.044395 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:59.044371    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/724ce6ed-3557-4232-9b1c-1035c5c14be7-proxy-tls\") pod \"error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h\" (UID: \"724ce6ed-3557-4232-9b1c-1035c5c14be7\") " pod="kserve-ci-e2e-test/error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h"
Jul 21 17:57:59.052453 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:59.052420    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c4dg\" (UniqueName: \"kubernetes.io/projected/724ce6ed-3557-4232-9b1c-1035c5c14be7-kube-api-access-4c4dg\") pod \"error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h\" (UID: \"724ce6ed-3557-4232-9b1c-1035c5c14be7\") " pod="kserve-ci-e2e-test/error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h"
Jul 21 17:57:59.102939 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:59.102908    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h"
Jul 21 17:57:59.236931 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:59.236905    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h"]
Jul 21 17:57:59.239065 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:57:59.239038    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod724ce6ed_3557_4232_9b1c_1035c5c14be7.slice/crio-cda58e921c45271ea359e89c99db6e97f746d87e98d37aeb494accea89be8a07 WatchSource:0}: Error finding container cda58e921c45271ea359e89c99db6e97f746d87e98d37aeb494accea89be8a07: Status 404 returned error can't find the container with id cda58e921c45271ea359e89c99db6e97f746d87e98d37aeb494accea89be8a07
Jul 21 17:57:59.272043 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:59.272018    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h" event={"ID":"724ce6ed-3557-4232-9b1c-1035c5c14be7","Type":"ContainerStarted","Data":"cda58e921c45271ea359e89c99db6e97f746d87e98d37aeb494accea89be8a07"}
Jul 21 17:57:59.273703 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:59.273680    2563 generic.go:356] "Generic (PLEG): container finished" podID="9622b1bb-2668-4e94-a02f-e1c605d3a187" containerID="570ae0d7b305cccc4a671d55016c2e47861f3fd2beb36a1c8533a3e4488ea419" exitCode=2
Jul 21 17:57:59.273812 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:59.273769    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8e991-predictor-799cbc5d5f-vzztz" event={"ID":"9622b1bb-2668-4e94-a02f-e1c605d3a187","Type":"ContainerDied","Data":"570ae0d7b305cccc4a671d55016c2e47861f3fd2beb36a1c8533a3e4488ea419"}
Jul 21 17:57:59.275378 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:59.275359    2563 generic.go:356] "Generic (PLEG): container finished" podID="d7a5c652-026f-4a13-9316-fde065088fb1" containerID="a29f5b5e0eaef97591f29cc4da7163683d3dfe71b648db64d5cb4cd46d386445" exitCode=2
Jul 21 17:57:59.275448 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:59.275402    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8e991-predictor-5749cbc5db-9qmzg" event={"ID":"d7a5c652-026f-4a13-9316-fde065088fb1","Type":"ContainerDied","Data":"a29f5b5e0eaef97591f29cc4da7163683d3dfe71b648db64d5cb4cd46d386445"}
Jul 21 17:57:59.445043 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:59.445008    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59d6e22f-3560-457e-8d5e-95cd24157bb9-proxy-tls\") pod \"success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9\" (UID: \"59d6e22f-3560-457e-8d5e-95cd24157bb9\") " pod="kserve-ci-e2e-test/success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9"
Jul 21 17:57:59.447587 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:59.447529    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59d6e22f-3560-457e-8d5e-95cd24157bb9-proxy-tls\") pod \"success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9\" (UID: \"59d6e22f-3560-457e-8d5e-95cd24157bb9\") " pod="kserve-ci-e2e-test/success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9"
Jul 21 17:57:59.609699 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:59.609620    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9"
Jul 21 17:57:59.745554 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:57:59.745530    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9"]
Jul 21 17:57:59.749174 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:57:59.748383    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59d6e22f_3560_457e_8d5e_95cd24157bb9.slice/crio-c0de17d52ea402db07d980efd36b804a6cd6e1ce1ab41a6d329d6676f95c5767 WatchSource:0}: Error finding container c0de17d52ea402db07d980efd36b804a6cd6e1ce1ab41a6d329d6676f95c5767: Status 404 returned error can't find the container with id c0de17d52ea402db07d980efd36b804a6cd6e1ce1ab41a6d329d6676f95c5767
Jul 21 17:58:00.280791 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:00.280757    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9" event={"ID":"59d6e22f-3560-457e-8d5e-95cd24157bb9","Type":"ContainerStarted","Data":"c71d51991c9b793ae57908b8c75624071e3d6b619b13c9cfb355d0296b949857"}
Jul 21 17:58:00.280987 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:00.280813    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9" event={"ID":"59d6e22f-3560-457e-8d5e-95cd24157bb9","Type":"ContainerStarted","Data":"5d3dfb2b2f18368624d62e3e6149e803b17d6314e49812fe859a50940ab41b1b"}
Jul 21 17:58:00.280987 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:00.280839    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9"
Jul 21 17:58:00.280987 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:00.280863    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9" event={"ID":"59d6e22f-3560-457e-8d5e-95cd24157bb9","Type":"ContainerStarted","Data":"c0de17d52ea402db07d980efd36b804a6cd6e1ce1ab41a6d329d6676f95c5767"}
Jul 21 17:58:00.282420 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:00.282399    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h" event={"ID":"724ce6ed-3557-4232-9b1c-1035c5c14be7","Type":"ContainerStarted","Data":"3d9c066046ea355312e57edbbafdfb7f3ad5441ca5dc563b65f8afe310ac589a"}
Jul 21 17:58:00.282512 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:00.282448    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h" event={"ID":"724ce6ed-3557-4232-9b1c-1035c5c14be7","Type":"ContainerStarted","Data":"d5955ecf75e490c2bad003e904c91940b98180e35e420bdc00f31d20b3175b51"}
Jul 21 17:58:00.282596 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:00.282580    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h"
Jul 21 17:58:00.282634 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:00.282627    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h"
Jul 21 17:58:00.283848 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:00.283828    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h" podUID="724ce6ed-3557-4232-9b1c-1035c5c14be7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 21 17:58:00.298960 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:00.298908    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9" podStartSLOduration=2.298893194 podStartE2EDuration="2.298893194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:57:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:58:00.297652493 +0000 UTC m=+1908.059792116" watchObservedRunningTime="2026-07-21 17:58:00.298893194 +0000 UTC m=+1908.061032799"
Jul 21 17:58:00.316930 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:00.316888    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h" podStartSLOduration=2.316877464 podStartE2EDuration="2.316877464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:57:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:58:00.315290505 +0000 UTC m=+1908.077430110" watchObservedRunningTime="2026-07-21 17:58:00.316877464 +0000 UTC m=+1908.079017067"
Jul 21 17:58:01.167574 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:01.167526    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8e991-predictor-799cbc5d5f-vzztz" podUID="9622b1bb-2668-4e94-a02f-e1c605d3a187" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Jul 21 17:58:01.172595 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:01.172567    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8e991-predictor-799cbc5d5f-vzztz" podUID="9622b1bb-2668-4e94-a02f-e1c605d3a187" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 21 17:58:01.286366 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:01.286322    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h" podUID="724ce6ed-3557-4232-9b1c-1035c5c14be7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 21 17:58:01.286552 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:01.286341    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9"
Jul 21 17:58:01.287518 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:01.287491    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9" podUID="59d6e22f-3560-457e-8d5e-95cd24157bb9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 21 17:58:02.171256 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:02.171208    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8e991-predictor-5749cbc5db-9qmzg" podUID="d7a5c652-026f-4a13-9316-fde065088fb1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Jul 21 17:58:02.175627 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:02.175590    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8e991-predictor-5749cbc5db-9qmzg" podUID="d7a5c652-026f-4a13-9316-fde065088fb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 21 17:58:02.290607 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:02.290563    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9" podUID="59d6e22f-3560-457e-8d5e-95cd24157bb9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 21 17:58:02.671425 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:02.671388    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8e991-predictor-799cbc5d5f-vzztz"
Jul 21 17:58:02.777329 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:02.777305    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9622b1bb-2668-4e94-a02f-e1c605d3a187-kube-rbac-proxy-sar-config\") pod \"9622b1bb-2668-4e94-a02f-e1c605d3a187\" (UID: \"9622b1bb-2668-4e94-a02f-e1c605d3a187\") "
Jul 21 17:58:02.777439 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:02.777364    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9622b1bb-2668-4e94-a02f-e1c605d3a187-proxy-tls\") pod \"9622b1bb-2668-4e94-a02f-e1c605d3a187\" (UID: \"9622b1bb-2668-4e94-a02f-e1c605d3a187\") "
Jul 21 17:58:02.777439 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:02.777409    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mffqv\" (UniqueName: \"kubernetes.io/projected/9622b1bb-2668-4e94-a02f-e1c605d3a187-kube-api-access-mffqv\") pod \"9622b1bb-2668-4e94-a02f-e1c605d3a187\" (UID: \"9622b1bb-2668-4e94-a02f-e1c605d3a187\") "
Jul 21 17:58:02.777706 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:02.777679    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9622b1bb-2668-4e94-a02f-e1c605d3a187-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9622b1bb-2668-4e94-a02f-e1c605d3a187" (UID: "9622b1bb-2668-4e94-a02f-e1c605d3a187"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:58:02.779504 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:02.779481    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9622b1bb-2668-4e94-a02f-e1c605d3a187-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9622b1bb-2668-4e94-a02f-e1c605d3a187" (UID: "9622b1bb-2668-4e94-a02f-e1c605d3a187"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:58:02.779586 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:02.779508    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9622b1bb-2668-4e94-a02f-e1c605d3a187-kube-api-access-mffqv" (OuterVolumeSpecName: "kube-api-access-mffqv") pod "9622b1bb-2668-4e94-a02f-e1c605d3a187" (UID: "9622b1bb-2668-4e94-a02f-e1c605d3a187"). InnerVolumeSpecName "kube-api-access-mffqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:58:02.785220 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:02.785200    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8e991-predictor-5749cbc5db-9qmzg"
Jul 21 17:58:02.878829 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:02.878780    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7a5c652-026f-4a13-9316-fde065088fb1-kube-rbac-proxy-sar-config\") pod \"d7a5c652-026f-4a13-9316-fde065088fb1\" (UID: \"d7a5c652-026f-4a13-9316-fde065088fb1\") "
Jul 21 17:58:02.879037 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:02.878864    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7a5c652-026f-4a13-9316-fde065088fb1-proxy-tls\") pod \"d7a5c652-026f-4a13-9316-fde065088fb1\" (UID: \"d7a5c652-026f-4a13-9316-fde065088fb1\") "
Jul 21 17:58:02.879037 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:02.878893    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bttxq\" (UniqueName: \"kubernetes.io/projected/d7a5c652-026f-4a13-9316-fde065088fb1-kube-api-access-bttxq\") pod \"d7a5c652-026f-4a13-9316-fde065088fb1\" (UID: \"d7a5c652-026f-4a13-9316-fde065088fb1\") "
Jul 21 17:58:02.879200 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:02.879177    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7a5c652-026f-4a13-9316-fde065088fb1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d7a5c652-026f-4a13-9316-fde065088fb1" (UID: "d7a5c652-026f-4a13-9316-fde065088fb1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:58:02.879269 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:02.879212    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9622b1bb-2668-4e94-a02f-e1c605d3a187-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:58:02.879269 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:02.879239    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9622b1bb-2668-4e94-a02f-e1c605d3a187-proxy-tls\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:58:02.879269 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:02.879255    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mffqv\" (UniqueName: \"kubernetes.io/projected/9622b1bb-2668-4e94-a02f-e1c605d3a187-kube-api-access-mffqv\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:58:02.881044 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:02.881025    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7a5c652-026f-4a13-9316-fde065088fb1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d7a5c652-026f-4a13-9316-fde065088fb1" (UID: "d7a5c652-026f-4a13-9316-fde065088fb1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:58:02.881044 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:02.881036    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7a5c652-026f-4a13-9316-fde065088fb1-kube-api-access-bttxq" (OuterVolumeSpecName: "kube-api-access-bttxq") pod "d7a5c652-026f-4a13-9316-fde065088fb1" (UID: "d7a5c652-026f-4a13-9316-fde065088fb1"). InnerVolumeSpecName "kube-api-access-bttxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:58:02.980682 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:02.980589    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7a5c652-026f-4a13-9316-fde065088fb1-proxy-tls\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:58:02.980682 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:02.980629    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bttxq\" (UniqueName: \"kubernetes.io/projected/d7a5c652-026f-4a13-9316-fde065088fb1-kube-api-access-bttxq\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:58:02.980682 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:02.980639    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7a5c652-026f-4a13-9316-fde065088fb1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:58:03.296165 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:03.296060    2563 generic.go:356] "Generic (PLEG): container finished" podID="9622b1bb-2668-4e94-a02f-e1c605d3a187" containerID="5ae44d193f106e37bd84bdc83aa606d5bd37cf6176e2e4207837cb2e8dc4c785" exitCode=0
Jul 21 17:58:03.296165 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:03.296110    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8e991-predictor-799cbc5d5f-vzztz"
Jul 21 17:58:03.296165 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:03.296157    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8e991-predictor-799cbc5d5f-vzztz" event={"ID":"9622b1bb-2668-4e94-a02f-e1c605d3a187","Type":"ContainerDied","Data":"5ae44d193f106e37bd84bdc83aa606d5bd37cf6176e2e4207837cb2e8dc4c785"}
Jul 21 17:58:03.296697 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:03.296211    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8e991-predictor-799cbc5d5f-vzztz" event={"ID":"9622b1bb-2668-4e94-a02f-e1c605d3a187","Type":"ContainerDied","Data":"18388bed19944a52da90009308bef174047b30f2114cef2ad9b1ff819c19ed82"}
Jul 21 17:58:03.296697 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:03.296237    2563 scope.go:117] "RemoveContainer" containerID="570ae0d7b305cccc4a671d55016c2e47861f3fd2beb36a1c8533a3e4488ea419"
Jul 21 17:58:03.297862 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:03.297829    2563 generic.go:356] "Generic (PLEG): container finished" podID="d7a5c652-026f-4a13-9316-fde065088fb1" containerID="f3ef3f03e0e4116bc607f81f35f16c01436c2e622abc3c3f53ce7d79378f482d" exitCode=0
Jul 21 17:58:03.297988 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:03.297878    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8e991-predictor-5749cbc5db-9qmzg" event={"ID":"d7a5c652-026f-4a13-9316-fde065088fb1","Type":"ContainerDied","Data":"f3ef3f03e0e4116bc607f81f35f16c01436c2e622abc3c3f53ce7d79378f482d"}
Jul 21 17:58:03.297988 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:03.297920    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8e991-predictor-5749cbc5db-9qmzg"
Jul 21 17:58:03.297988 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:03.297929    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8e991-predictor-5749cbc5db-9qmzg" event={"ID":"d7a5c652-026f-4a13-9316-fde065088fb1","Type":"ContainerDied","Data":"00c2ce538b2a25e176c4f6afed49308b456c84443af0db95f80e36887306d049"}
Jul 21 17:58:03.306057 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:03.306037    2563 scope.go:117] "RemoveContainer" containerID="5ae44d193f106e37bd84bdc83aa606d5bd37cf6176e2e4207837cb2e8dc4c785"
Jul 21 17:58:03.313828 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:03.313806    2563 scope.go:117] "RemoveContainer" containerID="570ae0d7b305cccc4a671d55016c2e47861f3fd2beb36a1c8533a3e4488ea419"
Jul 21 17:58:03.314096 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:58:03.314079    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"570ae0d7b305cccc4a671d55016c2e47861f3fd2beb36a1c8533a3e4488ea419\": container with ID starting with 570ae0d7b305cccc4a671d55016c2e47861f3fd2beb36a1c8533a3e4488ea419 not found: ID does not exist" containerID="570ae0d7b305cccc4a671d55016c2e47861f3fd2beb36a1c8533a3e4488ea419"
Jul 21 17:58:03.314174 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:03.314104    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"570ae0d7b305cccc4a671d55016c2e47861f3fd2beb36a1c8533a3e4488ea419"} err="failed to get container status \"570ae0d7b305cccc4a671d55016c2e47861f3fd2beb36a1c8533a3e4488ea419\": rpc error: code = NotFound desc = could not find container \"570ae0d7b305cccc4a671d55016c2e47861f3fd2beb36a1c8533a3e4488ea419\": container with ID starting with 570ae0d7b305cccc4a671d55016c2e47861f3fd2beb36a1c8533a3e4488ea419 not found: ID does not exist"
Jul 21 17:58:03.314174 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:03.314121    2563 scope.go:117] "RemoveContainer" containerID="5ae44d193f106e37bd84bdc83aa606d5bd37cf6176e2e4207837cb2e8dc4c785"
Jul 21 17:58:03.314396 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:58:03.314376    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ae44d193f106e37bd84bdc83aa606d5bd37cf6176e2e4207837cb2e8dc4c785\": container with ID starting with 5ae44d193f106e37bd84bdc83aa606d5bd37cf6176e2e4207837cb2e8dc4c785 not found: ID does not exist" containerID="5ae44d193f106e37bd84bdc83aa606d5bd37cf6176e2e4207837cb2e8dc4c785"
Jul 21 17:58:03.314469 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:03.314400    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ae44d193f106e37bd84bdc83aa606d5bd37cf6176e2e4207837cb2e8dc4c785"} err="failed to get container status \"5ae44d193f106e37bd84bdc83aa606d5bd37cf6176e2e4207837cb2e8dc4c785\": rpc error: code = NotFound desc = could not find container \"5ae44d193f106e37bd84bdc83aa606d5bd37cf6176e2e4207837cb2e8dc4c785\": container with ID starting with 5ae44d193f106e37bd84bdc83aa606d5bd37cf6176e2e4207837cb2e8dc4c785 not found: ID does not exist"
Jul 21 17:58:03.314469 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:03.314416    2563 scope.go:117] "RemoveContainer" containerID="a29f5b5e0eaef97591f29cc4da7163683d3dfe71b648db64d5cb4cd46d386445"
Jul 21 17:58:03.318090 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:03.318069    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8e991-predictor-799cbc5d5f-vzztz"]
Jul 21 17:58:03.324902 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:03.324881    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8e991-predictor-799cbc5d5f-vzztz"]
Jul 21 17:58:03.325306 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:03.325292    2563 scope.go:117] "RemoveContainer" containerID="f3ef3f03e0e4116bc607f81f35f16c01436c2e622abc3c3f53ce7d79378f482d"
Jul 21 17:58:03.332107 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:03.332085    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8e991-predictor-5749cbc5db-9qmzg"]
Jul 21 17:58:03.333236 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:03.333214    2563 scope.go:117] "RemoveContainer" containerID="a29f5b5e0eaef97591f29cc4da7163683d3dfe71b648db64d5cb4cd46d386445"
Jul 21 17:58:03.333534 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:58:03.333511    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a29f5b5e0eaef97591f29cc4da7163683d3dfe71b648db64d5cb4cd46d386445\": container with ID starting with a29f5b5e0eaef97591f29cc4da7163683d3dfe71b648db64d5cb4cd46d386445 not found: ID does not exist" containerID="a29f5b5e0eaef97591f29cc4da7163683d3dfe71b648db64d5cb4cd46d386445"
Jul 21 17:58:03.333609 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:03.333543    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a29f5b5e0eaef97591f29cc4da7163683d3dfe71b648db64d5cb4cd46d386445"} err="failed to get container status \"a29f5b5e0eaef97591f29cc4da7163683d3dfe71b648db64d5cb4cd46d386445\": rpc error: code = NotFound desc = could not find container \"a29f5b5e0eaef97591f29cc4da7163683d3dfe71b648db64d5cb4cd46d386445\": container with ID starting with a29f5b5e0eaef97591f29cc4da7163683d3dfe71b648db64d5cb4cd46d386445 not found: ID does not exist"
Jul 21 17:58:03.333609 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:03.333560    2563 scope.go:117] "RemoveContainer" containerID="f3ef3f03e0e4116bc607f81f35f16c01436c2e622abc3c3f53ce7d79378f482d"
Jul 21 17:58:03.333828 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:58:03.333806    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3ef3f03e0e4116bc607f81f35f16c01436c2e622abc3c3f53ce7d79378f482d\": container with ID starting with f3ef3f03e0e4116bc607f81f35f16c01436c2e622abc3c3f53ce7d79378f482d not found: ID does not exist" containerID="f3ef3f03e0e4116bc607f81f35f16c01436c2e622abc3c3f53ce7d79378f482d"
Jul 21 17:58:03.333867 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:03.333838    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3ef3f03e0e4116bc607f81f35f16c01436c2e622abc3c3f53ce7d79378f482d"} err="failed to get container status \"f3ef3f03e0e4116bc607f81f35f16c01436c2e622abc3c3f53ce7d79378f482d\": rpc error: code = NotFound desc = could not find container \"f3ef3f03e0e4116bc607f81f35f16c01436c2e622abc3c3f53ce7d79378f482d\": container with ID starting with f3ef3f03e0e4116bc607f81f35f16c01436c2e622abc3c3f53ce7d79378f482d not found: ID does not exist"
Jul 21 17:58:03.334777 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:03.334761    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8e991-predictor-5749cbc5db-9qmzg"]
Jul 21 17:58:04.763441 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:04.763411    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9622b1bb-2668-4e94-a02f-e1c605d3a187" path="/var/lib/kubelet/pods/9622b1bb-2668-4e94-a02f-e1c605d3a187/volumes"
Jul 21 17:58:04.763848 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:04.763834    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7a5c652-026f-4a13-9316-fde065088fb1" path="/var/lib/kubelet/pods/d7a5c652-026f-4a13-9316-fde065088fb1/volumes"
Jul 21 17:58:06.198738 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:06.198703    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p" podUID="e7f959bd-b166-4bf3-9d39-061839b2f5fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 21 17:58:06.290665 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:06.290635    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h"
Jul 21 17:58:06.291276 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:06.291245    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h" podUID="724ce6ed-3557-4232-9b1c-1035c5c14be7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 21 17:58:07.202865 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:07.202827    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-edfd2-predictor-84887cddcb-x27z7" podUID="06da2dde-96ec-4bbe-9eac-69de21849e6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 21 17:58:07.294749 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:07.294717    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9"
Jul 21 17:58:07.295324 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:07.295294    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9" podUID="59d6e22f-3560-457e-8d5e-95cd24157bb9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 21 17:58:16.198341 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:16.198300    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p" podUID="e7f959bd-b166-4bf3-9d39-061839b2f5fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 21 17:58:16.291858 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:16.291821    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h" podUID="724ce6ed-3557-4232-9b1c-1035c5c14be7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 21 17:58:17.202938 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:17.202898    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-edfd2-predictor-84887cddcb-x27z7" podUID="06da2dde-96ec-4bbe-9eac-69de21849e6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 21 17:58:17.295723 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:17.295680    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9" podUID="59d6e22f-3560-457e-8d5e-95cd24157bb9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 21 17:58:26.198740 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:26.198708    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p"
Jul 21 17:58:26.291688 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:26.291647    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h" podUID="724ce6ed-3557-4232-9b1c-1035c5c14be7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 21 17:58:27.204088 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:27.204060    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-edfd2-predictor-84887cddcb-x27z7"
Jul 21 17:58:27.295518 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:27.295478    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9" podUID="59d6e22f-3560-457e-8d5e-95cd24157bb9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 21 17:58:36.292124 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:36.292082    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h" podUID="724ce6ed-3557-4232-9b1c-1035c5c14be7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 21 17:58:37.295255 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:37.295219    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9" podUID="59d6e22f-3560-457e-8d5e-95cd24157bb9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 21 17:58:46.292009 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:46.291976    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h"
Jul 21 17:58:47.296491 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:47.296463    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9"
Jul 21 17:58:49.087277 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.087199    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p"]
Jul 21 17:58:49.087661 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.087513    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p" podUID="e7f959bd-b166-4bf3-9d39-061839b2f5fa" containerName="kserve-container" containerID="cri-o://cc79644cc1ee7a1400ae29b8c07ec28affa8b322dcdfcb5749c1d3dbab31f52e" gracePeriod=30
Jul 21 17:58:49.087661 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.087532    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p" podUID="e7f959bd-b166-4bf3-9d39-061839b2f5fa" containerName="kube-rbac-proxy" containerID="cri-o://e7ec9e69edd1b1897e3a94f42ce85b04ad763b7c548db8d25a00200cf3aa838f" gracePeriod=30
Jul 21 17:58:49.140476 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.140437    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-edfd2-predictor-84887cddcb-x27z7"]
Jul 21 17:58:49.140763 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.140715    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-edfd2-predictor-84887cddcb-x27z7" podUID="06da2dde-96ec-4bbe-9eac-69de21849e6a" containerName="kserve-container" containerID="cri-o://47ea08b9889600f8af6095ea291e8448db2d004c230e371703f672767b5bc002" gracePeriod=30
Jul 21 17:58:49.140842 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.140744    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-edfd2-predictor-84887cddcb-x27z7" podUID="06da2dde-96ec-4bbe-9eac-69de21849e6a" containerName="kube-rbac-proxy" containerID="cri-o://5877f5a317d6daa3fecc11f9430c7fe212a8614dbfb359e31a1250bf93d4a484" gracePeriod=30
Jul 21 17:58:49.155875 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.155851    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx"]
Jul 21 17:58:49.156193 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.156180    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7a5c652-026f-4a13-9316-fde065088fb1" containerName="kube-rbac-proxy"
Jul 21 17:58:49.156245 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.156195    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7a5c652-026f-4a13-9316-fde065088fb1" containerName="kube-rbac-proxy"
Jul 21 17:58:49.156245 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.156211    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7a5c652-026f-4a13-9316-fde065088fb1" containerName="kserve-container"
Jul 21 17:58:49.156245 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.156217    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7a5c652-026f-4a13-9316-fde065088fb1" containerName="kserve-container"
Jul 21 17:58:49.156245 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.156228    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9622b1bb-2668-4e94-a02f-e1c605d3a187" containerName="kserve-container"
Jul 21 17:58:49.156245 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.156234    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9622b1bb-2668-4e94-a02f-e1c605d3a187" containerName="kserve-container"
Jul 21 17:58:49.156245 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.156242    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9622b1bb-2668-4e94-a02f-e1c605d3a187" containerName="kube-rbac-proxy"
Jul 21 17:58:49.156421 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.156248    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9622b1bb-2668-4e94-a02f-e1c605d3a187" containerName="kube-rbac-proxy"
Jul 21 17:58:49.156421 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.156298    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7a5c652-026f-4a13-9316-fde065088fb1" containerName="kube-rbac-proxy"
Jul 21 17:58:49.156421 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.156305    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7a5c652-026f-4a13-9316-fde065088fb1" containerName="kserve-container"
Jul 21 17:58:49.156421 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.156312    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9622b1bb-2668-4e94-a02f-e1c605d3a187" containerName="kube-rbac-proxy"
Jul 21 17:58:49.156421 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.156319    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9622b1bb-2668-4e94-a02f-e1c605d3a187" containerName="kserve-container"
Jul 21 17:58:49.159593 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.159576    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx"
Jul 21 17:58:49.162226 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.162197    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-afaec-predictor-serving-cert\""
Jul 21 17:58:49.162226 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.162216    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-afaec-kube-rbac-proxy-sar-config\""
Jul 21 17:58:49.168934 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.168912    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx"]
Jul 21 17:58:49.232149 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.232102    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1827ba9b-0f13-4f48-bf1e-66cc74ba519a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx\" (UID: \"1827ba9b-0f13-4f48-bf1e-66cc74ba519a\") " pod="kserve-ci-e2e-test/success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx"
Jul 21 17:58:49.232295 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.232193    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw27f\" (UniqueName: \"kubernetes.io/projected/1827ba9b-0f13-4f48-bf1e-66cc74ba519a-kube-api-access-kw27f\") pod \"success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx\" (UID: \"1827ba9b-0f13-4f48-bf1e-66cc74ba519a\") " pod="kserve-ci-e2e-test/success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx"
Jul 21 17:58:49.232295 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.232226    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1827ba9b-0f13-4f48-bf1e-66cc74ba519a-proxy-tls\") pod \"success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx\" (UID: \"1827ba9b-0f13-4f48-bf1e-66cc74ba519a\") " pod="kserve-ci-e2e-test/success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx"
Jul 21 17:58:49.254950 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.254929    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl"]
Jul 21 17:58:49.258176 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.258157    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl"
Jul 21 17:58:49.260528 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.260510    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-afaec-predictor-serving-cert\""
Jul 21 17:58:49.260645 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.260586    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-afaec-kube-rbac-proxy-sar-config\""
Jul 21 17:58:49.266824 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.266798    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl"]
Jul 21 17:58:49.332766 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.332730    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc170a90-69da-4897-a7d7-6d176fdd36ba-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl\" (UID: \"fc170a90-69da-4897-a7d7-6d176fdd36ba\") " pod="kserve-ci-e2e-test/error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl"
Jul 21 17:58:49.332929 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.332784    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc170a90-69da-4897-a7d7-6d176fdd36ba-proxy-tls\") pod \"error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl\" (UID: \"fc170a90-69da-4897-a7d7-6d176fdd36ba\") " pod="kserve-ci-e2e-test/error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl"
Jul 21 17:58:49.332929 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.332863    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1827ba9b-0f13-4f48-bf1e-66cc74ba519a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx\" (UID: \"1827ba9b-0f13-4f48-bf1e-66cc74ba519a\") " pod="kserve-ci-e2e-test/success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx"
Jul 21 17:58:49.332929 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.332912    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw27f\" (UniqueName: \"kubernetes.io/projected/1827ba9b-0f13-4f48-bf1e-66cc74ba519a-kube-api-access-kw27f\") pod \"success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx\" (UID: \"1827ba9b-0f13-4f48-bf1e-66cc74ba519a\") " pod="kserve-ci-e2e-test/success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx"
Jul 21 17:58:49.333105 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.332937    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1827ba9b-0f13-4f48-bf1e-66cc74ba519a-proxy-tls\") pod \"success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx\" (UID: \"1827ba9b-0f13-4f48-bf1e-66cc74ba519a\") " pod="kserve-ci-e2e-test/success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx"
Jul 21 17:58:49.333105 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.332957    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcsk2\" (UniqueName: \"kubernetes.io/projected/fc170a90-69da-4897-a7d7-6d176fdd36ba-kube-api-access-zcsk2\") pod \"error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl\" (UID: \"fc170a90-69da-4897-a7d7-6d176fdd36ba\") " pod="kserve-ci-e2e-test/error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl"
Jul 21 17:58:49.333105 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:58:49.333064    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-afaec-predictor-serving-cert: secret "success-200-isvc-afaec-predictor-serving-cert" not found
Jul 21 17:58:49.333310 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:58:49.333115    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1827ba9b-0f13-4f48-bf1e-66cc74ba519a-proxy-tls podName:1827ba9b-0f13-4f48-bf1e-66cc74ba519a nodeName:}" failed. No retries permitted until 2026-07-21 17:58:49.833099556 +0000 UTC m=+1957.595239138 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1827ba9b-0f13-4f48-bf1e-66cc74ba519a-proxy-tls") pod "success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx" (UID: "1827ba9b-0f13-4f48-bf1e-66cc74ba519a") : secret "success-200-isvc-afaec-predictor-serving-cert" not found
Jul 21 17:58:49.333594 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.333570    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1827ba9b-0f13-4f48-bf1e-66cc74ba519a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx\" (UID: \"1827ba9b-0f13-4f48-bf1e-66cc74ba519a\") " pod="kserve-ci-e2e-test/success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx"
Jul 21 17:58:49.343831 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.343788    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw27f\" (UniqueName: \"kubernetes.io/projected/1827ba9b-0f13-4f48-bf1e-66cc74ba519a-kube-api-access-kw27f\") pod \"success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx\" (UID: \"1827ba9b-0f13-4f48-bf1e-66cc74ba519a\") " pod="kserve-ci-e2e-test/success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx"
Jul 21 17:58:49.434066 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.434032    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcsk2\" (UniqueName: \"kubernetes.io/projected/fc170a90-69da-4897-a7d7-6d176fdd36ba-kube-api-access-zcsk2\") pod \"error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl\" (UID: \"fc170a90-69da-4897-a7d7-6d176fdd36ba\") " pod="kserve-ci-e2e-test/error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl"
Jul 21 17:58:49.434227 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.434076    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc170a90-69da-4897-a7d7-6d176fdd36ba-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl\" (UID: \"fc170a90-69da-4897-a7d7-6d176fdd36ba\") " pod="kserve-ci-e2e-test/error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl"
Jul 21 17:58:49.434227 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.434108    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc170a90-69da-4897-a7d7-6d176fdd36ba-proxy-tls\") pod \"error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl\" (UID: \"fc170a90-69da-4897-a7d7-6d176fdd36ba\") " pod="kserve-ci-e2e-test/error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl"
Jul 21 17:58:49.434337 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:58:49.434254    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-afaec-predictor-serving-cert: secret "error-404-isvc-afaec-predictor-serving-cert" not found
Jul 21 17:58:49.434337 ip-10-0-132-249 kubenswrapper[2563]: E0721 17:58:49.434307    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc170a90-69da-4897-a7d7-6d176fdd36ba-proxy-tls podName:fc170a90-69da-4897-a7d7-6d176fdd36ba nodeName:}" failed. No retries permitted until 2026-07-21 17:58:49.93429457 +0000 UTC m=+1957.696434152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fc170a90-69da-4897-a7d7-6d176fdd36ba-proxy-tls") pod "error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl" (UID: "fc170a90-69da-4897-a7d7-6d176fdd36ba") : secret "error-404-isvc-afaec-predictor-serving-cert" not found
Jul 21 17:58:49.434708 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.434688    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc170a90-69da-4897-a7d7-6d176fdd36ba-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl\" (UID: \"fc170a90-69da-4897-a7d7-6d176fdd36ba\") " pod="kserve-ci-e2e-test/error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl"
Jul 21 17:58:49.443474 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.443452    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcsk2\" (UniqueName: \"kubernetes.io/projected/fc170a90-69da-4897-a7d7-6d176fdd36ba-kube-api-access-zcsk2\") pod \"error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl\" (UID: \"fc170a90-69da-4897-a7d7-6d176fdd36ba\") " pod="kserve-ci-e2e-test/error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl"
Jul 21 17:58:49.479062 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.479035    2563 generic.go:356] "Generic (PLEG): container finished" podID="06da2dde-96ec-4bbe-9eac-69de21849e6a" containerID="5877f5a317d6daa3fecc11f9430c7fe212a8614dbfb359e31a1250bf93d4a484" exitCode=2
Jul 21 17:58:49.479226 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.479111    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-edfd2-predictor-84887cddcb-x27z7" event={"ID":"06da2dde-96ec-4bbe-9eac-69de21849e6a","Type":"ContainerDied","Data":"5877f5a317d6daa3fecc11f9430c7fe212a8614dbfb359e31a1250bf93d4a484"}
Jul 21 17:58:49.480847 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.480824    2563 generic.go:356] "Generic (PLEG): container finished" podID="e7f959bd-b166-4bf3-9d39-061839b2f5fa" containerID="e7ec9e69edd1b1897e3a94f42ce85b04ad763b7c548db8d25a00200cf3aa838f" exitCode=2
Jul 21 17:58:49.480946 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.480858    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p" event={"ID":"e7f959bd-b166-4bf3-9d39-061839b2f5fa","Type":"ContainerDied","Data":"e7ec9e69edd1b1897e3a94f42ce85b04ad763b7c548db8d25a00200cf3aa838f"}
Jul 21 17:58:49.837032 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.836997    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1827ba9b-0f13-4f48-bf1e-66cc74ba519a-proxy-tls\") pod \"success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx\" (UID: \"1827ba9b-0f13-4f48-bf1e-66cc74ba519a\") " pod="kserve-ci-e2e-test/success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx"
Jul 21 17:58:49.839613 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.839594    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1827ba9b-0f13-4f48-bf1e-66cc74ba519a-proxy-tls\") pod \"success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx\" (UID: \"1827ba9b-0f13-4f48-bf1e-66cc74ba519a\") " pod="kserve-ci-e2e-test/success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx"
Jul 21 17:58:49.937976 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.937940    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc170a90-69da-4897-a7d7-6d176fdd36ba-proxy-tls\") pod \"error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl\" (UID: \"fc170a90-69da-4897-a7d7-6d176fdd36ba\") " pod="kserve-ci-e2e-test/error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl"
Jul 21 17:58:49.940449 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:49.940430    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc170a90-69da-4897-a7d7-6d176fdd36ba-proxy-tls\") pod \"error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl\" (UID: \"fc170a90-69da-4897-a7d7-6d176fdd36ba\") " pod="kserve-ci-e2e-test/error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl"
Jul 21 17:58:50.070023 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:50.069984    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx"
Jul 21 17:58:50.169498 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:50.169464    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl"
Jul 21 17:58:50.203457 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:50.203415    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx"]
Jul 21 17:58:50.206686 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:58:50.206658    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1827ba9b_0f13_4f48_bf1e_66cc74ba519a.slice/crio-047fbdc99155eed5ae51d9130094471653bd0a4ae115083c4f3f1a63e5d2d908 WatchSource:0}: Error finding container 047fbdc99155eed5ae51d9130094471653bd0a4ae115083c4f3f1a63e5d2d908: Status 404 returned error can't find the container with id 047fbdc99155eed5ae51d9130094471653bd0a4ae115083c4f3f1a63e5d2d908
Jul 21 17:58:50.316342 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:50.316321    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl"]
Jul 21 17:58:50.319171 ip-10-0-132-249 kubenswrapper[2563]: W0721 17:58:50.319143    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc170a90_69da_4897_a7d7_6d176fdd36ba.slice/crio-b3ccd6514d32e0a32f2fb4c9f786fe84f4828e1246fab328f7d122abcd6bb5c5 WatchSource:0}: Error finding container b3ccd6514d32e0a32f2fb4c9f786fe84f4828e1246fab328f7d122abcd6bb5c5: Status 404 returned error can't find the container with id b3ccd6514d32e0a32f2fb4c9f786fe84f4828e1246fab328f7d122abcd6bb5c5
Jul 21 17:58:50.486499 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:50.486461    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx" event={"ID":"1827ba9b-0f13-4f48-bf1e-66cc74ba519a","Type":"ContainerStarted","Data":"bde4e6700d91984d2d6b79f9c5603229b20052103ad750c4f3e0aa8de5109ea3"}
Jul 21 17:58:50.486705 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:50.486526    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx" event={"ID":"1827ba9b-0f13-4f48-bf1e-66cc74ba519a","Type":"ContainerStarted","Data":"609eb75d94d1f3d0447fa1b12256a1b257857f17e11daf5b759f728392c10c5f"}
Jul 21 17:58:50.486705 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:50.486556    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx" event={"ID":"1827ba9b-0f13-4f48-bf1e-66cc74ba519a","Type":"ContainerStarted","Data":"047fbdc99155eed5ae51d9130094471653bd0a4ae115083c4f3f1a63e5d2d908"}
Jul 21 17:58:50.486705 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:50.486590    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx"
Jul 21 17:58:50.488351 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:50.488326    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl" event={"ID":"fc170a90-69da-4897-a7d7-6d176fdd36ba","Type":"ContainerStarted","Data":"2d59edf583717e2fcf828323f171ae970a7d5a642d594f20307b3e8b65b34c5d"}
Jul 21 17:58:50.488457 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:50.488376    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl" event={"ID":"fc170a90-69da-4897-a7d7-6d176fdd36ba","Type":"ContainerStarted","Data":"e6b9be82b80c017bad12de31337cc27085a8ddf4d9db07fe89216d84653b9035"}
Jul 21 17:58:50.488457 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:50.488400    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl"
Jul 21 17:58:50.488457 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:50.488427    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl" event={"ID":"fc170a90-69da-4897-a7d7-6d176fdd36ba","Type":"ContainerStarted","Data":"b3ccd6514d32e0a32f2fb4c9f786fe84f4828e1246fab328f7d122abcd6bb5c5"}
Jul 21 17:58:50.488457 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:50.488451    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl"
Jul 21 17:58:50.489874 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:50.489850    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl" podUID="fc170a90-69da-4897-a7d7-6d176fdd36ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 21 17:58:50.503726 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:50.503684    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx" podStartSLOduration=1.5036722089999999 podStartE2EDuration="1.503672209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:58:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:58:50.502853631 +0000 UTC m=+1958.264993237" watchObservedRunningTime="2026-07-21 17:58:50.503672209 +0000 UTC m=+1958.265811813"
Jul 21 17:58:50.520741 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:50.520701    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl" podStartSLOduration=1.520677039 podStartE2EDuration="1.520677039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:58:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:58:50.519096651 +0000 UTC m=+1958.281236254" watchObservedRunningTime="2026-07-21 17:58:50.520677039 +0000 UTC m=+1958.282816643"
Jul 21 17:58:51.191667 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:51.191631    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p" podUID="e7f959bd-b166-4bf3-9d39-061839b2f5fa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Jul 21 17:58:51.497996 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:51.495834    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx"
Jul 21 17:58:51.498183 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:51.497994    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl" podUID="fc170a90-69da-4897-a7d7-6d176fdd36ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 21 17:58:51.498183 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:51.497996    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx" podUID="1827ba9b-0f13-4f48-bf1e-66cc74ba519a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 21 17:58:52.199065 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:52.199022    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-edfd2-predictor-84887cddcb-x27z7" podUID="06da2dde-96ec-4bbe-9eac-69de21849e6a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Jul 21 17:58:52.498930 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:52.498897    2563 generic.go:356] "Generic (PLEG): container finished" podID="06da2dde-96ec-4bbe-9eac-69de21849e6a" containerID="47ea08b9889600f8af6095ea291e8448db2d004c230e371703f672767b5bc002" exitCode=0
Jul 21 17:58:52.499086 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:52.498939    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-edfd2-predictor-84887cddcb-x27z7" event={"ID":"06da2dde-96ec-4bbe-9eac-69de21849e6a","Type":"ContainerDied","Data":"47ea08b9889600f8af6095ea291e8448db2d004c230e371703f672767b5bc002"}
Jul 21 17:58:52.500998 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:52.500968    2563 generic.go:356] "Generic (PLEG): container finished" podID="e7f959bd-b166-4bf3-9d39-061839b2f5fa" containerID="cc79644cc1ee7a1400ae29b8c07ec28affa8b322dcdfcb5749c1d3dbab31f52e" exitCode=0
Jul 21 17:58:52.501162 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:52.501034    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p" event={"ID":"e7f959bd-b166-4bf3-9d39-061839b2f5fa","Type":"ContainerDied","Data":"cc79644cc1ee7a1400ae29b8c07ec28affa8b322dcdfcb5749c1d3dbab31f52e"}
Jul 21 17:58:52.501609 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:52.501587    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx" podUID="1827ba9b-0f13-4f48-bf1e-66cc74ba519a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 21 17:58:52.603004 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:52.602980    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p"
Jul 21 17:58:52.605622 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:52.605604    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-edfd2-predictor-84887cddcb-x27z7"
Jul 21 17:58:52.663803 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:52.663771    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwn72\" (UniqueName: \"kubernetes.io/projected/e7f959bd-b166-4bf3-9d39-061839b2f5fa-kube-api-access-kwn72\") pod \"e7f959bd-b166-4bf3-9d39-061839b2f5fa\" (UID: \"e7f959bd-b166-4bf3-9d39-061839b2f5fa\") "
Jul 21 17:58:52.663931 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:52.663855    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7f959bd-b166-4bf3-9d39-061839b2f5fa-kube-rbac-proxy-sar-config\") pod \"e7f959bd-b166-4bf3-9d39-061839b2f5fa\" (UID: \"e7f959bd-b166-4bf3-9d39-061839b2f5fa\") "
Jul 21 17:58:52.663931 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:52.663894    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06da2dde-96ec-4bbe-9eac-69de21849e6a-kube-rbac-proxy-sar-config\") pod \"06da2dde-96ec-4bbe-9eac-69de21849e6a\" (UID: \"06da2dde-96ec-4bbe-9eac-69de21849e6a\") "
Jul 21 17:58:52.664041 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:52.663939    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7f959bd-b166-4bf3-9d39-061839b2f5fa-proxy-tls\") pod \"e7f959bd-b166-4bf3-9d39-061839b2f5fa\" (UID: \"e7f959bd-b166-4bf3-9d39-061839b2f5fa\") "
Jul 21 17:58:52.664041 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:52.663971    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06da2dde-96ec-4bbe-9eac-69de21849e6a-proxy-tls\") pod \"06da2dde-96ec-4bbe-9eac-69de21849e6a\" (UID: \"06da2dde-96ec-4bbe-9eac-69de21849e6a\") "
Jul 21 17:58:52.664041 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:52.664017    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mkdz\" (UniqueName: \"kubernetes.io/projected/06da2dde-96ec-4bbe-9eac-69de21849e6a-kube-api-access-7mkdz\") pod \"06da2dde-96ec-4bbe-9eac-69de21849e6a\" (UID: \"06da2dde-96ec-4bbe-9eac-69de21849e6a\") "
Jul 21 17:58:52.664319 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:52.664288    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7f959bd-b166-4bf3-9d39-061839b2f5fa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e7f959bd-b166-4bf3-9d39-061839b2f5fa" (UID: "e7f959bd-b166-4bf3-9d39-061839b2f5fa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:58:52.664451 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:52.664351    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06da2dde-96ec-4bbe-9eac-69de21849e6a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "06da2dde-96ec-4bbe-9eac-69de21849e6a" (UID: "06da2dde-96ec-4bbe-9eac-69de21849e6a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:58:52.666374 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:52.666338    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06da2dde-96ec-4bbe-9eac-69de21849e6a-kube-api-access-7mkdz" (OuterVolumeSpecName: "kube-api-access-7mkdz") pod "06da2dde-96ec-4bbe-9eac-69de21849e6a" (UID: "06da2dde-96ec-4bbe-9eac-69de21849e6a"). InnerVolumeSpecName "kube-api-access-7mkdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:58:52.666478 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:52.666419    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7f959bd-b166-4bf3-9d39-061839b2f5fa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e7f959bd-b166-4bf3-9d39-061839b2f5fa" (UID: "e7f959bd-b166-4bf3-9d39-061839b2f5fa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:58:52.666578 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:52.666560    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06da2dde-96ec-4bbe-9eac-69de21849e6a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "06da2dde-96ec-4bbe-9eac-69de21849e6a" (UID: "06da2dde-96ec-4bbe-9eac-69de21849e6a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:58:52.666645 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:52.666623    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f959bd-b166-4bf3-9d39-061839b2f5fa-kube-api-access-kwn72" (OuterVolumeSpecName: "kube-api-access-kwn72") pod "e7f959bd-b166-4bf3-9d39-061839b2f5fa" (UID: "e7f959bd-b166-4bf3-9d39-061839b2f5fa"). InnerVolumeSpecName "kube-api-access-kwn72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:58:52.764853 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:52.764771    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7f959bd-b166-4bf3-9d39-061839b2f5fa-proxy-tls\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:58:52.764853 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:52.764798    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06da2dde-96ec-4bbe-9eac-69de21849e6a-proxy-tls\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:58:52.764853 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:52.764809    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mkdz\" (UniqueName: \"kubernetes.io/projected/06da2dde-96ec-4bbe-9eac-69de21849e6a-kube-api-access-7mkdz\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:58:52.764853 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:52.764821    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwn72\" (UniqueName: \"kubernetes.io/projected/e7f959bd-b166-4bf3-9d39-061839b2f5fa-kube-api-access-kwn72\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:58:52.764853 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:52.764833    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7f959bd-b166-4bf3-9d39-061839b2f5fa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:58:52.764853 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:52.764846    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06da2dde-96ec-4bbe-9eac-69de21849e6a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 17:58:53.506086 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:53.506050    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-edfd2-predictor-84887cddcb-x27z7" event={"ID":"06da2dde-96ec-4bbe-9eac-69de21849e6a","Type":"ContainerDied","Data":"7015f0e2671ae41290a1a1a0dc460b9d5108d78c5702d914e78196aef99a98c6"}
Jul 21 17:58:53.506518 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:53.506067    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-edfd2-predictor-84887cddcb-x27z7"
Jul 21 17:58:53.506518 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:53.506150    2563 scope.go:117] "RemoveContainer" containerID="5877f5a317d6daa3fecc11f9430c7fe212a8614dbfb359e31a1250bf93d4a484"
Jul 21 17:58:53.507839 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:53.507819    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p" event={"ID":"e7f959bd-b166-4bf3-9d39-061839b2f5fa","Type":"ContainerDied","Data":"dfa4d98eeed7079d9596bd0aec9bfd071e324a04d21d0eca884a1763c6eb128e"}
Jul 21 17:58:53.507955 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:53.507849    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p"
Jul 21 17:58:53.514649 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:53.514632    2563 scope.go:117] "RemoveContainer" containerID="47ea08b9889600f8af6095ea291e8448db2d004c230e371703f672767b5bc002"
Jul 21 17:58:53.521957 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:53.521943    2563 scope.go:117] "RemoveContainer" containerID="e7ec9e69edd1b1897e3a94f42ce85b04ad763b7c548db8d25a00200cf3aa838f"
Jul 21 17:58:53.523071 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:53.523052    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p"]
Jul 21 17:58:53.526208 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:53.526191    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-edfd2-predictor-6f4b9cd55c-n6g4p"]
Jul 21 17:58:53.529510 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:53.529494    2563 scope.go:117] "RemoveContainer" containerID="cc79644cc1ee7a1400ae29b8c07ec28affa8b322dcdfcb5749c1d3dbab31f52e"
Jul 21 17:58:53.535712 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:53.535691    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-edfd2-predictor-84887cddcb-x27z7"]
Jul 21 17:58:53.539615 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:53.539594    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-edfd2-predictor-84887cddcb-x27z7"]
Jul 21 17:58:54.763956 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:54.763925    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06da2dde-96ec-4bbe-9eac-69de21849e6a" path="/var/lib/kubelet/pods/06da2dde-96ec-4bbe-9eac-69de21849e6a/volumes"
Jul 21 17:58:54.764372 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:54.764357    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7f959bd-b166-4bf3-9d39-061839b2f5fa" path="/var/lib/kubelet/pods/e7f959bd-b166-4bf3-9d39-061839b2f5fa/volumes"
Jul 21 17:58:56.497287 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:56.497261    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl"
Jul 21 17:58:56.497835 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:56.497794    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl" podUID="fc170a90-69da-4897-a7d7-6d176fdd36ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 21 17:58:57.505706 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:57.505679    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx"
Jul 21 17:58:57.506299 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:58:57.506264    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx" podUID="1827ba9b-0f13-4f48-bf1e-66cc74ba519a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 21 17:59:06.497826 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:59:06.497741    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl" podUID="fc170a90-69da-4897-a7d7-6d176fdd36ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 21 17:59:07.506152 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:59:07.506093    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx" podUID="1827ba9b-0f13-4f48-bf1e-66cc74ba519a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 21 17:59:16.498578 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:59:16.498533    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl" podUID="fc170a90-69da-4897-a7d7-6d176fdd36ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 21 17:59:17.506837 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:59:17.506791    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx" podUID="1827ba9b-0f13-4f48-bf1e-66cc74ba519a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 21 17:59:26.497750 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:59:26.497710    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl" podUID="fc170a90-69da-4897-a7d7-6d176fdd36ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 21 17:59:27.506315 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:59:27.506275    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx" podUID="1827ba9b-0f13-4f48-bf1e-66cc74ba519a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 21 17:59:36.498289 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:59:36.498254    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl"
Jul 21 17:59:37.507281 ip-10-0-132-249 kubenswrapper[2563]: I0721 17:59:37.507249    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx"
Jul 21 18:08:03.940284 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:03.940198    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx"]
Jul 21 18:08:03.940734 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:03.940531    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx" podUID="1827ba9b-0f13-4f48-bf1e-66cc74ba519a" containerName="kserve-container" containerID="cri-o://609eb75d94d1f3d0447fa1b12256a1b257857f17e11daf5b759f728392c10c5f" gracePeriod=30
Jul 21 18:08:03.940734 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:03.940563    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx" podUID="1827ba9b-0f13-4f48-bf1e-66cc74ba519a" containerName="kube-rbac-proxy" containerID="cri-o://bde4e6700d91984d2d6b79f9c5603229b20052103ad750c4f3e0aa8de5109ea3" gracePeriod=30
Jul 21 18:08:03.989025 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:03.988994    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl"]
Jul 21 18:08:03.989268 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:03.989225    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl" podUID="fc170a90-69da-4897-a7d7-6d176fdd36ba" containerName="kserve-container" containerID="cri-o://e6b9be82b80c017bad12de31337cc27085a8ddf4d9db07fe89216d84653b9035" gracePeriod=30
Jul 21 18:08:03.989354 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:03.989281    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl" podUID="fc170a90-69da-4897-a7d7-6d176fdd36ba" containerName="kube-rbac-proxy" containerID="cri-o://2d59edf583717e2fcf828323f171ae970a7d5a642d594f20307b3e8b65b34c5d" gracePeriod=30
Jul 21 18:08:04.579399 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:04.579367    2563 generic.go:356] "Generic (PLEG): container finished" podID="fc170a90-69da-4897-a7d7-6d176fdd36ba" containerID="2d59edf583717e2fcf828323f171ae970a7d5a642d594f20307b3e8b65b34c5d" exitCode=2
Jul 21 18:08:04.579700 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:04.579451    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl" event={"ID":"fc170a90-69da-4897-a7d7-6d176fdd36ba","Type":"ContainerDied","Data":"2d59edf583717e2fcf828323f171ae970a7d5a642d594f20307b3e8b65b34c5d"}
Jul 21 18:08:04.581184 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:04.581162    2563 generic.go:356] "Generic (PLEG): container finished" podID="1827ba9b-0f13-4f48-bf1e-66cc74ba519a" containerID="bde4e6700d91984d2d6b79f9c5603229b20052103ad750c4f3e0aa8de5109ea3" exitCode=2
Jul 21 18:08:04.581419 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:04.581209    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx" event={"ID":"1827ba9b-0f13-4f48-bf1e-66cc74ba519a","Type":"ContainerDied","Data":"bde4e6700d91984d2d6b79f9c5603229b20052103ad750c4f3e0aa8de5109ea3"}
Jul 21 18:08:06.493117 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:06.493075    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl" podUID="fc170a90-69da-4897-a7d7-6d176fdd36ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Jul 21 18:08:06.498438 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:06.498412    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl" podUID="fc170a90-69da-4897-a7d7-6d176fdd36ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 21 18:08:07.188084 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.188059    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx"
Jul 21 18:08:07.237584 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.237562    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl"
Jul 21 18:08:07.276308 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.276280    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc170a90-69da-4897-a7d7-6d176fdd36ba-proxy-tls\") pod \"fc170a90-69da-4897-a7d7-6d176fdd36ba\" (UID: \"fc170a90-69da-4897-a7d7-6d176fdd36ba\") "
Jul 21 18:08:07.276438 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.276313    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1827ba9b-0f13-4f48-bf1e-66cc74ba519a-kube-rbac-proxy-sar-config\") pod \"1827ba9b-0f13-4f48-bf1e-66cc74ba519a\" (UID: \"1827ba9b-0f13-4f48-bf1e-66cc74ba519a\") "
Jul 21 18:08:07.276438 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.276331    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1827ba9b-0f13-4f48-bf1e-66cc74ba519a-proxy-tls\") pod \"1827ba9b-0f13-4f48-bf1e-66cc74ba519a\" (UID: \"1827ba9b-0f13-4f48-bf1e-66cc74ba519a\") "
Jul 21 18:08:07.276438 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.276370    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcsk2\" (UniqueName: \"kubernetes.io/projected/fc170a90-69da-4897-a7d7-6d176fdd36ba-kube-api-access-zcsk2\") pod \"fc170a90-69da-4897-a7d7-6d176fdd36ba\" (UID: \"fc170a90-69da-4897-a7d7-6d176fdd36ba\") "
Jul 21 18:08:07.276438 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.276401    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kw27f\" (UniqueName: \"kubernetes.io/projected/1827ba9b-0f13-4f48-bf1e-66cc74ba519a-kube-api-access-kw27f\") pod \"1827ba9b-0f13-4f48-bf1e-66cc74ba519a\" (UID: \"1827ba9b-0f13-4f48-bf1e-66cc74ba519a\") "
Jul 21 18:08:07.276617 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.276446    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc170a90-69da-4897-a7d7-6d176fdd36ba-kube-rbac-proxy-sar-config\") pod \"fc170a90-69da-4897-a7d7-6d176fdd36ba\" (UID: \"fc170a90-69da-4897-a7d7-6d176fdd36ba\") "
Jul 21 18:08:07.276717 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.276675    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1827ba9b-0f13-4f48-bf1e-66cc74ba519a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1827ba9b-0f13-4f48-bf1e-66cc74ba519a" (UID: "1827ba9b-0f13-4f48-bf1e-66cc74ba519a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:08:07.276885 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.276856    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc170a90-69da-4897-a7d7-6d176fdd36ba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fc170a90-69da-4897-a7d7-6d176fdd36ba" (UID: "fc170a90-69da-4897-a7d7-6d176fdd36ba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:08:07.278658 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.278635    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1827ba9b-0f13-4f48-bf1e-66cc74ba519a-kube-api-access-kw27f" (OuterVolumeSpecName: "kube-api-access-kw27f") pod "1827ba9b-0f13-4f48-bf1e-66cc74ba519a" (UID: "1827ba9b-0f13-4f48-bf1e-66cc74ba519a"). InnerVolumeSpecName "kube-api-access-kw27f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:08:07.278782 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.278661    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc170a90-69da-4897-a7d7-6d176fdd36ba-kube-api-access-zcsk2" (OuterVolumeSpecName: "kube-api-access-zcsk2") pod "fc170a90-69da-4897-a7d7-6d176fdd36ba" (UID: "fc170a90-69da-4897-a7d7-6d176fdd36ba"). InnerVolumeSpecName "kube-api-access-zcsk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:08:07.279021 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.279007    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc170a90-69da-4897-a7d7-6d176fdd36ba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fc170a90-69da-4897-a7d7-6d176fdd36ba" (UID: "fc170a90-69da-4897-a7d7-6d176fdd36ba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:08:07.279065 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.279016    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1827ba9b-0f13-4f48-bf1e-66cc74ba519a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1827ba9b-0f13-4f48-bf1e-66cc74ba519a" (UID: "1827ba9b-0f13-4f48-bf1e-66cc74ba519a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:08:07.377313 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.377283    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc170a90-69da-4897-a7d7-6d176fdd36ba-proxy-tls\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 18:08:07.377313 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.377309    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1827ba9b-0f13-4f48-bf1e-66cc74ba519a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 18:08:07.377313 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.377320    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1827ba9b-0f13-4f48-bf1e-66cc74ba519a-proxy-tls\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 18:08:07.377542 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.377330    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcsk2\" (UniqueName: \"kubernetes.io/projected/fc170a90-69da-4897-a7d7-6d176fdd36ba-kube-api-access-zcsk2\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 18:08:07.377542 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.377339    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw27f\" (UniqueName: \"kubernetes.io/projected/1827ba9b-0f13-4f48-bf1e-66cc74ba519a-kube-api-access-kw27f\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 18:08:07.377542 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.377348    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc170a90-69da-4897-a7d7-6d176fdd36ba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 18:08:07.593853 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.593771    2563 generic.go:356] "Generic (PLEG): container finished" podID="1827ba9b-0f13-4f48-bf1e-66cc74ba519a" containerID="609eb75d94d1f3d0447fa1b12256a1b257857f17e11daf5b759f728392c10c5f" exitCode=0
Jul 21 18:08:07.593853 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.593817    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx"
Jul 21 18:08:07.593853 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.593846    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx" event={"ID":"1827ba9b-0f13-4f48-bf1e-66cc74ba519a","Type":"ContainerDied","Data":"609eb75d94d1f3d0447fa1b12256a1b257857f17e11daf5b759f728392c10c5f"}
Jul 21 18:08:07.594439 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.593909    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx" event={"ID":"1827ba9b-0f13-4f48-bf1e-66cc74ba519a","Type":"ContainerDied","Data":"047fbdc99155eed5ae51d9130094471653bd0a4ae115083c4f3f1a63e5d2d908"}
Jul 21 18:08:07.594439 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.593951    2563 scope.go:117] "RemoveContainer" containerID="bde4e6700d91984d2d6b79f9c5603229b20052103ad750c4f3e0aa8de5109ea3"
Jul 21 18:08:07.595587 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.595514    2563 generic.go:356] "Generic (PLEG): container finished" podID="fc170a90-69da-4897-a7d7-6d176fdd36ba" containerID="e6b9be82b80c017bad12de31337cc27085a8ddf4d9db07fe89216d84653b9035" exitCode=0
Jul 21 18:08:07.595587 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.595571    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl"
Jul 21 18:08:07.595587 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.595595    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl" event={"ID":"fc170a90-69da-4897-a7d7-6d176fdd36ba","Type":"ContainerDied","Data":"e6b9be82b80c017bad12de31337cc27085a8ddf4d9db07fe89216d84653b9035"}
Jul 21 18:08:07.595894 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.595655    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl" event={"ID":"fc170a90-69da-4897-a7d7-6d176fdd36ba","Type":"ContainerDied","Data":"b3ccd6514d32e0a32f2fb4c9f786fe84f4828e1246fab328f7d122abcd6bb5c5"}
Jul 21 18:08:07.602091 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.602073    2563 scope.go:117] "RemoveContainer" containerID="609eb75d94d1f3d0447fa1b12256a1b257857f17e11daf5b759f728392c10c5f"
Jul 21 18:08:07.611746 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.611731    2563 scope.go:117] "RemoveContainer" containerID="bde4e6700d91984d2d6b79f9c5603229b20052103ad750c4f3e0aa8de5109ea3"
Jul 21 18:08:07.612011 ip-10-0-132-249 kubenswrapper[2563]: E0721 18:08:07.611993    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bde4e6700d91984d2d6b79f9c5603229b20052103ad750c4f3e0aa8de5109ea3\": container with ID starting with bde4e6700d91984d2d6b79f9c5603229b20052103ad750c4f3e0aa8de5109ea3 not found: ID does not exist" containerID="bde4e6700d91984d2d6b79f9c5603229b20052103ad750c4f3e0aa8de5109ea3"
Jul 21 18:08:07.612068 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.612017    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bde4e6700d91984d2d6b79f9c5603229b20052103ad750c4f3e0aa8de5109ea3"} err="failed to get container status \"bde4e6700d91984d2d6b79f9c5603229b20052103ad750c4f3e0aa8de5109ea3\": rpc error: code = NotFound desc = could not find container \"bde4e6700d91984d2d6b79f9c5603229b20052103ad750c4f3e0aa8de5109ea3\": container with ID starting with bde4e6700d91984d2d6b79f9c5603229b20052103ad750c4f3e0aa8de5109ea3 not found: ID does not exist"
Jul 21 18:08:07.612068 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.612032    2563 scope.go:117] "RemoveContainer" containerID="609eb75d94d1f3d0447fa1b12256a1b257857f17e11daf5b759f728392c10c5f"
Jul 21 18:08:07.612265 ip-10-0-132-249 kubenswrapper[2563]: E0721 18:08:07.612250    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"609eb75d94d1f3d0447fa1b12256a1b257857f17e11daf5b759f728392c10c5f\": container with ID starting with 609eb75d94d1f3d0447fa1b12256a1b257857f17e11daf5b759f728392c10c5f not found: ID does not exist" containerID="609eb75d94d1f3d0447fa1b12256a1b257857f17e11daf5b759f728392c10c5f"
Jul 21 18:08:07.612323 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.612270    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"609eb75d94d1f3d0447fa1b12256a1b257857f17e11daf5b759f728392c10c5f"} err="failed to get container status \"609eb75d94d1f3d0447fa1b12256a1b257857f17e11daf5b759f728392c10c5f\": rpc error: code = NotFound desc = could not find container \"609eb75d94d1f3d0447fa1b12256a1b257857f17e11daf5b759f728392c10c5f\": container with ID starting with 609eb75d94d1f3d0447fa1b12256a1b257857f17e11daf5b759f728392c10c5f not found: ID does not exist"
Jul 21 18:08:07.612323 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.612283    2563 scope.go:117] "RemoveContainer" containerID="2d59edf583717e2fcf828323f171ae970a7d5a642d594f20307b3e8b65b34c5d"
Jul 21 18:08:07.619829 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.619799    2563 scope.go:117] "RemoveContainer" containerID="e6b9be82b80c017bad12de31337cc27085a8ddf4d9db07fe89216d84653b9035"
Jul 21 18:08:07.621274 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.621257    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx"]
Jul 21 18:08:07.624615 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.624595    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-afaec-predictor-66b7f9bdff-rm6wx"]
Jul 21 18:08:07.627191 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.627176    2563 scope.go:117] "RemoveContainer" containerID="2d59edf583717e2fcf828323f171ae970a7d5a642d594f20307b3e8b65b34c5d"
Jul 21 18:08:07.627454 ip-10-0-132-249 kubenswrapper[2563]: E0721 18:08:07.627436    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d59edf583717e2fcf828323f171ae970a7d5a642d594f20307b3e8b65b34c5d\": container with ID starting with 2d59edf583717e2fcf828323f171ae970a7d5a642d594f20307b3e8b65b34c5d not found: ID does not exist" containerID="2d59edf583717e2fcf828323f171ae970a7d5a642d594f20307b3e8b65b34c5d"
Jul 21 18:08:07.627508 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.627461    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d59edf583717e2fcf828323f171ae970a7d5a642d594f20307b3e8b65b34c5d"} err="failed to get container status \"2d59edf583717e2fcf828323f171ae970a7d5a642d594f20307b3e8b65b34c5d\": rpc error: code = NotFound desc = could not find container \"2d59edf583717e2fcf828323f171ae970a7d5a642d594f20307b3e8b65b34c5d\": container with ID starting with 2d59edf583717e2fcf828323f171ae970a7d5a642d594f20307b3e8b65b34c5d not found: ID does not exist"
Jul 21 18:08:07.627508 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.627479    2563 scope.go:117] "RemoveContainer" containerID="e6b9be82b80c017bad12de31337cc27085a8ddf4d9db07fe89216d84653b9035"
Jul 21 18:08:07.627710 ip-10-0-132-249 kubenswrapper[2563]: E0721 18:08:07.627695    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6b9be82b80c017bad12de31337cc27085a8ddf4d9db07fe89216d84653b9035\": container with ID starting with e6b9be82b80c017bad12de31337cc27085a8ddf4d9db07fe89216d84653b9035 not found: ID does not exist" containerID="e6b9be82b80c017bad12de31337cc27085a8ddf4d9db07fe89216d84653b9035"
Jul 21 18:08:07.627751 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.627712    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6b9be82b80c017bad12de31337cc27085a8ddf4d9db07fe89216d84653b9035"} err="failed to get container status \"e6b9be82b80c017bad12de31337cc27085a8ddf4d9db07fe89216d84653b9035\": rpc error: code = NotFound desc = could not find container \"e6b9be82b80c017bad12de31337cc27085a8ddf4d9db07fe89216d84653b9035\": container with ID starting with e6b9be82b80c017bad12de31337cc27085a8ddf4d9db07fe89216d84653b9035 not found: ID does not exist"
Jul 21 18:08:07.635666 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.635647    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl"]
Jul 21 18:08:07.639371 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:07.639352    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-afaec-predictor-6f6ddbd654-jjvnl"]
Jul 21 18:08:08.763568 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:08.763529    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1827ba9b-0f13-4f48-bf1e-66cc74ba519a" path="/var/lib/kubelet/pods/1827ba9b-0f13-4f48-bf1e-66cc74ba519a/volumes"
Jul 21 18:08:08.764127 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:08:08.764107    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc170a90-69da-4897-a7d7-6d176fdd36ba" path="/var/lib/kubelet/pods/fc170a90-69da-4897-a7d7-6d176fdd36ba/volumes"
Jul 21 18:15:18.261828 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:18.261791    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9"]
Jul 21 18:15:18.264445 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:18.262066    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9" podUID="59d6e22f-3560-457e-8d5e-95cd24157bb9" containerName="kserve-container" containerID="cri-o://5d3dfb2b2f18368624d62e3e6149e803b17d6314e49812fe859a50940ab41b1b" gracePeriod=30
Jul 21 18:15:18.264445 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:18.262117    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9" podUID="59d6e22f-3560-457e-8d5e-95cd24157bb9" containerName="kube-rbac-proxy" containerID="cri-o://c71d51991c9b793ae57908b8c75624071e3d6b619b13c9cfb355d0296b949857" gracePeriod=30
Jul 21 18:15:18.318738 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:18.318708    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h"]
Jul 21 18:15:18.318956 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:18.318929    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h" podUID="724ce6ed-3557-4232-9b1c-1035c5c14be7" containerName="kserve-container" containerID="cri-o://d5955ecf75e490c2bad003e904c91940b98180e35e420bdc00f31d20b3175b51" gracePeriod=30
Jul 21 18:15:18.319085 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:18.318978    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h" podUID="724ce6ed-3557-4232-9b1c-1035c5c14be7" containerName="kube-rbac-proxy" containerID="cri-o://3d9c066046ea355312e57edbbafdfb7f3ad5441ca5dc563b65f8afe310ac589a" gracePeriod=30
Jul 21 18:15:19.172003 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:19.171970    2563 generic.go:356] "Generic (PLEG): container finished" podID="59d6e22f-3560-457e-8d5e-95cd24157bb9" containerID="c71d51991c9b793ae57908b8c75624071e3d6b619b13c9cfb355d0296b949857" exitCode=2
Jul 21 18:15:19.172183 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:19.172045    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9" event={"ID":"59d6e22f-3560-457e-8d5e-95cd24157bb9","Type":"ContainerDied","Data":"c71d51991c9b793ae57908b8c75624071e3d6b619b13c9cfb355d0296b949857"}
Jul 21 18:15:19.173556 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:19.173536    2563 generic.go:356] "Generic (PLEG): container finished" podID="724ce6ed-3557-4232-9b1c-1035c5c14be7" containerID="3d9c066046ea355312e57edbbafdfb7f3ad5441ca5dc563b65f8afe310ac589a" exitCode=2
Jul 21 18:15:19.173664 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:19.173604    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h" event={"ID":"724ce6ed-3557-4232-9b1c-1035c5c14be7","Type":"ContainerDied","Data":"3d9c066046ea355312e57edbbafdfb7f3ad5441ca5dc563b65f8afe310ac589a"}
Jul 21 18:15:19.194275 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:19.194247    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hrpnq/must-gather-6hxmb"]
Jul 21 18:15:19.194534 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:19.194522    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06da2dde-96ec-4bbe-9eac-69de21849e6a" containerName="kserve-container"
Jul 21 18:15:19.194583 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:19.194535    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="06da2dde-96ec-4bbe-9eac-69de21849e6a" containerName="kserve-container"
Jul 21 18:15:19.194583 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:19.194547    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06da2dde-96ec-4bbe-9eac-69de21849e6a" containerName="kube-rbac-proxy"
Jul 21 18:15:19.194583 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:19.194552    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="06da2dde-96ec-4bbe-9eac-69de21849e6a" containerName="kube-rbac-proxy"
Jul 21 18:15:19.194583 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:19.194559    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc170a90-69da-4897-a7d7-6d176fdd36ba" containerName="kube-rbac-proxy"
Jul 21 18:15:19.194583 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:19.194565    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc170a90-69da-4897-a7d7-6d176fdd36ba" containerName="kube-rbac-proxy"
Jul 21 18:15:19.194583 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:19.194575    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7f959bd-b166-4bf3-9d39-061839b2f5fa" containerName="kube-rbac-proxy"
Jul 21 18:15:19.194583 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:19.194580    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7f959bd-b166-4bf3-9d39-061839b2f5fa" containerName="kube-rbac-proxy"
Jul 21 18:15:19.194797 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:19.194589    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1827ba9b-0f13-4f48-bf1e-66cc74ba519a" containerName="kserve-container"
Jul 21 18:15:19.194797 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:19.194595    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1827ba9b-0f13-4f48-bf1e-66cc74ba519a" containerName="kserve-container"
Jul 21 18:15:19.194797 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:19.194603    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc170a90-69da-4897-a7d7-6d176fdd36ba" containerName="kserve-container"
Jul 21 18:15:19.194797 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:19.194608    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc170a90-69da-4897-a7d7-6d176fdd36ba" containerName="kserve-container"
Jul 21 18:15:19.194797 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:19.194617    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1827ba9b-0f13-4f48-bf1e-66cc74ba519a" containerName="kube-rbac-proxy"
Jul 21 18:15:19.194797 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:19.194622    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1827ba9b-0f13-4f48-bf1e-66cc74ba519a" containerName="kube-rbac-proxy"
Jul 21 18:15:19.194797 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:19.194628    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7f959bd-b166-4bf3-9d39-061839b2f5fa" containerName="kserve-container"
Jul 21 18:15:19.194797 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:19.194632    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7f959bd-b166-4bf3-9d39-061839b2f5fa" containerName="kserve-container"
Jul 21 18:15:19.194797 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:19.194675    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1827ba9b-0f13-4f48-bf1e-66cc74ba519a" containerName="kserve-container"
Jul 21 18:15:19.194797 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:19.194682    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="06da2dde-96ec-4bbe-9eac-69de21849e6a" containerName="kserve-container"
Jul 21 18:15:19.194797 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:19.194689    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1827ba9b-0f13-4f48-bf1e-66cc74ba519a" containerName="kube-rbac-proxy"
Jul 21 18:15:19.194797 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:19.194695    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7f959bd-b166-4bf3-9d39-061839b2f5fa" containerName="kserve-container"
Jul 21 18:15:19.194797 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:19.194700    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc170a90-69da-4897-a7d7-6d176fdd36ba" containerName="kserve-container"
Jul 21 18:15:19.194797 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:19.194706    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7f959bd-b166-4bf3-9d39-061839b2f5fa" containerName="kube-rbac-proxy"
Jul 21 18:15:19.194797 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:19.194711    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="06da2dde-96ec-4bbe-9eac-69de21849e6a" containerName="kube-rbac-proxy"
Jul 21 18:15:19.194797 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:19.194716    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc170a90-69da-4897-a7d7-6d176fdd36ba" containerName="kube-rbac-proxy"
Jul 21 18:15:19.197611 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:19.197596    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hrpnq/must-gather-6hxmb"
Jul 21 18:15:19.200316 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:19.200295    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hrpnq\"/\"openshift-service-ca.crt\""
Jul 21 18:15:19.200415 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:19.200366    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hrpnq\"/\"default-dockercfg-zfzvn\""
Jul 21 18:15:19.200459 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:19.200408    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hrpnq\"/\"kube-root-ca.crt\""
Jul 21 18:15:19.205932 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:19.205899    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hrpnq/must-gather-6hxmb"]
Jul 21 18:15:19.243525 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:19.243495    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d09c4d0d-4c69-4134-a830-63675dd6d817-must-gather-output\") pod \"must-gather-6hxmb\" (UID: \"d09c4d0d-4c69-4134-a830-63675dd6d817\") " pod="openshift-must-gather-hrpnq/must-gather-6hxmb"
Jul 21 18:15:19.243641 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:19.243541    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvnkk\" (UniqueName: \"kubernetes.io/projected/d09c4d0d-4c69-4134-a830-63675dd6d817-kube-api-access-tvnkk\") pod \"must-gather-6hxmb\" (UID: \"d09c4d0d-4c69-4134-a830-63675dd6d817\") " pod="openshift-must-gather-hrpnq/must-gather-6hxmb"
Jul 21 18:15:19.344488 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:19.344457    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d09c4d0d-4c69-4134-a830-63675dd6d817-must-gather-output\") pod \"must-gather-6hxmb\" (UID: \"d09c4d0d-4c69-4134-a830-63675dd6d817\") " pod="openshift-must-gather-hrpnq/must-gather-6hxmb"
Jul 21 18:15:19.344824 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:19.344501    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvnkk\" (UniqueName: \"kubernetes.io/projected/d09c4d0d-4c69-4134-a830-63675dd6d817-kube-api-access-tvnkk\") pod \"must-gather-6hxmb\" (UID: \"d09c4d0d-4c69-4134-a830-63675dd6d817\") " pod="openshift-must-gather-hrpnq/must-gather-6hxmb"
Jul 21 18:15:19.344824 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:19.344774    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d09c4d0d-4c69-4134-a830-63675dd6d817-must-gather-output\") pod \"must-gather-6hxmb\" (UID: \"d09c4d0d-4c69-4134-a830-63675dd6d817\") " pod="openshift-must-gather-hrpnq/must-gather-6hxmb"
Jul 21 18:15:19.353356 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:19.353324    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvnkk\" (UniqueName: \"kubernetes.io/projected/d09c4d0d-4c69-4134-a830-63675dd6d817-kube-api-access-tvnkk\") pod \"must-gather-6hxmb\" (UID: \"d09c4d0d-4c69-4134-a830-63675dd6d817\") " pod="openshift-must-gather-hrpnq/must-gather-6hxmb"
Jul 21 18:15:19.518338 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:19.518258    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hrpnq/must-gather-6hxmb"
Jul 21 18:15:19.644831 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:19.644793    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hrpnq/must-gather-6hxmb"]
Jul 21 18:15:19.647851 ip-10-0-132-249 kubenswrapper[2563]: W0721 18:15:19.647812    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd09c4d0d_4c69_4134_a830_63675dd6d817.slice/crio-532a12d595b8afff76ef733b05756edcc486060e8616a0bfe822a1c91583bab4 WatchSource:0}: Error finding container 532a12d595b8afff76ef733b05756edcc486060e8616a0bfe822a1c91583bab4: Status 404 returned error can't find the container with id 532a12d595b8afff76ef733b05756edcc486060e8616a0bfe822a1c91583bab4
Jul 21 18:15:19.649830 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:19.649811    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 21 18:15:20.178564 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:20.178531    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hrpnq/must-gather-6hxmb" event={"ID":"d09c4d0d-4c69-4134-a830-63675dd6d817","Type":"ContainerStarted","Data":"532a12d595b8afff76ef733b05756edcc486060e8616a0bfe822a1c91583bab4"}
Jul 21 18:15:21.286835 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:21.286785    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h" podUID="724ce6ed-3557-4232-9b1c-1035c5c14be7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Jul 21 18:15:21.743090 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:21.743004    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9"
Jul 21 18:15:21.866811 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:21.866777    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95l75\" (UniqueName: \"kubernetes.io/projected/59d6e22f-3560-457e-8d5e-95cd24157bb9-kube-api-access-95l75\") pod \"59d6e22f-3560-457e-8d5e-95cd24157bb9\" (UID: \"59d6e22f-3560-457e-8d5e-95cd24157bb9\") "
Jul 21 18:15:21.866995 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:21.866879    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59d6e22f-3560-457e-8d5e-95cd24157bb9-kube-rbac-proxy-sar-config\") pod \"59d6e22f-3560-457e-8d5e-95cd24157bb9\" (UID: \"59d6e22f-3560-457e-8d5e-95cd24157bb9\") "
Jul 21 18:15:21.866995 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:21.866923    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59d6e22f-3560-457e-8d5e-95cd24157bb9-proxy-tls\") pod \"59d6e22f-3560-457e-8d5e-95cd24157bb9\" (UID: \"59d6e22f-3560-457e-8d5e-95cd24157bb9\") "
Jul 21 18:15:21.867541 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:21.867510    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59d6e22f-3560-457e-8d5e-95cd24157bb9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "59d6e22f-3560-457e-8d5e-95cd24157bb9" (UID: "59d6e22f-3560-457e-8d5e-95cd24157bb9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:15:21.870083 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:21.870042    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59d6e22f-3560-457e-8d5e-95cd24157bb9-kube-api-access-95l75" (OuterVolumeSpecName: "kube-api-access-95l75") pod "59d6e22f-3560-457e-8d5e-95cd24157bb9" (UID: "59d6e22f-3560-457e-8d5e-95cd24157bb9"). InnerVolumeSpecName "kube-api-access-95l75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:15:21.870326 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:21.870292    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59d6e22f-3560-457e-8d5e-95cd24157bb9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "59d6e22f-3560-457e-8d5e-95cd24157bb9" (UID: "59d6e22f-3560-457e-8d5e-95cd24157bb9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:15:21.968532 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:21.968496    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59d6e22f-3560-457e-8d5e-95cd24157bb9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 18:15:21.968532 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:21.968528    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59d6e22f-3560-457e-8d5e-95cd24157bb9-proxy-tls\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 18:15:21.968727 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:21.968542    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95l75\" (UniqueName: \"kubernetes.io/projected/59d6e22f-3560-457e-8d5e-95cd24157bb9-kube-api-access-95l75\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 18:15:22.189602 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:22.189567    2563 generic.go:356] "Generic (PLEG): container finished" podID="724ce6ed-3557-4232-9b1c-1035c5c14be7" containerID="d5955ecf75e490c2bad003e904c91940b98180e35e420bdc00f31d20b3175b51" exitCode=0
Jul 21 18:15:22.189780 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:22.189658    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h" event={"ID":"724ce6ed-3557-4232-9b1c-1035c5c14be7","Type":"ContainerDied","Data":"d5955ecf75e490c2bad003e904c91940b98180e35e420bdc00f31d20b3175b51"}
Jul 21 18:15:22.191449 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:22.191427    2563 generic.go:356] "Generic (PLEG): container finished" podID="59d6e22f-3560-457e-8d5e-95cd24157bb9" containerID="5d3dfb2b2f18368624d62e3e6149e803b17d6314e49812fe859a50940ab41b1b" exitCode=0
Jul 21 18:15:22.191578 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:22.191472    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9" event={"ID":"59d6e22f-3560-457e-8d5e-95cd24157bb9","Type":"ContainerDied","Data":"5d3dfb2b2f18368624d62e3e6149e803b17d6314e49812fe859a50940ab41b1b"}
Jul 21 18:15:22.191578 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:22.191482    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9"
Jul 21 18:15:22.191578 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:22.191526    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9" event={"ID":"59d6e22f-3560-457e-8d5e-95cd24157bb9","Type":"ContainerDied","Data":"c0de17d52ea402db07d980efd36b804a6cd6e1ce1ab41a6d329d6676f95c5767"}
Jul 21 18:15:22.191578 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:22.191563    2563 scope.go:117] "RemoveContainer" containerID="c71d51991c9b793ae57908b8c75624071e3d6b619b13c9cfb355d0296b949857"
Jul 21 18:15:22.213648 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:22.213625    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9"]
Jul 21 18:15:22.216919 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:22.216899    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-984cf-predictor-7bdc6476c9-nrlz9"]
Jul 21 18:15:22.765450 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:22.765418    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59d6e22f-3560-457e-8d5e-95cd24157bb9" path="/var/lib/kubelet/pods/59d6e22f-3560-457e-8d5e-95cd24157bb9/volumes"
Jul 21 18:15:23.444844 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:23.444821    2563 scope.go:117] "RemoveContainer" containerID="5d3dfb2b2f18368624d62e3e6149e803b17d6314e49812fe859a50940ab41b1b"
Jul 21 18:15:23.461393 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:23.461368    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h"
Jul 21 18:15:23.583146 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:23.583098    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4c4dg\" (UniqueName: \"kubernetes.io/projected/724ce6ed-3557-4232-9b1c-1035c5c14be7-kube-api-access-4c4dg\") pod \"724ce6ed-3557-4232-9b1c-1035c5c14be7\" (UID: \"724ce6ed-3557-4232-9b1c-1035c5c14be7\") "
Jul 21 18:15:23.583431 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:23.583185    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/724ce6ed-3557-4232-9b1c-1035c5c14be7-proxy-tls\") pod \"724ce6ed-3557-4232-9b1c-1035c5c14be7\" (UID: \"724ce6ed-3557-4232-9b1c-1035c5c14be7\") "
Jul 21 18:15:23.583431 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:23.583265    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/724ce6ed-3557-4232-9b1c-1035c5c14be7-kube-rbac-proxy-sar-config\") pod \"724ce6ed-3557-4232-9b1c-1035c5c14be7\" (UID: \"724ce6ed-3557-4232-9b1c-1035c5c14be7\") "
Jul 21 18:15:23.583672 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:23.583646    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/724ce6ed-3557-4232-9b1c-1035c5c14be7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "724ce6ed-3557-4232-9b1c-1035c5c14be7" (UID: "724ce6ed-3557-4232-9b1c-1035c5c14be7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:15:23.585973 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:23.585947    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/724ce6ed-3557-4232-9b1c-1035c5c14be7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "724ce6ed-3557-4232-9b1c-1035c5c14be7" (UID: "724ce6ed-3557-4232-9b1c-1035c5c14be7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:15:23.586067 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:23.585988    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/724ce6ed-3557-4232-9b1c-1035c5c14be7-kube-api-access-4c4dg" (OuterVolumeSpecName: "kube-api-access-4c4dg") pod "724ce6ed-3557-4232-9b1c-1035c5c14be7" (UID: "724ce6ed-3557-4232-9b1c-1035c5c14be7"). InnerVolumeSpecName "kube-api-access-4c4dg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:15:23.684526 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:23.684483    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c4dg\" (UniqueName: \"kubernetes.io/projected/724ce6ed-3557-4232-9b1c-1035c5c14be7-kube-api-access-4c4dg\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 18:15:23.684526 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:23.684518    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/724ce6ed-3557-4232-9b1c-1035c5c14be7-proxy-tls\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 18:15:23.684526 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:23.684534    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/724ce6ed-3557-4232-9b1c-1035c5c14be7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 18:15:24.203555 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:24.203511    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h" event={"ID":"724ce6ed-3557-4232-9b1c-1035c5c14be7","Type":"ContainerDied","Data":"cda58e921c45271ea359e89c99db6e97f746d87e98d37aeb494accea89be8a07"}
Jul 21 18:15:24.203960 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:24.203562    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h"
Jul 21 18:15:24.229179 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:24.229146    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h"]
Jul 21 18:15:24.231986 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:24.231960    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-984cf-predictor-58f8b9cf74-xkd4h"]
Jul 21 18:15:24.638766 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:24.638533    2563 scope.go:117] "RemoveContainer" containerID="c71d51991c9b793ae57908b8c75624071e3d6b619b13c9cfb355d0296b949857"
Jul 21 18:15:24.638902 ip-10-0-132-249 kubenswrapper[2563]: E0721 18:15:24.638882    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c71d51991c9b793ae57908b8c75624071e3d6b619b13c9cfb355d0296b949857\": container with ID starting with c71d51991c9b793ae57908b8c75624071e3d6b619b13c9cfb355d0296b949857 not found: ID does not exist" containerID="c71d51991c9b793ae57908b8c75624071e3d6b619b13c9cfb355d0296b949857"
Jul 21 18:15:24.638950 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:24.638915    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c71d51991c9b793ae57908b8c75624071e3d6b619b13c9cfb355d0296b949857"} err="failed to get container status \"c71d51991c9b793ae57908b8c75624071e3d6b619b13c9cfb355d0296b949857\": rpc error: code = NotFound desc = could not find container \"c71d51991c9b793ae57908b8c75624071e3d6b619b13c9cfb355d0296b949857\": container with ID starting with c71d51991c9b793ae57908b8c75624071e3d6b619b13c9cfb355d0296b949857 not found: ID does not exist"
Jul 21 18:15:24.638950 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:24.638937    2563 scope.go:117] "RemoveContainer" containerID="5d3dfb2b2f18368624d62e3e6149e803b17d6314e49812fe859a50940ab41b1b"
Jul 21 18:15:24.639237 ip-10-0-132-249 kubenswrapper[2563]: E0721 18:15:24.639219    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d3dfb2b2f18368624d62e3e6149e803b17d6314e49812fe859a50940ab41b1b\": container with ID starting with 5d3dfb2b2f18368624d62e3e6149e803b17d6314e49812fe859a50940ab41b1b not found: ID does not exist" containerID="5d3dfb2b2f18368624d62e3e6149e803b17d6314e49812fe859a50940ab41b1b"
Jul 21 18:15:24.639326 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:24.639246    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d3dfb2b2f18368624d62e3e6149e803b17d6314e49812fe859a50940ab41b1b"} err="failed to get container status \"5d3dfb2b2f18368624d62e3e6149e803b17d6314e49812fe859a50940ab41b1b\": rpc error: code = NotFound desc = could not find container \"5d3dfb2b2f18368624d62e3e6149e803b17d6314e49812fe859a50940ab41b1b\": container with ID starting with 5d3dfb2b2f18368624d62e3e6149e803b17d6314e49812fe859a50940ab41b1b not found: ID does not exist"
Jul 21 18:15:24.639326 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:24.639268    2563 scope.go:117] "RemoveContainer" containerID="3d9c066046ea355312e57edbbafdfb7f3ad5441ca5dc563b65f8afe310ac589a"
Jul 21 18:15:24.765270 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:24.765239    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="724ce6ed-3557-4232-9b1c-1035c5c14be7" path="/var/lib/kubelet/pods/724ce6ed-3557-4232-9b1c-1035c5c14be7/volumes"
Jul 21 18:15:24.824945 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:24.824922    2563 scope.go:117] "RemoveContainer" containerID="d5955ecf75e490c2bad003e904c91940b98180e35e420bdc00f31d20b3175b51"
Jul 21 18:15:26.216607 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:26.216568    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hrpnq/must-gather-6hxmb" event={"ID":"d09c4d0d-4c69-4134-a830-63675dd6d817","Type":"ContainerStarted","Data":"7cb9e142d4342f794caed4b638d89be51ef67cd839c0a86d26e0601728e72d3d"}
Jul 21 18:15:26.217043 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:26.216616    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hrpnq/must-gather-6hxmb" event={"ID":"d09c4d0d-4c69-4134-a830-63675dd6d817","Type":"ContainerStarted","Data":"c646cd39f5531ff4bd93ab75dc0bbbf3455c7b13415ed3d07c53650dffc3a1de"}
Jul 21 18:15:26.234003 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:26.233951    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hrpnq/must-gather-6hxmb" podStartSLOduration=1.045660973 podStartE2EDuration="7.233933219s" totalImagesPullingTime="6.188272246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 18:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 18:15:26.232963707 +0000 UTC m=+2953.995103330" watchObservedRunningTime="2026-07-21 18:15:26.233933219 +0000 UTC m=+2953.996072823"
Jul 21 18:15:48.316888 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:48.316855    2563 generic.go:356] "Generic (PLEG): container finished" podID="d09c4d0d-4c69-4134-a830-63675dd6d817" containerID="c646cd39f5531ff4bd93ab75dc0bbbf3455c7b13415ed3d07c53650dffc3a1de" exitCode=0
Jul 21 18:15:48.317344 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:48.316928    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hrpnq/must-gather-6hxmb" event={"ID":"d09c4d0d-4c69-4134-a830-63675dd6d817","Type":"ContainerDied","Data":"c646cd39f5531ff4bd93ab75dc0bbbf3455c7b13415ed3d07c53650dffc3a1de"}
Jul 21 18:15:48.317344 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:48.317272    2563 scope.go:117] "RemoveContainer" containerID="c646cd39f5531ff4bd93ab75dc0bbbf3455c7b13415ed3d07c53650dffc3a1de"
Jul 21 18:15:48.740386 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:48.740358    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-hrpnq_must-gather-6hxmb_d09c4d0d-4c69-4134-a830-63675dd6d817/gather/0.log"
Jul 21 18:15:49.380317 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:49.380288    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-p4dfr/must-gather-5tqrw"]
Jul 21 18:15:49.380695 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:49.380547    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59d6e22f-3560-457e-8d5e-95cd24157bb9" containerName="kserve-container"
Jul 21 18:15:49.380695 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:49.380557    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="59d6e22f-3560-457e-8d5e-95cd24157bb9" containerName="kserve-container"
Jul 21 18:15:49.380695 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:49.380567    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="724ce6ed-3557-4232-9b1c-1035c5c14be7" containerName="kserve-container"
Jul 21 18:15:49.380695 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:49.380572    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="724ce6ed-3557-4232-9b1c-1035c5c14be7" containerName="kserve-container"
Jul 21 18:15:49.380695 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:49.380580    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59d6e22f-3560-457e-8d5e-95cd24157bb9" containerName="kube-rbac-proxy"
Jul 21 18:15:49.380695 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:49.380586    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="59d6e22f-3560-457e-8d5e-95cd24157bb9" containerName="kube-rbac-proxy"
Jul 21 18:15:49.380695 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:49.380599    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="724ce6ed-3557-4232-9b1c-1035c5c14be7" containerName="kube-rbac-proxy"
Jul 21 18:15:49.380695 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:49.380604    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="724ce6ed-3557-4232-9b1c-1035c5c14be7" containerName="kube-rbac-proxy"
Jul 21 18:15:49.380695 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:49.380653    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="724ce6ed-3557-4232-9b1c-1035c5c14be7" containerName="kserve-container"
Jul 21 18:15:49.380695 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:49.380659    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="59d6e22f-3560-457e-8d5e-95cd24157bb9" containerName="kserve-container"
Jul 21 18:15:49.380695 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:49.380670    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="724ce6ed-3557-4232-9b1c-1035c5c14be7" containerName="kube-rbac-proxy"
Jul 21 18:15:49.380695 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:49.380674    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="59d6e22f-3560-457e-8d5e-95cd24157bb9" containerName="kube-rbac-proxy"
Jul 21 18:15:49.382587 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:49.382569    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p4dfr/must-gather-5tqrw"
Jul 21 18:15:49.384989 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:49.384968    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p4dfr\"/\"kube-root-ca.crt\""
Jul 21 18:15:49.385081 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:49.385026    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-p4dfr\"/\"default-dockercfg-85wwv\""
Jul 21 18:15:49.385209 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:49.385193    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p4dfr\"/\"openshift-service-ca.crt\""
Jul 21 18:15:49.395051 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:49.395029    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p4dfr/must-gather-5tqrw"]
Jul 21 18:15:49.507517 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:49.507485    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/944d4e6d-7bfa-4a7c-b7d5-e16866fef041-must-gather-output\") pod \"must-gather-5tqrw\" (UID: \"944d4e6d-7bfa-4a7c-b7d5-e16866fef041\") " pod="openshift-must-gather-p4dfr/must-gather-5tqrw"
Jul 21 18:15:49.507678 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:49.507532    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twxhh\" (UniqueName: \"kubernetes.io/projected/944d4e6d-7bfa-4a7c-b7d5-e16866fef041-kube-api-access-twxhh\") pod \"must-gather-5tqrw\" (UID: \"944d4e6d-7bfa-4a7c-b7d5-e16866fef041\") " pod="openshift-must-gather-p4dfr/must-gather-5tqrw"
Jul 21 18:15:49.608358 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:49.608326    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/944d4e6d-7bfa-4a7c-b7d5-e16866fef041-must-gather-output\") pod \"must-gather-5tqrw\" (UID: \"944d4e6d-7bfa-4a7c-b7d5-e16866fef041\") " pod="openshift-must-gather-p4dfr/must-gather-5tqrw"
Jul 21 18:15:49.608507 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:49.608371    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twxhh\" (UniqueName: \"kubernetes.io/projected/944d4e6d-7bfa-4a7c-b7d5-e16866fef041-kube-api-access-twxhh\") pod \"must-gather-5tqrw\" (UID: \"944d4e6d-7bfa-4a7c-b7d5-e16866fef041\") " pod="openshift-must-gather-p4dfr/must-gather-5tqrw"
Jul 21 18:15:49.608647 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:49.608630    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/944d4e6d-7bfa-4a7c-b7d5-e16866fef041-must-gather-output\") pod \"must-gather-5tqrw\" (UID: \"944d4e6d-7bfa-4a7c-b7d5-e16866fef041\") " pod="openshift-must-gather-p4dfr/must-gather-5tqrw"
Jul 21 18:15:49.616105 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:49.616084    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twxhh\" (UniqueName: \"kubernetes.io/projected/944d4e6d-7bfa-4a7c-b7d5-e16866fef041-kube-api-access-twxhh\") pod \"must-gather-5tqrw\" (UID: \"944d4e6d-7bfa-4a7c-b7d5-e16866fef041\") " pod="openshift-must-gather-p4dfr/must-gather-5tqrw"
Jul 21 18:15:49.691745 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:49.691679    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p4dfr/must-gather-5tqrw"
Jul 21 18:15:49.817705 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:49.817675    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p4dfr/must-gather-5tqrw"]
Jul 21 18:15:49.820780 ip-10-0-132-249 kubenswrapper[2563]: W0721 18:15:49.820754    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod944d4e6d_7bfa_4a7c_b7d5_e16866fef041.slice/crio-83dc97bde2c2c43931b5c78df212091f787acd80b00a88af9287d6d2d2202882 WatchSource:0}: Error finding container 83dc97bde2c2c43931b5c78df212091f787acd80b00a88af9287d6d2d2202882: Status 404 returned error can't find the container with id 83dc97bde2c2c43931b5c78df212091f787acd80b00a88af9287d6d2d2202882
Jul 21 18:15:50.325382 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:50.325349    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p4dfr/must-gather-5tqrw" event={"ID":"944d4e6d-7bfa-4a7c-b7d5-e16866fef041","Type":"ContainerStarted","Data":"83dc97bde2c2c43931b5c78df212091f787acd80b00a88af9287d6d2d2202882"}
Jul 21 18:15:51.331840 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:51.331805    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p4dfr/must-gather-5tqrw" event={"ID":"944d4e6d-7bfa-4a7c-b7d5-e16866fef041","Type":"ContainerStarted","Data":"3f0b6852e484c267df19003fbc5dc5199c3f6bd9374e9d9eed1dd384d1ea62f1"}
Jul 21 18:15:51.332324 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:51.331849    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p4dfr/must-gather-5tqrw" event={"ID":"944d4e6d-7bfa-4a7c-b7d5-e16866fef041","Type":"ContainerStarted","Data":"2d540f80e27db90448411e7ee0da2c5be9d390c5064ca917f2b97dd0e4a149e1"}
Jul 21 18:15:51.348560 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:51.348513    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-p4dfr/must-gather-5tqrw" podStartSLOduration=1.598627612 podStartE2EDuration="2.348495702s" totalImagesPullingTime="749.86809ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 18:15:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 18:15:51.347449775 +0000 UTC m=+2979.109589410" watchObservedRunningTime="2026-07-21 18:15:51.348495702 +0000 UTC m=+2979.110635305"
Jul 21 18:15:52.014777 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:52.014744    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-g76xh_fa6ca187-d1f3-4d33-bd20-f70784dbbe02/global-pull-secret-syncer/0.log"
Jul 21 18:15:52.176440 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:52.176408    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rg2jl_1b38cc54-ebe3-48aa-b1ca-671f5f427f93/konnectivity-agent/0.log"
Jul 21 18:15:52.241078 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:52.241046    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-249.ec2.internal_fb93a1b33c0d6ced6dfb0f4505a58dc6/haproxy/0.log"
Jul 21 18:15:54.205996 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:54.205949    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-hrpnq/must-gather-6hxmb"]
Jul 21 18:15:54.206612 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:54.206237    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-hrpnq/must-gather-6hxmb" podUID="d09c4d0d-4c69-4134-a830-63675dd6d817" containerName="copy" containerID="cri-o://7cb9e142d4342f794caed4b638d89be51ef67cd839c0a86d26e0601728e72d3d" gracePeriod=2
Jul 21 18:15:54.211986 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:54.211955    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-hrpnq/must-gather-6hxmb"]
Jul 21 18:15:54.212263 ip-10-0-132-249 kubenswrapper[2563]: E0721 18:15:54.212229    2563 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"d09c4d0d-4c69-4134-a830-63675dd6d817\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"openshift-must-gather-hrpnq\"/\"must-gather-6hxmb\": pods \"must-gather-6hxmb\" not found" pod="openshift-must-gather-hrpnq/must-gather-6hxmb"
Jul 21 18:15:54.353035 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:54.352952    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-hrpnq_must-gather-6hxmb_d09c4d0d-4c69-4134-a830-63675dd6d817/copy/0.log"
Jul 21 18:15:54.356379 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:54.355468    2563 generic.go:356] "Generic (PLEG): container finished" podID="d09c4d0d-4c69-4134-a830-63675dd6d817" containerID="7cb9e142d4342f794caed4b638d89be51ef67cd839c0a86d26e0601728e72d3d" exitCode=143
Jul 21 18:15:54.582528 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:54.582225    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-hrpnq_must-gather-6hxmb_d09c4d0d-4c69-4134-a830-63675dd6d817/copy/0.log"
Jul 21 18:15:54.585668 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:54.585401    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hrpnq/must-gather-6hxmb"
Jul 21 18:15:54.591794 ip-10-0-132-249 kubenswrapper[2563]: E0721 18:15:54.590690    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-6hxmb\" is forbidden: User \"system:node:ip-10-0-132-249.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-hrpnq\": no relationship found between node 'ip-10-0-132-249.ec2.internal' and this object" podUID="d09c4d0d-4c69-4134-a830-63675dd6d817" pod="openshift-must-gather-hrpnq/must-gather-6hxmb"
Jul 21 18:15:54.761960 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:54.761921    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvnkk\" (UniqueName: \"kubernetes.io/projected/d09c4d0d-4c69-4134-a830-63675dd6d817-kube-api-access-tvnkk\") pod \"d09c4d0d-4c69-4134-a830-63675dd6d817\" (UID: \"d09c4d0d-4c69-4134-a830-63675dd6d817\") "
Jul 21 18:15:54.762184 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:54.762056    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d09c4d0d-4c69-4134-a830-63675dd6d817-must-gather-output\") pod \"d09c4d0d-4c69-4134-a830-63675dd6d817\" (UID: \"d09c4d0d-4c69-4134-a830-63675dd6d817\") "
Jul 21 18:15:54.763921 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:54.763852    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d09c4d0d-4c69-4134-a830-63675dd6d817-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "d09c4d0d-4c69-4134-a830-63675dd6d817" (UID: "d09c4d0d-4c69-4134-a830-63675dd6d817"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 18:15:54.766476 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:54.766426    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d09c4d0d-4c69-4134-a830-63675dd6d817-kube-api-access-tvnkk" (OuterVolumeSpecName: "kube-api-access-tvnkk") pod "d09c4d0d-4c69-4134-a830-63675dd6d817" (UID: "d09c4d0d-4c69-4134-a830-63675dd6d817"). InnerVolumeSpecName "kube-api-access-tvnkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:15:54.771324 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:54.767919    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d09c4d0d-4c69-4134-a830-63675dd6d817" path="/var/lib/kubelet/pods/d09c4d0d-4c69-4134-a830-63675dd6d817/volumes"
Jul 21 18:15:54.864160 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:54.863991    2563 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d09c4d0d-4c69-4134-a830-63675dd6d817-must-gather-output\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 18:15:54.864160 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:54.864031    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvnkk\" (UniqueName: \"kubernetes.io/projected/d09c4d0d-4c69-4134-a830-63675dd6d817-kube-api-access-tvnkk\") on node \"ip-10-0-132-249.ec2.internal\" DevicePath \"\""
Jul 21 18:15:55.363758 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:55.363726    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-hrpnq_must-gather-6hxmb_d09c4d0d-4c69-4134-a830-63675dd6d817/copy/0.log"
Jul 21 18:15:55.364551 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:55.364332    2563 scope.go:117] "RemoveContainer" containerID="7cb9e142d4342f794caed4b638d89be51ef67cd839c0a86d26e0601728e72d3d"
Jul 21 18:15:55.364551 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:55.364487    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hrpnq/must-gather-6hxmb"
Jul 21 18:15:55.376962 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:55.376917    2563 scope.go:117] "RemoveContainer" containerID="c646cd39f5531ff4bd93ab75dc0bbbf3455c7b13415ed3d07c53650dffc3a1de"
Jul 21 18:15:56.108931 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:56.108847    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-rn2q5_7fe58db6-f487-48b9-a4cc-16d5a4844298/kube-state-metrics/0.log"
Jul 21 18:15:56.150742 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:56.150706    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-rn2q5_7fe58db6-f487-48b9-a4cc-16d5a4844298/kube-rbac-proxy-main/0.log"
Jul 21 18:15:56.191277 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:56.191244    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-rn2q5_7fe58db6-f487-48b9-a4cc-16d5a4844298/kube-rbac-proxy-self/0.log"
Jul 21 18:15:56.281202 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:56.281167    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-d2htd_79e3c744-e62b-4582-b305-6a3586a7a37e/monitoring-plugin/0.log"
Jul 21 18:15:56.447539 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:56.447503    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5hwc2_ee650e41-f4c6-4ff7-bcf1-e9b74259434e/node-exporter/0.log"
Jul 21 18:15:56.479076 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:56.479049    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5hwc2_ee650e41-f4c6-4ff7-bcf1-e9b74259434e/kube-rbac-proxy/0.log"
Jul 21 18:15:56.506467 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:56.506440    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5hwc2_ee650e41-f4c6-4ff7-bcf1-e9b74259434e/init-textfile/0.log"
Jul 21 18:15:56.731186 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:56.731068    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_df270e60-53ca-4479-9e00-78d0d0378cb1/prometheus/0.log"
Jul 21 18:15:56.760254 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:56.760223    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_df270e60-53ca-4479-9e00-78d0d0378cb1/config-reloader/0.log"
Jul 21 18:15:56.791983 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:56.791918    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_df270e60-53ca-4479-9e00-78d0d0378cb1/thanos-sidecar/0.log"
Jul 21 18:15:56.818337 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:56.818306    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_df270e60-53ca-4479-9e00-78d0d0378cb1/kube-rbac-proxy-web/0.log"
Jul 21 18:15:56.849698 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:56.849668    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_df270e60-53ca-4479-9e00-78d0d0378cb1/kube-rbac-proxy/0.log"
Jul 21 18:15:56.883992 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:56.883928    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_df270e60-53ca-4479-9e00-78d0d0378cb1/kube-rbac-proxy-thanos/0.log"
Jul 21 18:15:56.914954 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:56.914921    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_df270e60-53ca-4479-9e00-78d0d0378cb1/init-config-reloader/0.log"
Jul 21 18:15:56.958964 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:56.958922    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-9cd5l_23805b10-82e7-4dde-9ecc-4c294d17e1e8/prometheus-operator/0.log"
Jul 21 18:15:56.977552 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:56.977519    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-9cd5l_23805b10-82e7-4dde-9ecc-4c294d17e1e8/kube-rbac-proxy/0.log"
Jul 21 18:15:57.008359 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:15:57.008275    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-znnms_d7cd9ffd-b8cf-4df1-a7a0-1087096d20a5/prometheus-operator-admission-webhook/0.log"
Jul 21 18:16:00.957325 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:00.957293    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-p4dfr/perf-node-gather-daemonset-vb2vf"]
Jul 21 18:16:00.958277 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:00.958252    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d09c4d0d-4c69-4134-a830-63675dd6d817" containerName="copy"
Jul 21 18:16:00.958429 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:00.958417    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d09c4d0d-4c69-4134-a830-63675dd6d817" containerName="copy"
Jul 21 18:16:00.958512 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:00.958504    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d09c4d0d-4c69-4134-a830-63675dd6d817" containerName="gather"
Jul 21 18:16:00.958587 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:00.958579    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d09c4d0d-4c69-4134-a830-63675dd6d817" containerName="gather"
Jul 21 18:16:00.958755 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:00.958735    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d09c4d0d-4c69-4134-a830-63675dd6d817" containerName="gather"
Jul 21 18:16:00.958755 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:00.958754    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d09c4d0d-4c69-4134-a830-63675dd6d817" containerName="copy"
Jul 21 18:16:00.965112 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:00.965088    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p4dfr/perf-node-gather-daemonset-vb2vf"
Jul 21 18:16:00.970351 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:00.970322    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p4dfr/perf-node-gather-daemonset-vb2vf"]
Jul 21 18:16:01.017582 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:01.017543    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/12758ea6-c25c-4575-a8ff-e604849d2061-podres\") pod \"perf-node-gather-daemonset-vb2vf\" (UID: \"12758ea6-c25c-4575-a8ff-e604849d2061\") " pod="openshift-must-gather-p4dfr/perf-node-gather-daemonset-vb2vf"
Jul 21 18:16:01.017783 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:01.017596    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/12758ea6-c25c-4575-a8ff-e604849d2061-proc\") pod \"perf-node-gather-daemonset-vb2vf\" (UID: \"12758ea6-c25c-4575-a8ff-e604849d2061\") " pod="openshift-must-gather-p4dfr/perf-node-gather-daemonset-vb2vf"
Jul 21 18:16:01.017783 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:01.017615    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/12758ea6-c25c-4575-a8ff-e604849d2061-sys\") pod \"perf-node-gather-daemonset-vb2vf\" (UID: \"12758ea6-c25c-4575-a8ff-e604849d2061\") " pod="openshift-must-gather-p4dfr/perf-node-gather-daemonset-vb2vf"
Jul 21 18:16:01.017783 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:01.017629    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/12758ea6-c25c-4575-a8ff-e604849d2061-lib-modules\") pod \"perf-node-gather-daemonset-vb2vf\" (UID: \"12758ea6-c25c-4575-a8ff-e604849d2061\") " pod="openshift-must-gather-p4dfr/perf-node-gather-daemonset-vb2vf"
Jul 21 18:16:01.017783 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:01.017651    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8jjp\" (UniqueName: \"kubernetes.io/projected/12758ea6-c25c-4575-a8ff-e604849d2061-kube-api-access-g8jjp\") pod \"perf-node-gather-daemonset-vb2vf\" (UID: \"12758ea6-c25c-4575-a8ff-e604849d2061\") " pod="openshift-must-gather-p4dfr/perf-node-gather-daemonset-vb2vf"
Jul 21 18:16:01.114398 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:01.114353    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c9sl8_0c546573-c93f-4765-bc1d-94c195e3a6da/dns/0.log"
Jul 21 18:16:01.118742 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:01.118708    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/12758ea6-c25c-4575-a8ff-e604849d2061-podres\") pod \"perf-node-gather-daemonset-vb2vf\" (UID: \"12758ea6-c25c-4575-a8ff-e604849d2061\") " pod="openshift-must-gather-p4dfr/perf-node-gather-daemonset-vb2vf"
Jul 21 18:16:01.118890 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:01.118766    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/12758ea6-c25c-4575-a8ff-e604849d2061-proc\") pod \"perf-node-gather-daemonset-vb2vf\" (UID: \"12758ea6-c25c-4575-a8ff-e604849d2061\") " pod="openshift-must-gather-p4dfr/perf-node-gather-daemonset-vb2vf"
Jul 21 18:16:01.118890 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:01.118796    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/12758ea6-c25c-4575-a8ff-e604849d2061-sys\") pod \"perf-node-gather-daemonset-vb2vf\" (UID: \"12758ea6-c25c-4575-a8ff-e604849d2061\") " pod="openshift-must-gather-p4dfr/perf-node-gather-daemonset-vb2vf"
Jul 21 18:16:01.118890 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:01.118818    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/12758ea6-c25c-4575-a8ff-e604849d2061-lib-modules\") pod \"perf-node-gather-daemonset-vb2vf\" (UID: \"12758ea6-c25c-4575-a8ff-e604849d2061\") " pod="openshift-must-gather-p4dfr/perf-node-gather-daemonset-vb2vf"
Jul 21 18:16:01.118890 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:01.118845    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8jjp\" (UniqueName: \"kubernetes.io/projected/12758ea6-c25c-4575-a8ff-e604849d2061-kube-api-access-g8jjp\") pod \"perf-node-gather-daemonset-vb2vf\" (UID: \"12758ea6-c25c-4575-a8ff-e604849d2061\") " pod="openshift-must-gather-p4dfr/perf-node-gather-daemonset-vb2vf"
Jul 21 18:16:01.118890 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:01.118884    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/12758ea6-c25c-4575-a8ff-e604849d2061-podres\") pod \"perf-node-gather-daemonset-vb2vf\" (UID: \"12758ea6-c25c-4575-a8ff-e604849d2061\") " pod="openshift-must-gather-p4dfr/perf-node-gather-daemonset-vb2vf"
Jul 21 18:16:01.119159 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:01.118941    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/12758ea6-c25c-4575-a8ff-e604849d2061-sys\") pod \"perf-node-gather-daemonset-vb2vf\" (UID: \"12758ea6-c25c-4575-a8ff-e604849d2061\") " pod="openshift-must-gather-p4dfr/perf-node-gather-daemonset-vb2vf"
Jul 21 18:16:01.119159 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:01.118984    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/12758ea6-c25c-4575-a8ff-e604849d2061-proc\") pod \"perf-node-gather-daemonset-vb2vf\" (UID: \"12758ea6-c25c-4575-a8ff-e604849d2061\") " pod="openshift-must-gather-p4dfr/perf-node-gather-daemonset-vb2vf"
Jul 21 18:16:01.119159 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:01.119076    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/12758ea6-c25c-4575-a8ff-e604849d2061-lib-modules\") pod \"perf-node-gather-daemonset-vb2vf\" (UID: \"12758ea6-c25c-4575-a8ff-e604849d2061\") " pod="openshift-must-gather-p4dfr/perf-node-gather-daemonset-vb2vf"
Jul 21 18:16:01.126393 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:01.126370    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8jjp\" (UniqueName: \"kubernetes.io/projected/12758ea6-c25c-4575-a8ff-e604849d2061-kube-api-access-g8jjp\") pod \"perf-node-gather-daemonset-vb2vf\" (UID: \"12758ea6-c25c-4575-a8ff-e604849d2061\") " pod="openshift-must-gather-p4dfr/perf-node-gather-daemonset-vb2vf"
Jul 21 18:16:01.134734 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:01.134709    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c9sl8_0c546573-c93f-4765-bc1d-94c195e3a6da/kube-rbac-proxy/0.log"
Jul 21 18:16:01.279168 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:01.279054    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p4dfr/perf-node-gather-daemonset-vb2vf"
Jul 21 18:16:01.303072 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:01.303047    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gxckp_189f8b1f-23e2-4447-90bd-ffa91affc6c4/dns-node-resolver/0.log"
Jul 21 18:16:01.450630 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:01.450595    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p4dfr/perf-node-gather-daemonset-vb2vf"]
Jul 21 18:16:01.454049 ip-10-0-132-249 kubenswrapper[2563]: W0721 18:16:01.454018    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod12758ea6_c25c_4575_a8ff_e604849d2061.slice/crio-f4a4bced80968fcc76d2c9ebb3d527ebf53bcd0374d3aa9376edfe9367445ce8 WatchSource:0}: Error finding container f4a4bced80968fcc76d2c9ebb3d527ebf53bcd0374d3aa9376edfe9367445ce8: Status 404 returned error can't find the container with id f4a4bced80968fcc76d2c9ebb3d527ebf53bcd0374d3aa9376edfe9367445ce8
Jul 21 18:16:01.848560 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:01.848487    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mgp2n_5f7ddf4c-27b9-4b30-8d4a-483cfa1fb74a/node-ca/0.log"
Jul 21 18:16:02.402220 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:02.402185    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p4dfr/perf-node-gather-daemonset-vb2vf" event={"ID":"12758ea6-c25c-4575-a8ff-e604849d2061","Type":"ContainerStarted","Data":"14eb04d977d6cb65bbfec5ab91f02388ef10a93eda2d26e66282f649da7519dc"}
Jul 21 18:16:02.402648 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:02.402232    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p4dfr/perf-node-gather-daemonset-vb2vf" event={"ID":"12758ea6-c25c-4575-a8ff-e604849d2061","Type":"ContainerStarted","Data":"f4a4bced80968fcc76d2c9ebb3d527ebf53bcd0374d3aa9376edfe9367445ce8"}
Jul 21 18:16:02.402648 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:02.402324    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-p4dfr/perf-node-gather-daemonset-vb2vf"
Jul 21 18:16:02.417382 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:02.417328    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-p4dfr/perf-node-gather-daemonset-vb2vf" podStartSLOduration=2.4173095829999998 podStartE2EDuration="2.417309583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 18:16:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 18:16:02.417204667 +0000 UTC m=+2990.179344271" watchObservedRunningTime="2026-07-21 18:16:02.417309583 +0000 UTC m=+2990.179449188"
Jul 21 18:16:03.007660 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:03.007625    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x99p9_c359a027-359d-42f9-91e4-27af042f8d9d/serve-healthcheck-canary/0.log"
Jul 21 18:16:03.461787 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:03.461760    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dptd4_15d6706f-5736-4ae6-b58d-677e456478c7/kube-rbac-proxy/0.log"
Jul 21 18:16:03.481649 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:03.481621    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dptd4_15d6706f-5736-4ae6-b58d-677e456478c7/exporter/0.log"
Jul 21 18:16:03.502816 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:03.502788    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dptd4_15d6706f-5736-4ae6-b58d-677e456478c7/extractor/0.log"
Jul 21 18:16:05.768023 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:05.767994    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-cnzn7_75a8d379-853b-456d-9434-1ae6451caf47/server/0.log"
Jul 21 18:16:05.947929 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:05.947891    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-qwdl2_52ad3d48-f859-4ac0-b406-4944f6e62afa/manager/0.log"
Jul 21 18:16:05.990429 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:05.990395    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-gjxfp_0153b238-ec76-4384-b471-226b0feb6df7/s3-init/0.log"
Jul 21 18:16:06.006322 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:06.006294    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-gjxfp_0153b238-ec76-4384-b471-226b0feb6df7/download-hf-model/0.log"
Jul 21 18:16:08.417783 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:08.417758    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-p4dfr/perf-node-gather-daemonset-vb2vf"
Jul 21 18:16:11.387852 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:11.387820    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5872t_35f84d29-3ea1-4c65-b000-d910c8012948/kube-multus-additional-cni-plugins/0.log"
Jul 21 18:16:11.414818 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:11.414789    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5872t_35f84d29-3ea1-4c65-b000-d910c8012948/egress-router-binary-copy/0.log"
Jul 21 18:16:11.438575 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:11.438545    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5872t_35f84d29-3ea1-4c65-b000-d910c8012948/cni-plugins/0.log"
Jul 21 18:16:11.463436 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:11.463371    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5872t_35f84d29-3ea1-4c65-b000-d910c8012948/bond-cni-plugin/0.log"
Jul 21 18:16:11.486406 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:11.486381    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5872t_35f84d29-3ea1-4c65-b000-d910c8012948/routeoverride-cni/0.log"
Jul 21 18:16:11.510113 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:11.510083    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5872t_35f84d29-3ea1-4c65-b000-d910c8012948/whereabouts-cni-bincopy/0.log"
Jul 21 18:16:11.534843 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:11.534818    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5872t_35f84d29-3ea1-4c65-b000-d910c8012948/whereabouts-cni/0.log"
Jul 21 18:16:11.930417 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:11.930387    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-zp2df_2998a9d6-86dc-4ab8-961d-7f873ce559c6/kube-multus/0.log"
Jul 21 18:16:11.952343 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:11.952314    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-sxmzk_82279305-9475-429f-a495-143fc616fb34/network-metrics-daemon/0.log"
Jul 21 18:16:11.971217 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:11.971191    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-sxmzk_82279305-9475-429f-a495-143fc616fb34/kube-rbac-proxy/0.log"
Jul 21 18:16:13.028605 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:13.028571    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dq7d6_a61ba40a-d3c2-45a9-8525-5e40c9de49b6/ovn-controller/0.log"
Jul 21 18:16:13.043892 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:13.043869    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dq7d6_a61ba40a-d3c2-45a9-8525-5e40c9de49b6/ovn-acl-logging/0.log"
Jul 21 18:16:13.066108 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:13.066086    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dq7d6_a61ba40a-d3c2-45a9-8525-5e40c9de49b6/kube-rbac-proxy-node/0.log"
Jul 21 18:16:13.088569 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:13.088525    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dq7d6_a61ba40a-d3c2-45a9-8525-5e40c9de49b6/kube-rbac-proxy-ovn-metrics/0.log"
Jul 21 18:16:13.106928 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:13.106904    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dq7d6_a61ba40a-d3c2-45a9-8525-5e40c9de49b6/northd/0.log"
Jul 21 18:16:13.125847 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:13.125822    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dq7d6_a61ba40a-d3c2-45a9-8525-5e40c9de49b6/nbdb/0.log"
Jul 21 18:16:13.145745 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:13.145725    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dq7d6_a61ba40a-d3c2-45a9-8525-5e40c9de49b6/sbdb/0.log"
Jul 21 18:16:13.214520 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:13.214433    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dq7d6_a61ba40a-d3c2-45a9-8525-5e40c9de49b6/ovnkube-controller/0.log"
Jul 21 18:16:14.493018 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:14.492990    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-d8gnt_b92707b0-e59d-4123-82e2-658e4361cb6e/network-check-target-container/0.log"
Jul 21 18:16:15.491847 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:15.491822    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-zqxmf_c5401269-8543-4227-a538-95e2f3051c4b/iptables-alerter/0.log"
Jul 21 18:16:16.103119 ip-10-0-132-249 kubenswrapper[2563]: I0721 18:16:16.103089    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-fwcw8_8e07071d-5eed-4b15-844b-8dd2c6bf1eaa/tuned/0.log"