Jul 02 12:36:40.423260 ip-10-0-143-88 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 02 12:36:40.423274 ip-10-0-143-88 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 02 12:36:40.423284 ip-10-0-143-88 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 02 12:36:40.423591 ip-10-0-143-88 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 02 12:36:50.379612 ip-10-0-143-88 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 02 12:36:50.379627 ip-10-0-143-88 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 0e0d247fd01341e5bd640e5c81832c17 --
Jul 02 12:38:29.810544 ip-10-0-143-88 systemd[1]: Starting Kubernetes Kubelet...
Jul 02 12:38:30.277469 ip-10-0-143-88 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 12:38:30.277469 ip-10-0-143-88 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 02 12:38:30.277469 ip-10-0-143-88 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 12:38:30.277469 ip-10-0-143-88 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 02 12:38:30.277469 ip-10-0-143-88 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 12:38:30.277469 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.269745    2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 02 12:38:30.277469 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274717    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 12:38:30.277469 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274732    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 12:38:30.277469 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274737    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 12:38:30.277469 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274743    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 12:38:30.277469 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274747    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 12:38:30.277469 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274752    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 12:38:30.277469 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274756    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 12:38:30.277469 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274761    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 12:38:30.299669 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274765    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 12:38:30.299669 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274769    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 12:38:30.299669 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274773    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 12:38:30.299669 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274777    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 12:38:30.299669 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274781    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 12:38:30.299669 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274785    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 12:38:30.299669 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274790    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 12:38:30.299669 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274794    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 12:38:30.299669 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274800    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 12:38:30.299669 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274805    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 12:38:30.299669 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274810    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 12:38:30.299669 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274815    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 12:38:30.299669 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274828    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 12:38:30.299669 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274832    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 12:38:30.299669 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274837    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 12:38:30.299669 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274841    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 12:38:30.299669 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274846    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 12:38:30.299669 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274850    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 12:38:30.299669 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274854    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 12:38:30.303781 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274860    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 12:38:30.303781 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274864    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 12:38:30.303781 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274868    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 12:38:30.303781 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274872    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 12:38:30.303781 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274877    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 12:38:30.303781 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274881    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 12:38:30.303781 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274885    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 12:38:30.303781 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274889    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 12:38:30.303781 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274893    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 12:38:30.303781 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274896    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 12:38:30.303781 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274901    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 12:38:30.303781 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274904    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 12:38:30.303781 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274910    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 12:38:30.303781 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274915    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 12:38:30.303781 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274919    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 12:38:30.303781 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274924    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 12:38:30.303781 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274928    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 12:38:30.303781 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274932    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 12:38:30.303781 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274937    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 12:38:30.307246 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274941    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 12:38:30.307246 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274945    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 12:38:30.307246 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274948    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 12:38:30.307246 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274952    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 12:38:30.307246 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274956    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 12:38:30.307246 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274960    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 12:38:30.307246 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274964    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 12:38:30.307246 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274968    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 12:38:30.307246 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274972    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 12:38:30.307246 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274976    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 12:38:30.307246 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274980    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 12:38:30.307246 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274987    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 12:38:30.307246 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274993    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 12:38:30.307246 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.274998    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 12:38:30.307246 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.275002    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 12:38:30.307246 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.275006    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 12:38:30.307246 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.275010    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 12:38:30.307246 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.275014    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 12:38:30.307246 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.275018    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 12:38:30.307246 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.275022    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 12:38:30.309974 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.275026    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 12:38:30.309974 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.275030    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 12:38:30.309974 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.275034    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 12:38:30.309974 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.275038    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 12:38:30.309974 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.275043    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 12:38:30.309974 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.275048    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 12:38:30.309974 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.275053    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 12:38:30.309974 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.275058    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 12:38:30.309974 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.275061    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 12:38:30.309974 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.275067    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 12:38:30.309974 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.275072    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 12:38:30.309974 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.275077    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 12:38:30.309974 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.275082    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 12:38:30.309974 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.275086    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 12:38:30.309974 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.275091    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 12:38:30.309974 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.275095    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 12:38:30.309974 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.275099    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 12:38:30.309974 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.275103    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 12:38:30.309974 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.275108    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 12:38:30.313905 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.275112    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 12:38:30.313905 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.275117    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 12:38:30.313905 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.275121    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 12:38:30.313905 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.275125    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 12:38:30.313905 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.275129    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 12:38:30.313905 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.275133    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 12:38:30.313905 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.275137    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 12:38:30.313905 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.275141    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 12:38:30.313905 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.275145    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 12:38:30.313905 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.275149    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 12:38:30.313905 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.275153    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 12:38:30.313905 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.275174    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 12:38:30.313905 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.275179    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 12:38:30.313905 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.275184    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 12:38:30.313905 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276238    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 12:38:30.313905 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276248    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 12:38:30.313905 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276253    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 12:38:30.313905 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276257    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 12:38:30.313905 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276262    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 12:38:30.313905 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276271    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 12:38:30.325592 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276278    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 12:38:30.325592 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276282    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 12:38:30.325592 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276287    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 12:38:30.325592 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276291    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 12:38:30.325592 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276295    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 12:38:30.325592 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276299    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 12:38:30.325592 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276304    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 12:38:30.325592 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276308    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 12:38:30.325592 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276312    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 12:38:30.325592 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276316    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 12:38:30.325592 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276320    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 12:38:30.325592 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276324    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 12:38:30.325592 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276328    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 12:38:30.325592 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276332    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 12:38:30.325592 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276337    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 12:38:30.325592 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276341    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 12:38:30.325592 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276346    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 12:38:30.325592 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276350    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 12:38:30.325592 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276354    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 12:38:30.472219 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276358    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 12:38:30.472219 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276361    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 12:38:30.472219 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276366    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 12:38:30.472219 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276370    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 12:38:30.472219 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276374    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 12:38:30.472219 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276378    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 12:38:30.472219 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276383    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 12:38:30.472219 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276387    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 12:38:30.472219 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276393    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 12:38:30.472219 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276398    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 12:38:30.472219 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276403    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 12:38:30.472219 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276408    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 12:38:30.472219 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276412    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 12:38:30.472219 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276417    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 12:38:30.472219 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276421    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 12:38:30.472219 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276426    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 12:38:30.472219 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276430    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 12:38:30.472219 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276434    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 12:38:30.472219 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276437    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 12:38:30.475493 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276441    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 12:38:30.475493 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276446    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 12:38:30.475493 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276449    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 12:38:30.475493 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276454    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 12:38:30.475493 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276458    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 12:38:30.475493 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276462    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 12:38:30.475493 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276465    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 12:38:30.475493 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276469    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 12:38:30.475493 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276473    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 12:38:30.475493 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276477    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 12:38:30.475493 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276481    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 12:38:30.475493 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276485    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 12:38:30.475493 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276490    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 12:38:30.475493 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276496    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 12:38:30.475493 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276502    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 12:38:30.475493 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276507    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 12:38:30.475493 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276511    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 12:38:30.475493 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276516    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 12:38:30.475493 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276520    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 12:38:30.480385 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276525    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 12:38:30.480385 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276530    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 12:38:30.480385 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276533    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 12:38:30.480385 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276538    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 12:38:30.480385 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276542    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 12:38:30.480385 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276546    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 12:38:30.480385 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276550    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 12:38:30.480385 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276554    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 12:38:30.480385 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276559    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 12:38:30.480385 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276563    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 12:38:30.480385 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276566    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 12:38:30.480385 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276570    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 12:38:30.480385 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276574    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 12:38:30.480385 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276578    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 12:38:30.480385 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276582    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 12:38:30.480385 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276586    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 12:38:30.480385 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276590    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 12:38:30.480385 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276594    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 12:38:30.480385 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276597    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 12:38:30.480385 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276603    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 12:38:30.480385 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276608    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 12:38:30.491289 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276612    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 12:38:30.491289 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276616    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 12:38:30.491289 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276620    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 12:38:30.491289 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276624    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 12:38:30.491289 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276627    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 12:38:30.491289 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276631    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 12:38:30.491289 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276635    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 12:38:30.491289 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276639    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 12:38:30.491289 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276643    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 12:38:30.491289 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276648    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 12:38:30.491289 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276652    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 12:38:30.491289 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276656    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 12:38:30.491289 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276660    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 12:38:30.491289 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276664    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 12:38:30.491289 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.276668    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 12:38:30.491289 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277252    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 12:38:30.491289 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277260    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 12:38:30.491289 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277265    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 12:38:30.491289 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277269    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 12:38:30.491289 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277275    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 12:38:30.490619 ip-10-0-143-88 systemd[1]: Started Kubernetes Kubelet.
Jul 02 12:38:30.493286 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277279    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 12:38:30.493286 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277284    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 12:38:30.493286 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277289    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 12:38:30.493286 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277293    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 12:38:30.493286 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277297    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 12:38:30.493286 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277301    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 12:38:30.493286 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277306    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 12:38:30.493286 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277310    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 12:38:30.493286 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277314    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 12:38:30.493286 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277318    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 12:38:30.493286 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277322    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 12:38:30.493286 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277326    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 12:38:30.493286 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277333    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 12:38:30.493286 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277339    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 12:38:30.493286 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277345    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 12:38:30.493286 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277349    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 12:38:30.493286 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277354    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 12:38:30.493286 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277358    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 12:38:30.493286 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277362    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 12:38:30.493286 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277366    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 12:38:30.496753 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277370    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 12:38:30.496753 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277374    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 12:38:30.496753 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277379    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 12:38:30.496753 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277383    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 12:38:30.496753 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277389    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 12:38:30.496753 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277393    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 12:38:30.496753 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277398    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 12:38:30.496753 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277402    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 12:38:30.496753 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277406    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 12:38:30.496753 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277410    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 12:38:30.496753 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277414    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 12:38:30.496753 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277419    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 12:38:30.496753 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277424    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 12:38:30.496753 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277429    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 12:38:30.496753 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277433    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 12:38:30.496753 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277438    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 12:38:30.496753 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277443    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 12:38:30.496753 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277446    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 12:38:30.496753 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277448    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 12:38:30.497611 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277451    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 12:38:30.497611 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277454    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 12:38:30.497611 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277456    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 12:38:30.497611 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277459    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 12:38:30.497611 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277462    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 12:38:30.497611 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277466    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 12:38:30.497611 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277468    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 12:38:30.497611 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277471    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 12:38:30.497611 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277474    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 12:38:30.497611 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277476    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 12:38:30.497611 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277480    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 12:38:30.497611 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277483    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 12:38:30.497611 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277486    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 12:38:30.497611 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277488    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 12:38:30.497611 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277491    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 12:38:30.497611 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277494    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 12:38:30.497611 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277496    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 12:38:30.497611 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277499    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 12:38:30.497611 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277502    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 12:38:30.497611 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277505    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 12:38:30.499228 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277507    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 12:38:30.499228 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277510    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 12:38:30.499228 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277515    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 12:38:30.499228 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277519    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 12:38:30.499228 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277522    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 12:38:30.499228 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277524    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 12:38:30.499228 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277528    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 12:38:30.499228 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277530    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 12:38:30.499228 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277533    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 12:38:30.499228 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277535    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 12:38:30.499228 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277538    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 12:38:30.499228 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277540    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 12:38:30.499228 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277543    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 12:38:30.499228 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277545    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 12:38:30.499228 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277547    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 12:38:30.499228 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277550    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 12:38:30.499228 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277552    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 12:38:30.499228 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277556    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 12:38:30.499228 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277559    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 12:38:30.500030 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277561    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 12:38:30.500030 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277564    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 12:38:30.500030 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277566    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 12:38:30.500030 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277569    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 12:38:30.500030 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277574    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 12:38:30.500030 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277576    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 12:38:30.500030 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277579    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 12:38:30.500030 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277582    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 12:38:30.500030 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277584    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 12:38:30.500030 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277587    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 12:38:30.500030 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277589    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 12:38:30.500030 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277591    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 12:38:30.500030 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277594    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 12:38:30.500030 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277596    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 12:38:30.500030 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277599    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 12:38:30.500030 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.277602    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 12:38:30.500030 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277689    2576 flags.go:64] FLAG: --address="0.0.0.0"
Jul 02 12:38:30.500030 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277700    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 02 12:38:30.500030 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277708    2576 flags.go:64] FLAG: --anonymous-auth="true"
Jul 02 12:38:30.500030 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277713    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 02 12:38:30.500030 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277718    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 02 12:38:30.501643 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277721    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 02 12:38:30.501643 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277725    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 02 12:38:30.501643 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277730    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 02 12:38:30.501643 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277733    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 02 12:38:30.501643 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277736    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 02 12:38:30.501643 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277739    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 02 12:38:30.501643 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277743    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 02 12:38:30.501643 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277746    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 02 12:38:30.501643 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277748    2576 flags.go:64] FLAG: --cgroup-root=""
Jul 02 12:38:30.501643 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277754    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 02 12:38:30.501643 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277757    2576 flags.go:64] FLAG: --client-ca-file=""
Jul 02 12:38:30.501643 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277760    2576 flags.go:64] FLAG: --cloud-provider="external"
Jul 02 12:38:30.501643 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277763    2576 flags.go:64] FLAG: --cluster-dns="[]"
Jul 02 12:38:30.501643 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277767    2576 flags.go:64] FLAG: --cluster-domain=""
Jul 02 12:38:30.501643 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277770    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 02 12:38:30.501643 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277774    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 02 12:38:30.501643 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277778    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 02 12:38:30.501643 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277781    2576 flags.go:64] FLAG: --container-log-max-files="5"
Jul 02 12:38:30.501643 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277785    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 02 12:38:30.501643 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277788    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 02 12:38:30.501643 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277791    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 02 12:38:30.501643 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277795    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 02 12:38:30.501643 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277798    2576 flags.go:64] FLAG: --contention-profiling="false"
Jul 02 12:38:30.501643 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277801    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 02 12:38:30.502458 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277804    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 02 12:38:30.502458 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277807    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 02 12:38:30.502458 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277810    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 02 12:38:30.502458 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277814    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 02 12:38:30.502458 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277817    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 02 12:38:30.502458 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277819    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 02 12:38:30.502458 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277822    2576 flags.go:64] FLAG: --enable-load-reader="false"
Jul 02 12:38:30.502458 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277825    2576 flags.go:64] FLAG: --enable-server="true"
Jul 02 12:38:30.502458 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277828    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 02 12:38:30.502458 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277833    2576 flags.go:64] FLAG: --event-burst="100"
Jul 02 12:38:30.502458 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277836    2576 flags.go:64] FLAG: --event-qps="50"
Jul 02 12:38:30.502458 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277839    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 02 12:38:30.502458 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277842    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 02 12:38:30.502458 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277845    2576 flags.go:64] FLAG: --eviction-hard=""
Jul 02 12:38:30.502458 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277849    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 02 12:38:30.502458 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277851    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 02 12:38:30.502458 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277855    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 02 12:38:30.502458 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277859    2576 flags.go:64] FLAG: --eviction-soft=""
Jul 02 12:38:30.502458 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277861    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 02 12:38:30.502458 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277864    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 02 12:38:30.502458 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277867    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 02 12:38:30.502458 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277870    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 02 12:38:30.502458 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277873    2576 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 02 12:38:30.502458 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277877    2576 flags.go:64] FLAG: --fail-swap-on="true"
Jul 02 12:38:30.502458 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277879    2576 flags.go:64] FLAG: --feature-gates=""
Jul 02 12:38:30.504091 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277883    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 02 12:38:30.504091 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277886    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 02 12:38:30.504091 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277890    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 02 12:38:30.504091 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277893    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 02 12:38:30.504091 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277896    2576 flags.go:64] FLAG: --healthz-port="10248"
Jul 02 12:38:30.504091 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277899    2576 flags.go:64] FLAG: --help="false"
Jul 02 12:38:30.504091 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277902    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-143-88.ec2.internal"
Jul 02 12:38:30.504091 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277905    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 02 12:38:30.504091 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277908    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 02 12:38:30.504091 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277911    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 02 12:38:30.504091 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277914    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 02 12:38:30.504091 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277918    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 02 12:38:30.504091 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277921    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 02 12:38:30.504091 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277924    2576 flags.go:64] FLAG: --image-service-endpoint=""
Jul 02 12:38:30.504091 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277926    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 02 12:38:30.504091 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277929    2576 flags.go:64] FLAG: --kube-api-burst="100"
Jul 02 12:38:30.504091 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277933    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 02 12:38:30.504091 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277936    2576 flags.go:64] FLAG: --kube-api-qps="50"
Jul 02 12:38:30.504091 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277939    2576 flags.go:64] FLAG: --kube-reserved=""
Jul 02 12:38:30.504091 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277942    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 02 12:38:30.504091 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277945    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 02 12:38:30.504091 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277948    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 02 12:38:30.504091 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277951    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 02 12:38:30.504091 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277954    2576 flags.go:64] FLAG: --lock-file=""
Jul 02 12:38:30.504893 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277958    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 02 12:38:30.504893 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277961    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 02 12:38:30.504893 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277964    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 02 12:38:30.504893 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277969    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 02 12:38:30.504893 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277972    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 02 12:38:30.504893 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277975    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 02 12:38:30.504893 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277980    2576 flags.go:64] FLAG: --logging-format="text"
Jul 02 12:38:30.504893 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277983    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 02 12:38:30.504893 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277986    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 02 12:38:30.504893 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277989    2576 flags.go:64] FLAG: --manifest-url=""
Jul 02 12:38:30.504893 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277992    2576 flags.go:64] FLAG: --manifest-url-header=""
Jul 02 12:38:30.504893 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277996    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 02 12:38:30.504893 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.277999    2576 flags.go:64] FLAG: --max-open-files="1000000"
Jul 02 12:38:30.504893 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278003    2576 flags.go:64] FLAG: --max-pods="110"
Jul 02 12:38:30.504893 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278006    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 02 12:38:30.504893 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278009    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 02 12:38:30.504893 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278011    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 02 12:38:30.504893 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278014    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 02 12:38:30.504893 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278017    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 02 12:38:30.504893 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278020    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 02 12:38:30.504893 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278023    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 02 12:38:30.504893 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278030    2576 flags.go:64] FLAG: --node-status-max-images="50"
Jul 02 12:38:30.504893 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278033    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 02 12:38:30.504893 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278036    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 02 12:38:30.504893 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278039    2576 flags.go:64] FLAG: --pod-cidr=""
Jul 02 12:38:30.507780 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278050    2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a2632cd31d78133be7b5fb6999469b2acf9b98d077c7f6ae5d0d94083d76b04"
Jul 02 12:38:30.507780 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278056    2576 flags.go:64] FLAG: --pod-manifest-path=""
Jul 02 12:38:30.507780 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278059    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 02 12:38:30.507780 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278062    2576 flags.go:64] FLAG: --pods-per-core="0"
Jul 02 12:38:30.507780 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278065    2576 flags.go:64] FLAG: --port="10250"
Jul 02 12:38:30.507780 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278068    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 02 12:38:30.507780 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278070    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01fdac2db5c27117c"
Jul 02 12:38:30.507780 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278074    2576 flags.go:64] FLAG: --qos-reserved=""
Jul 02 12:38:30.507780 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278079    2576 flags.go:64] FLAG: --read-only-port="10255"
Jul 02 12:38:30.507780 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278082    2576 flags.go:64] FLAG: --register-node="true"
Jul 02 12:38:30.507780 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278085    2576 flags.go:64] FLAG: --register-with-taints=""
Jul 02 12:38:30.507780 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278089    2576 flags.go:64] FLAG: --registry-burst="10"
Jul 02 12:38:30.507780 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278092    2576 flags.go:64] FLAG: --registry-qps="5"
Jul 02 12:38:30.507780 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278097    2576 flags.go:64] FLAG: --reserved-cpus=""
Jul 02 12:38:30.507780 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278100    2576 flags.go:64] FLAG: --reserved-memory=""
Jul 02 12:38:30.507780 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278103    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 02 12:38:30.507780 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278107    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 02 12:38:30.507780 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278109    2576 flags.go:64] FLAG: --rotate-certificates="false"
Jul 02 12:38:30.507780 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278112    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 02 12:38:30.507780 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278116    2576 flags.go:64] FLAG: --runonce="false"
Jul 02 12:38:30.507780 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278119    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 02 12:38:30.507780 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278122    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 02 12:38:30.507780 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278125    2576 flags.go:64] FLAG: --seccomp-default="false"
Jul 02 12:38:30.507780 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278128    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 02 12:38:30.507780 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278131    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 02 12:38:30.508589 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278134    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 02 12:38:30.508589 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278137    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 02 12:38:30.508589 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278140    2576 flags.go:64] FLAG: --storage-driver-password="root"
Jul 02 12:38:30.508589 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278143    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 02 12:38:30.508589 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278146    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 02 12:38:30.508589 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278148    2576 flags.go:64] FLAG: --storage-driver-user="root"
Jul 02 12:38:30.508589 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278152    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 02 12:38:30.508589 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278154    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 02 12:38:30.508589 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278200    2576 flags.go:64] FLAG: --system-cgroups=""
Jul 02 12:38:30.508589 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278203    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 02 12:38:30.508589 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278209    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 02 12:38:30.508589 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278212    2576 flags.go:64] FLAG: --tls-cert-file=""
Jul 02 12:38:30.508589 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278215    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 02 12:38:30.508589 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278220    2576 flags.go:64] FLAG: --tls-min-version=""
Jul 02 12:38:30.508589 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278222    2576 flags.go:64] FLAG: --tls-private-key-file=""
Jul 02 12:38:30.508589 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278226    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 02 12:38:30.508589 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278229    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 02 12:38:30.508589 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278232    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 02 12:38:30.508589 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278235    2576 flags.go:64] FLAG: --v="2"
Jul 02 12:38:30.508589 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278240    2576 flags.go:64] FLAG: --version="false"
Jul 02 12:38:30.508589 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278245    2576 flags.go:64] FLAG: --vmodule=""
Jul 02 12:38:30.508589 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278249    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 02 12:38:30.508589 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278252    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 02 12:38:30.508589 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278413    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 12:38:30.508589 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278419    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 12:38:30.509401 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278423    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 12:38:30.509401 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278427    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 12:38:30.509401 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278431    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 12:38:30.509401 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278436    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 12:38:30.509401 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278440    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 12:38:30.509401 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278444    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 12:38:30.509401 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278448    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 12:38:30.509401 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278452    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 12:38:30.509401 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278456    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 12:38:30.509401 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278459    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 12:38:30.509401 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278463    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 12:38:30.509401 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278468    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 12:38:30.509401 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278471    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 12:38:30.509401 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278475    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 12:38:30.509401 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278480    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 12:38:30.509401 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278483    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 12:38:30.509401 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278489    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 12:38:30.509401 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278493    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 12:38:30.509401 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278497    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 12:38:30.509401 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278501    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 12:38:30.510210 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278505    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 12:38:30.510210 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278509    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 12:38:30.510210 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278514    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 12:38:30.510210 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278519    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 12:38:30.510210 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278523    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 12:38:30.510210 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278527    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 12:38:30.510210 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278531    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 12:38:30.510210 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278538    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 12:38:30.510210 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278542    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 12:38:30.510210 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278547    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 12:38:30.510210 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278551    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 12:38:30.510210 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278554    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 12:38:30.510210 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278559    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 12:38:30.510210 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278564    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 12:38:30.510210 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278568    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 12:38:30.510210 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278572    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 12:38:30.510210 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278576    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 12:38:30.510210 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278580    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 12:38:30.510210 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278584    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 12:38:30.510210 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278588    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 12:38:30.511444 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278592    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 12:38:30.511444 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278596    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 12:38:30.511444 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278600    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 12:38:30.511444 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278605    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 12:38:30.511444 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278609    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 12:38:30.511444 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278613    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 12:38:30.511444 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278617    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 12:38:30.511444 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278621    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 12:38:30.511444 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278626    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 12:38:30.511444 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278631    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 12:38:30.511444 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278635    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 12:38:30.511444 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278639    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 12:38:30.511444 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278645    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 12:38:30.511444 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278650    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 12:38:30.511444 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278658    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 12:38:30.511444 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278662    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 12:38:30.511444 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278667    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 12:38:30.511444 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278671    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 12:38:30.511444 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278675    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 12:38:30.511444 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278681    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 12:38:30.512255 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278685    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 12:38:30.512255 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278689    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 12:38:30.512255 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278693    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 12:38:30.512255 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278697    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 12:38:30.512255 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278701    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 12:38:30.512255 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278706    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 12:38:30.512255 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278710    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 12:38:30.512255 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278714    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 12:38:30.512255 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278718    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 12:38:30.512255 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278722    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 12:38:30.512255 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278726    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 12:38:30.512255 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278730    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 12:38:30.512255 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278734    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 12:38:30.512255 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278738    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 12:38:30.512255 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278742    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 12:38:30.512255 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278746    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 12:38:30.512255 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278750    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 12:38:30.512255 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278754    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 12:38:30.512255 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278758    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 12:38:30.512255 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278762    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 12:38:30.513862 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278766    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 12:38:30.513862 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278770    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 12:38:30.513862 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278774    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 12:38:30.513862 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278778    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 12:38:30.513862 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278782    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 12:38:30.513862 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278786    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 12:38:30.513862 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278792    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 12:38:30.513862 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278797    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 12:38:30.513862 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278801    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 12:38:30.513862 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278807    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 12:38:30.513862 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278812    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 12:38:30.513862 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278819    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 12:38:30.513862 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278823    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 12:38:30.513862 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278829    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 12:38:30.513862 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278834    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 12:38:30.513862 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278839    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 12:38:30.513862 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.278844    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 12:38:30.514667 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.278851    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 12:38:30.514667 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.286382    2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 02 12:38:30.514667 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.286397    2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 02 12:38:30.514667 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.287333    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 02 12:38:30.514667 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.287343    2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 02 12:38:30.514667 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287407    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 12:38:30.514667 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287412    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 12:38:30.514667 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287415    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 12:38:30.514667 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287418    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 12:38:30.514667 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287421    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 12:38:30.514667 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287424    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 12:38:30.514667 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287427    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 12:38:30.514667 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287430    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 12:38:30.514667 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287433    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 12:38:30.514667 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287435    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 12:38:30.515872 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287438    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 12:38:30.515872 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287441    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 12:38:30.515872 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287444    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 12:38:30.515872 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287447    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 12:38:30.515872 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287449    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 12:38:30.515872 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287452    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 12:38:30.515872 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287454    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 12:38:30.515872 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287457    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 12:38:30.515872 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287460    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 12:38:30.515872 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287462    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 12:38:30.515872 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287465    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 12:38:30.515872 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287467    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 12:38:30.515872 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287474    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 12:38:30.515872 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287478    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 12:38:30.515872 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287481    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 12:38:30.515872 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287483    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 12:38:30.515872 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287486    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 12:38:30.515872 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287488    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 12:38:30.515872 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287491    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 12:38:30.515872 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287494    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 12:38:30.516675 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287497    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 12:38:30.516675 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287500    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 12:38:30.516675 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287502    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 12:38:30.516675 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287505    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 12:38:30.516675 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287508    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 12:38:30.516675 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287512    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 12:38:30.516675 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287516    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 12:38:30.516675 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287520    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 12:38:30.516675 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287523    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 12:38:30.516675 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287527    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 12:38:30.516675 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287530    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 12:38:30.516675 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287533    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 12:38:30.516675 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287536    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 12:38:30.516675 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287539    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 12:38:30.516675 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287542    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 12:38:30.516675 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287554    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 12:38:30.516675 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287557    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 12:38:30.516675 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287559    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 12:38:30.516675 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287562    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 12:38:30.517920 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287565    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 12:38:30.517920 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287569    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 12:38:30.517920 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287572    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 12:38:30.517920 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287575    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 12:38:30.517920 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287578    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 12:38:30.517920 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287580    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 12:38:30.517920 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287583    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 12:38:30.517920 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287585    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 12:38:30.517920 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287591    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 12:38:30.517920 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287594    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 12:38:30.517920 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287596    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 12:38:30.517920 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287601    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 12:38:30.517920 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287603    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 12:38:30.517920 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287606    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 12:38:30.517920 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287609    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 12:38:30.517920 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287611    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 12:38:30.517920 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287614    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 12:38:30.517920 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287616    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 12:38:30.517920 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287619    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 12:38:30.518725 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287622    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 12:38:30.518725 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287624    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 12:38:30.518725 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287627    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 12:38:30.518725 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287629    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 12:38:30.518725 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287632    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 12:38:30.518725 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287634    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 12:38:30.518725 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287637    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 12:38:30.518725 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287640    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 12:38:30.518725 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287642    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 12:38:30.518725 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287645    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 12:38:30.518725 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287653    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 12:38:30.518725 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287656    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 12:38:30.518725 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287659    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 12:38:30.518725 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287661    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 12:38:30.518725 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287664    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 12:38:30.518725 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287666    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 12:38:30.518725 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287669    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 12:38:30.518725 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287677    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 12:38:30.518725 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287679    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 12:38:30.518725 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287682    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 12:38:30.520362 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287684    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 12:38:30.520362 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287688    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 12:38:30.520362 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287690    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 12:38:30.520362 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287693    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 12:38:30.520362 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287697    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 12:38:30.520362 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287699    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 12:38:30.520362 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287702    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 12:38:30.520362 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287705    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 12:38:30.520362 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287707    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 12:38:30.520362 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287710    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 12:38:30.520362 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287713    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 12:38:30.520362 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.287718    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 12:38:30.520362 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287886    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 12:38:30.520362 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287891    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 12:38:30.520362 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287894    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 12:38:30.520362 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287897    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 12:38:30.521172 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287900    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 12:38:30.521172 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287903    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 12:38:30.521172 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287905    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 12:38:30.521172 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287908    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 12:38:30.521172 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287910    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 12:38:30.521172 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287913    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 12:38:30.521172 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287915    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 12:38:30.521172 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287923    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 12:38:30.521172 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287926    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 12:38:30.521172 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287929    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 12:38:30.521172 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287931    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 12:38:30.521172 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287934    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 12:38:30.521172 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287936    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 12:38:30.521172 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287939    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 12:38:30.521172 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287941    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 12:38:30.521172 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287944    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 12:38:30.521172 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287946    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 12:38:30.521172 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287950    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 12:38:30.521172 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287953    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 12:38:30.521172 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287955    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 12:38:30.522396 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287958    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 12:38:30.522396 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287963    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 12:38:30.522396 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287966    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 12:38:30.522396 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287969    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 12:38:30.522396 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287971    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 12:38:30.522396 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287974    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 12:38:30.522396 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287976    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 12:38:30.522396 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287979    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 12:38:30.522396 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287981    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 12:38:30.522396 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287984    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 12:38:30.522396 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287987    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 12:38:30.522396 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287989    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 12:38:30.522396 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287992    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 12:38:30.522396 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287995    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 12:38:30.522396 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.287997    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 12:38:30.522396 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288000    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 12:38:30.522396 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288002    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 12:38:30.522396 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288005    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 12:38:30.522396 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288007    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 12:38:30.522396 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288009    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 12:38:30.523201 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288017    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 12:38:30.523201 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288020    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 12:38:30.523201 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288023    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 12:38:30.523201 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288025    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 12:38:30.523201 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288028    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 12:38:30.523201 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288030    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 12:38:30.523201 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288033    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 12:38:30.523201 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288035    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 12:38:30.523201 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288038    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 12:38:30.523201 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288042    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 12:38:30.523201 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288045    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 12:38:30.523201 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288047    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 12:38:30.523201 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288049    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 12:38:30.523201 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288053    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 12:38:30.523201 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288055    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 12:38:30.523201 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288058    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 12:38:30.523201 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288060    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 12:38:30.523201 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288063    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 12:38:30.523201 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288065    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 12:38:30.523201 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288068    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 12:38:30.523996 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288070    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 12:38:30.523996 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288073    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 12:38:30.523996 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288077    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 12:38:30.523996 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288080    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 12:38:30.523996 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288083    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 12:38:30.523996 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288086    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 12:38:30.523996 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288089    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 12:38:30.523996 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288091    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 12:38:30.523996 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288094    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 12:38:30.523996 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288096    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 12:38:30.523996 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288099    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 12:38:30.523996 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288101    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 12:38:30.523996 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288104    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 12:38:30.523996 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288112    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 12:38:30.523996 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288115    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 12:38:30.523996 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288117    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 12:38:30.523996 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288120    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 12:38:30.523996 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288122    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 12:38:30.523996 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288126    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 12:38:30.524808 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288129    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 12:38:30.524808 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288132    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 12:38:30.524808 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288135    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 12:38:30.524808 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288138    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 12:38:30.524808 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288140    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 12:38:30.524808 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288143    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 12:38:30.524808 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288147    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 12:38:30.524808 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288151    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 12:38:30.524808 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288153    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 12:38:30.524808 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288156    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 12:38:30.524808 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288175    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 12:38:30.524808 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288179    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 12:38:30.524808 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288182    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 12:38:30.524808 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288185    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 12:38:30.524808 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288187    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 12:38:30.524808 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:30.288190    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 12:38:30.525628 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.288195    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 12:38:30.525628 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.288305    2576 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 02 12:38:30.525628 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.290681    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 02 12:38:30.525628 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.292339    2576 server.go:1019] "Starting client certificate rotation"
Jul 02 12:38:30.525628 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.292478    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 02 12:38:30.525628 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.292526    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 02 12:38:30.525628 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.296144    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 02 12:38:30.525628 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.300958    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 02 12:38:30.525628 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.313248    2576 log.go:25] "Validated CRI v1 runtime API"
Jul 02 12:38:30.525628 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.316072    2576 log.go:25] "Validated CRI v1 image API"
Jul 02 12:38:30.525628 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.324497    2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 02 12:38:30.525628 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.327498    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 02 12:38:30.526034 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.330235    2576 fs.go:135] Filesystem UUIDs: map[001dc95c-19a8-42d0-8fac-8153e094ce82:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 dcb20b4f-1621-4a23-a45e-3cf882857f93:/dev/nvme0n1p3]
Jul 02 12:38:30.526034 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.330261    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 02 12:38:30.527262 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.335450    2576 manager.go:216] Machine: {Timestamp:2026-07-02 12:38:30.333979372 +0000 UTC m=+0.394194128 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099382 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fe11898b71fd088bbb9c21214a5b4 SystemUUID:ec2fe118-98b7-1fd0-88bb-b9c21214a5b4 BootID:0e0d247f-d013-41e5-bd64-0e5c81832c17 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:14:dc:2c:02:b1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:14:dc:2c:02:b1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2e:2f:51:dd:de:07 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 02 12:38:30.527262 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.335557    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 02 12:38:30.527262 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.335635    2576 manager.go:232] Version: {KernelVersion:5.14.0-570.124.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260623-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 02 12:38:30.527262 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.337803    2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 02 12:38:30.527262 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.337832    2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-88.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 02 12:38:30.527262 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.338713    2576 topology_manager.go:138] "Creating topology manager with none policy"
Jul 02 12:38:30.527262 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.338726    2576 container_manager_linux.go:309] "Creating device plugin manager"
Jul 02 12:38:30.527262 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.338736    2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 02 12:38:30.527262 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.338755    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 02 12:38:30.527262 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.380086    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hgwq2"
Jul 02 12:38:30.527262 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.388313    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hgwq2"
Jul 02 12:38:30.527262 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.470137    2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 02 12:38:30.527262 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.477561    2576 state_mem.go:36] "Initialized new in-memory state store"
Jul 02 12:38:30.527262 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.477733    2576 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 02 12:38:30.527262 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.477763    2576 kubelet.go:486] "Attempting to sync node with API server"
Jul 02 12:38:30.527262 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.477778    2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 02 12:38:30.527262 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.477799    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 02 12:38:30.628844 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.477808    2576 kubelet.go:398] "Adding apiserver pod source"
Jul 02 12:38:30.628844 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.477821    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 02 12:38:30.628844 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.478281    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 02 12:38:30.628844 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.478297    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 02 12:38:30.628844 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.482603    2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 02 12:38:30.628844 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.484610    2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 02 12:38:30.628844 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.484639    2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 02 12:38:30.628844 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.485543    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 02 12:38:30.628844 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.485556    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 02 12:38:30.628844 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.485562    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 02 12:38:30.628844 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.485567    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 02 12:38:30.628844 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.485576    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 02 12:38:30.628844 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.485583    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 02 12:38:30.628844 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.485595    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 02 12:38:30.628844 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.485600    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 02 12:38:30.628844 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.485607    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 02 12:38:30.628844 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.485613    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 02 12:38:30.628844 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.485621    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 02 12:38:30.628844 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.485628    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 02 12:38:30.628844 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.486513    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 02 12:38:30.628844 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.486523    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 02 12:38:30.628844 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.489761    2576 server.go:1268] "Started kubelet"
Jul 02 12:38:30.628844 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.489925    2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 02 12:38:30.628844 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.489864    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 02 12:38:30.628844 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.490097    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 02 12:38:30.628844 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.490460    2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 12:38:30.628844 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.491977    2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 12:38:30.628844 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.492551    2576 server.go:310] "Adding debug handlers to kubelet server"
Jul 02 12:38:30.628844 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.492949    2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 02 12:38:30.628844 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.493319    2576 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-143-88.ec2.internal" not found
Jul 02 12:38:30.747560 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:30.702790    2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 02 12:38:32.478544 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:30.915255    2576 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-143-88.ec2.internal" not found
Jul 02 12:38:32.478544 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.425031    2576 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-143-88.ec2.internal" not found
Jul 02 12:38:32.707589 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.700311    2576 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-143-88.ec2.internal" not found
Jul 02 12:38:32.707589 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:32.700365    2576 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-143-88.ec2.internal" not found
Jul 02 12:38:32.743813 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.743786    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 02 12:38:32.744263 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.743972    2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 02 12:38:32.773920 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.743925    2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 02 12:38:32.774002 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.773927    2576 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-143-88.ec2.internal" not found
Jul 02 12:38:32.781512 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.781491    2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 02 12:38:32.781512 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.781500    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 02 12:38:32.781702 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:32.781677    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-88.ec2.internal\" not found"
Jul 02 12:38:32.782528 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.782508    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 02 12:38:32.782751 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.782729    2576 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 02 12:38:32.782751 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.782752    2576 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 02 12:38:32.782889 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.782847    2576 reconstruct.go:97] "Volume reconstruction finished"
Jul 02 12:38:32.782889 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.782856    2576 reconciler.go:29] "Reconciler: start to sync state"
Jul 02 12:38:32.783195 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.783178    2576 factory.go:55] Registering systemd factory
Jul 02 12:38:32.783278 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.783218    2576 factory.go:223] Registration of the systemd container factory successfully
Jul 02 12:38:32.783496 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.783477    2576 factory.go:153] Registering CRI-O factory
Jul 02 12:38:32.783496 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.783497    2576 factory.go:223] Registration of the crio container factory successfully
Jul 02 12:38:32.783639 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.783549    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 02 12:38:32.783639 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.783578    2576 factory.go:103] Registering Raw factory
Jul 02 12:38:32.783639 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.783590    2576 manager.go:1196] Started watching for new ooms in manager
Jul 02 12:38:32.784266 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.784248    2576 manager.go:349] Starting recovery of all containers
Jul 02 12:38:32.794891 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.794875    2576 manager.go:354] Recovery completed
Jul 02 12:38:32.795276 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.795253    2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 12:38:32.798503 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:32.798476    2576 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-143-88.ec2.internal\" not found" node="ip-10-0-143-88.ec2.internal"
Jul 02 12:38:32.798579 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.798529    2576 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-143-88.ec2.internal" not found
Jul 02 12:38:32.799302 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.799291    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:32.801214 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.801196    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:32.801293 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.801229    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:32.801293 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.801244    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:32.801802 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.801790    2576 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 02 12:38:32.801802 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.801801    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:32.801877 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.801803    2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 02 12:38:32.801877 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.801823    2576 state_mem.go:36] "Initialized new in-memory state store"
Jul 02 12:38:32.802738 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.802724    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:32.802779 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.802753    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:32.802779 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.802767    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:32.804090 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.804078    2576 policy_none.go:49] "None policy: Start"
Jul 02 12:38:32.804137 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.804094    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 02 12:38:32.804137 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.804105    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 02 12:38:32.806103 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.806089    2576 policy_none.go:47] "Start"
Jul 02 12:38:32.844241 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.844097    2576 manager.go:337] "Starting Device Plugin manager"
Jul 02 12:38:32.868428 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:32.844263    2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 02 12:38:32.868428 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.844275    2576 server.go:87] "Starting device plugin registration server"
Jul 02 12:38:32.868428 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.844524    2576 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 02 12:38:32.868428 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.844536    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 02 12:38:32.868428 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.844636    2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 02 12:38:32.868428 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.844759    2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 02 12:38:32.868428 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.844768    2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 02 12:38:32.868428 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:32.845284    2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 02 12:38:32.868428 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:32.845342    2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-88.ec2.internal\" not found"
Jul 02 12:38:32.868428 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.855321    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 02 12:38:32.868428 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.856435    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 02 12:38:32.868428 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.856467    2576 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 02 12:38:32.868428 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.856487    2576 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 02 12:38:32.868428 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:32.856518    2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 02 12:38:32.868428 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.857461    2576 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-143-88.ec2.internal" not found
Jul 02 12:38:32.868428 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.858451    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 12:38:32.930557 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.930489    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-01 12:33:31 +0000 UTC" deadline="2028-03-07 09:40:09.29350063 +0000 UTC"
Jul 02 12:38:32.930557 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.930550    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14733h1m36.362952621s"
Jul 02 12:38:32.931577 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.931566    2576 apiserver.go:52] "Watching apiserver"
Jul 02 12:38:32.935032 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.935017    2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 02 12:38:32.944949 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.944936    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:32.945940 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.945907    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:32.946013 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.945955    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:32.946013 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.945966    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:32.946013 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.945989    2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-88.ec2.internal"
Jul 02 12:38:32.955695 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.955681    2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-88.ec2.internal"
Jul 02 12:38:32.955738 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:32.955703    2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-88.ec2.internal\": node \"ip-10-0-143-88.ec2.internal\" not found"
Jul 02 12:38:32.956877 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.956862    2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-88.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-88.ec2.internal"]
Jul 02 12:38:32.956942 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.956933    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:32.957726 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.957711    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:32.957798 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.957736    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:32.957798 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.957751    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:32.960097 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.960081    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:32.960245 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.960228    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-88.ec2.internal"
Jul 02 12:38:32.960301 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.960287    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:32.962513 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.962496    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:32.962513 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.962509    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:32.962622 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.962524    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:32.962622 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.962533    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:32.962622 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.962535    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:32.962622 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.962550    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:32.964902 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.964886    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=[]
Jul 02 12:38:32.964995 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.964980    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-88.ec2.internal"
Jul 02 12:38:32.965029 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.965007    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:32.965657 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.965644    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:32.965723 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.965663    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:32.965723 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.965671    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:32.981445 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:32.981419    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-88.ec2.internal\" not found" node="ip-10-0-143-88.ec2.internal"
Jul 02 12:38:32.982808 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.982794    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 02 12:38:32.983573 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.983554    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6f05da72d6558d5d49cf0691bccf02a9-config\") pod \"kube-apiserver-proxy-ip-10-0-143-88.ec2.internal\" (UID: \"6f05da72d6558d5d49cf0691bccf02a9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-88.ec2.internal"
Jul 02 12:38:32.983651 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.983583    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f7c8733428fcf3eb08cd473039be930f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-88.ec2.internal\" (UID: \"f7c8733428fcf3eb08cd473039be930f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-88.ec2.internal"
Jul 02 12:38:32.983651 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:32.983612    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7c8733428fcf3eb08cd473039be930f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-88.ec2.internal\" (UID: \"f7c8733428fcf3eb08cd473039be930f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-88.ec2.internal"
Jul 02 12:38:32.984886 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:32.984869    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-88.ec2.internal\" not found" node="ip-10-0-143-88.ec2.internal"
Jul 02 12:38:33.008930 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:33.008895    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-88.ec2.internal\" not found"
Jul 02 12:38:33.035653 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.035625    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cltnv"]
Jul 02 12:38:33.035796 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.035749    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:33.036721 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.036695    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:33.036812 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.036728    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:33.036812 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.036739    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:33.039142 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.039128    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-sbnql"]
Jul 02 12:38:33.039265 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.039251    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cltnv"
Jul 02 12:38:33.039336 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.039288    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:33.039388 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.039289    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:33.040014 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.039997    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:33.040108 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.040025    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:33.040108 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.040035    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:33.040108 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.040098    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:33.040261 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.040118    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:33.040261 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.040131    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:33.042453 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.042437    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sbnql"
Jul 02 12:38:33.042519 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.042464    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:33.043094 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.043078    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:33.043148 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.043110    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:33.043148 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.043126    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:33.043225 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:33.043171    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sbnql" podUID="86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7"
Jul 02 12:38:33.052381 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.052366    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 02 12:38:33.052381 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.052374    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 02 12:38:33.052749 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.052734    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 02 12:38:33.053404 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.053387    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-whdfn\""
Jul 02 12:38:33.083821 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.083791    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3d27224c-42a6-4708-a061-1705cbdba994-socket-dir\") pod \"aws-ebs-csi-driver-node-cltnv\" (UID: \"3d27224c-42a6-4708-a061-1705cbdba994\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cltnv"
Jul 02 12:38:33.083821 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.083817    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3d27224c-42a6-4708-a061-1705cbdba994-registration-dir\") pod \"aws-ebs-csi-driver-node-cltnv\" (UID: \"3d27224c-42a6-4708-a061-1705cbdba994\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cltnv"
Jul 02 12:38:33.084013 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.083835    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fmm7\" (UniqueName: \"kubernetes.io/projected/3d27224c-42a6-4708-a061-1705cbdba994-kube-api-access-8fmm7\") pod \"aws-ebs-csi-driver-node-cltnv\" (UID: \"3d27224c-42a6-4708-a061-1705cbdba994\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cltnv"
Jul 02 12:38:33.084013 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.083850    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f76d4\" (UniqueName: \"kubernetes.io/projected/86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7-kube-api-access-f76d4\") pod \"network-metrics-daemon-sbnql\" (UID: \"86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7\") " pod="openshift-multus/network-metrics-daemon-sbnql"
Jul 02 12:38:33.084013 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.083865    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3d27224c-42a6-4708-a061-1705cbdba994-device-dir\") pod \"aws-ebs-csi-driver-node-cltnv\" (UID: \"3d27224c-42a6-4708-a061-1705cbdba994\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cltnv"
Jul 02 12:38:33.084013 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.083901    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3d27224c-42a6-4708-a061-1705cbdba994-etc-selinux\") pod \"aws-ebs-csi-driver-node-cltnv\" (UID: \"3d27224c-42a6-4708-a061-1705cbdba994\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cltnv"
Jul 02 12:38:33.084013 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.083947    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3d27224c-42a6-4708-a061-1705cbdba994-sys-fs\") pod \"aws-ebs-csi-driver-node-cltnv\" (UID: \"3d27224c-42a6-4708-a061-1705cbdba994\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cltnv"
Jul 02 12:38:33.084013 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.083971    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7-metrics-certs\") pod \"network-metrics-daemon-sbnql\" (UID: \"86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7\") " pod="openshift-multus/network-metrics-daemon-sbnql"
Jul 02 12:38:33.084013 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.083998    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6f05da72d6558d5d49cf0691bccf02a9-config\") pod \"kube-apiserver-proxy-ip-10-0-143-88.ec2.internal\" (UID: \"6f05da72d6558d5d49cf0691bccf02a9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-88.ec2.internal"
Jul 02 12:38:33.084241 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.084032    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f7c8733428fcf3eb08cd473039be930f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-88.ec2.internal\" (UID: \"f7c8733428fcf3eb08cd473039be930f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-88.ec2.internal"
Jul 02 12:38:33.084241 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.084072    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6f05da72d6558d5d49cf0691bccf02a9-config\") pod \"kube-apiserver-proxy-ip-10-0-143-88.ec2.internal\" (UID: \"6f05da72d6558d5d49cf0691bccf02a9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-88.ec2.internal"
Jul 02 12:38:33.084241 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.084096    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7c8733428fcf3eb08cd473039be930f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-88.ec2.internal\" (UID: \"f7c8733428fcf3eb08cd473039be930f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-88.ec2.internal"
Jul 02 12:38:33.084241 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.084122    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3d27224c-42a6-4708-a061-1705cbdba994-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cltnv\" (UID: \"3d27224c-42a6-4708-a061-1705cbdba994\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cltnv"
Jul 02 12:38:33.084241 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.084134    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7c8733428fcf3eb08cd473039be930f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-88.ec2.internal\" (UID: \"f7c8733428fcf3eb08cd473039be930f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-88.ec2.internal"
Jul 02 12:38:33.084241 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.084101    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f7c8733428fcf3eb08cd473039be930f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-88.ec2.internal\" (UID: \"f7c8733428fcf3eb08cd473039be930f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-88.ec2.internal"
Jul 02 12:38:33.106786 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.106758    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-tczr2"]
Jul 02 12:38:33.106958 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.106884    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:33.107802 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.107784    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:33.107876 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.107815    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:33.107876 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.107825    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:33.109935 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:33.109907    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-88.ec2.internal\" not found"
Jul 02 12:38:33.110242 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.110230    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-2c7ks"]
Jul 02 12:38:33.110363 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.110350    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tczr2"
Jul 02 12:38:33.110398 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.110385    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:33.110526 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.110514    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:33.111145 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.111127    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:33.111209 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.111160    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:33.111209 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.111171    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:33.111270 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.111134    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:33.111270 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.111237    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:33.111270 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:33.111230    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tczr2" podUID="ebc69c1d-a909-44d0-bcb1-691e39c5e754"
Jul 02 12:38:33.111270 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.111249    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:33.113587 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.113573    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.113633 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.113604    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:33.114246 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.114228    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:33.114369 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.114254    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:33.114369 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.114266    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:33.117137 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.117122    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 02 12:38:33.117214 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.117182    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-l9h4r\""
Jul 02 12:38:33.118153 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.118125    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 02 12:38:33.118669 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.118464    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 02 12:38:33.118669 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.118477    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 02 12:38:33.118669 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.118478    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 02 12:38:33.118669 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.118562    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 02 12:38:33.184303 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.184272    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3d27224c-42a6-4708-a061-1705cbdba994-sys-fs\") pod \"aws-ebs-csi-driver-node-cltnv\" (UID: \"3d27224c-42a6-4708-a061-1705cbdba994\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cltnv"
Jul 02 12:38:33.184464 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.184310    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-host-kubelet\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.184464 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.184333    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-run-ovn\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.184464 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.184352    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-log-socket\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.184464 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.184372    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-env-overrides\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.184464 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.184388    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3d27224c-42a6-4708-a061-1705cbdba994-sys-fs\") pod \"aws-ebs-csi-driver-node-cltnv\" (UID: \"3d27224c-42a6-4708-a061-1705cbdba994\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cltnv"
Jul 02 12:38:33.184464 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.184395    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3d27224c-42a6-4708-a061-1705cbdba994-registration-dir\") pod \"aws-ebs-csi-driver-node-cltnv\" (UID: \"3d27224c-42a6-4708-a061-1705cbdba994\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cltnv"
Jul 02 12:38:33.184464 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.184420    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f76d4\" (UniqueName: \"kubernetes.io/projected/86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7-kube-api-access-f76d4\") pod \"network-metrics-daemon-sbnql\" (UID: \"86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7\") " pod="openshift-multus/network-metrics-daemon-sbnql"
Jul 02 12:38:33.184464 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.184440    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-host-slash\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.184464 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.184461    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-var-lib-openvswitch\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.184845 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.184486    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkr4n\" (UniqueName: \"kubernetes.io/projected/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-kube-api-access-qkr4n\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.184845 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.184494    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3d27224c-42a6-4708-a061-1705cbdba994-registration-dir\") pod \"aws-ebs-csi-driver-node-cltnv\" (UID: \"3d27224c-42a6-4708-a061-1705cbdba994\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cltnv"
Jul 02 12:38:33.184845 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.184512    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:33.184845 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.184510    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7-metrics-certs\") pod \"network-metrics-daemon-sbnql\" (UID: \"86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7\") " pod="openshift-multus/network-metrics-daemon-sbnql"
Jul 02 12:38:33.184845 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.184549    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-host-run-netns\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.184845 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.184582    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-run-openvswitch\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.184845 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.184604    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-host-cni-bin\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.184845 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.184629    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.184845 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.184657    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-systemd-units\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.184845 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.184672    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:33.184845 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.184678    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-etc-openvswitch\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.184845 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.184727    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-host-run-ovn-kubernetes\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.184845 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.184757    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3d27224c-42a6-4708-a061-1705cbdba994-device-dir\") pod \"aws-ebs-csi-driver-node-cltnv\" (UID: \"3d27224c-42a6-4708-a061-1705cbdba994\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cltnv"
Jul 02 12:38:33.184845 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.184795    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3d27224c-42a6-4708-a061-1705cbdba994-etc-selinux\") pod \"aws-ebs-csi-driver-node-cltnv\" (UID: \"3d27224c-42a6-4708-a061-1705cbdba994\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cltnv"
Jul 02 12:38:33.184845 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.184815    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-node-log\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.184845 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.184834    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3d27224c-42a6-4708-a061-1705cbdba994-device-dir\") pod \"aws-ebs-csi-driver-node-cltnv\" (UID: \"3d27224c-42a6-4708-a061-1705cbdba994\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cltnv"
Jul 02 12:38:33.184845 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.184841    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3d27224c-42a6-4708-a061-1705cbdba994-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cltnv\" (UID: \"3d27224c-42a6-4708-a061-1705cbdba994\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cltnv"
Jul 02 12:38:33.185587 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.184860    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fmm7\" (UniqueName: \"kubernetes.io/projected/3d27224c-42a6-4708-a061-1705cbdba994-kube-api-access-8fmm7\") pod \"aws-ebs-csi-driver-node-cltnv\" (UID: \"3d27224c-42a6-4708-a061-1705cbdba994\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cltnv"
Jul 02 12:38:33.185587 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.184899    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-host-cni-netd\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.185587 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.184942    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-ovnkube-config\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.185587 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.184966    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-ovn-node-metrics-cert\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.185587 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.184976    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3d27224c-42a6-4708-a061-1705cbdba994-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cltnv\" (UID: \"3d27224c-42a6-4708-a061-1705cbdba994\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cltnv"
Jul 02 12:38:33.185587 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.184989    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-ovnkube-script-lib\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.185587 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.184997    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3d27224c-42a6-4708-a061-1705cbdba994-etc-selinux\") pod \"aws-ebs-csi-driver-node-cltnv\" (UID: \"3d27224c-42a6-4708-a061-1705cbdba994\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cltnv"
Jul 02 12:38:33.185587 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.185024    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:33.185587 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.185016    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3d27224c-42a6-4708-a061-1705cbdba994-socket-dir\") pod \"aws-ebs-csi-driver-node-cltnv\" (UID: \"3d27224c-42a6-4708-a061-1705cbdba994\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cltnv"
Jul 02 12:38:33.185587 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.185066    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zvwz\" (UniqueName: \"kubernetes.io/projected/ebc69c1d-a909-44d0-bcb1-691e39c5e754-kube-api-access-2zvwz\") pod \"network-check-target-tczr2\" (UID: \"ebc69c1d-a909-44d0-bcb1-691e39c5e754\") " pod="openshift-network-diagnostics/network-check-target-tczr2"
Jul 02 12:38:33.185587 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.185088    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-run-systemd\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.185587 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.185225    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3d27224c-42a6-4708-a061-1705cbdba994-socket-dir\") pod \"aws-ebs-csi-driver-node-cltnv\" (UID: \"3d27224c-42a6-4708-a061-1705cbdba994\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cltnv"
Jul 02 12:38:33.185587 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.185346    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-4bj55"]
Jul 02 12:38:33.185587 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.185509    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:33.185587 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.185545    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:33.185587 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.185571    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:33.185587 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.185582    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:33.186255 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.185546    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:33.186255 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.185661    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:33.186255 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.185677    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:33.186255 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:33.185753    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 12:38:33.186255 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:33.185830    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7-metrics-certs podName:86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7 nodeName:}" failed. No retries permitted until 2026-07-02 12:38:33.685793048 +0000 UTC m=+2.293256073 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7-metrics-certs") pod "network-metrics-daemon-sbnql" (UID: "86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 12:38:33.186255 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.186192    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:33.186255 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.186211    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:33.186255 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.186225    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:33.186513 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.186263    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:33.186513 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.186277    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:33.186513 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.186291    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:33.188689 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.188675    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-t6qwm"]
Jul 02 12:38:33.188791 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.188782    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:33.188868 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.188780    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4bj55"
Jul 02 12:38:33.188921 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.188883    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:33.189513 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.189497    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:33.189576 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.189517    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:33.189576 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.189523    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:33.189576 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.189542    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:33.189576 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.189552    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:33.189714 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.189526    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:33.191965 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.191948    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 02 12:38:33.192058 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.191948    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 02 12:38:33.192099 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.192069    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t6qwm"
Jul 02 12:38:33.192099 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.192094    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:33.192281 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.192265    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 02 12:38:33.192318 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.192269    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bcpfd\""
Jul 02 12:38:33.192992 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.192977    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:33.193076 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.192998    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:33.193076 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.193007    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:33.195001 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:33.194983    2576 projected.go:291] Couldn't get configMap openshift-multus/kube-root-ca.crt: object "openshift-multus"/"kube-root-ca.crt" not registered
Jul 02 12:38:33.195001 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:33.195003    2576 projected.go:291] Couldn't get configMap openshift-multus/openshift-service-ca.crt: object "openshift-multus"/"openshift-service-ca.crt" not registered
Jul 02 12:38:33.195140 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:33.195013    2576 projected.go:196] Error preparing data for projected volume kube-api-access-f76d4 for pod openshift-multus/network-metrics-daemon-sbnql: [object "openshift-multus"/"kube-root-ca.crt" not registered, object "openshift-multus"/"openshift-service-ca.crt" not registered]
Jul 02 12:38:33.195140 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:33.195066    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7-kube-api-access-f76d4 podName:86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7 nodeName:}" failed. No retries permitted until 2026-07-02 12:38:33.695051403 +0000 UTC m=+2.302514406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-f76d4" (UniqueName: "kubernetes.io/projected/86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7-kube-api-access-f76d4") pod "network-metrics-daemon-sbnql" (UID: "86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7") : [object "openshift-multus"/"kube-root-ca.crt" not registered, object "openshift-multus"/"openshift-service-ca.crt" not registered]
Jul 02 12:38:33.195240 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.195177    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7hb6h\""
Jul 02 12:38:33.195240 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.195177    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 02 12:38:33.195408 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.195396    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 02 12:38:33.197596 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.197578    2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 02 12:38:33.203706 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.203686    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fmm7\" (UniqueName: \"kubernetes.io/projected/3d27224c-42a6-4708-a061-1705cbdba994-kube-api-access-8fmm7\") pod \"aws-ebs-csi-driver-node-cltnv\" (UID: \"3d27224c-42a6-4708-a061-1705cbdba994\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cltnv"
Jul 02 12:38:33.210046 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:33.210026    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-88.ec2.internal\" not found"
Jul 02 12:38:33.281759 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.281727    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-mztvs"]
Jul 02 12:38:33.281932 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.281903    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:33.283073 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.283057    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:33.283167 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.283085    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:33.283167 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.283097    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:33.285225 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.285208    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-88.ec2.internal"
Jul 02 12:38:33.285315 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.285298    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-env-overrides\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.285380 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.285347    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-host-slash\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.285444 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.285375    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-var-lib-openvswitch\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.285444 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.285400    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkr4n\" (UniqueName: \"kubernetes.io/projected/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-kube-api-access-qkr4n\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.285444 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.285435    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7f9139f3-1b94-4ddf-854a-53db4e0693f1-agent-certs\") pod \"konnectivity-agent-t6qwm\" (UID: \"7f9139f3-1b94-4ddf-854a-53db4e0693f1\") " pod="kube-system/konnectivity-agent-t6qwm"
Jul 02 12:38:33.285591 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.285453    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-host-slash\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.285591 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.285469    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-host-run-netns\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.285591 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.285453    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-var-lib-openvswitch\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.285591 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.285495    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-run-openvswitch\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.285591 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.285537    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-host-run-netns\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.285591 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.285532    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-run-openvswitch\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.285591 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.285532    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-host-cni-bin\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.285591 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.285579    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-host-cni-bin\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.285591 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.285586    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.286003 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.285621    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5f219618-c523-45c1-9042-7e46c4994e49-serviceca\") pod \"node-ca-mztvs\" (UID: \"5f219618-c523-45c1-9042-7e46c4994e49\") " pod="openshift-image-registry/node-ca-mztvs"
Jul 02 12:38:33.286003 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.285624    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.286003 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.285649    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-systemd-units\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.286003 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.285693    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-etc-openvswitch\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.286003 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.285714    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mztvs"
Jul 02 12:38:33.286003 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.285718    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-host-run-ovn-kubernetes\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.286003 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.285721    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-systemd-units\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.286003 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.285693    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:33.286003 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.285743    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:33.286003 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.285746    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-node-log\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.286003 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.285745    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-etc-openvswitch\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.286003 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.285772    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bb22eeab-d1d6-47f7-be7f-0b16dc5de6ef-iptables-alerter-script\") pod \"iptables-alerter-4bj55\" (UID: \"bb22eeab-d1d6-47f7-be7f-0b16dc5de6ef\") " pod="openshift-network-operator/iptables-alerter-4bj55"
Jul 02 12:38:33.286003 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.285777    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-node-log\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.286003 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.285763    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-host-run-ovn-kubernetes\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.286003 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.285824    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f219618-c523-45c1-9042-7e46c4994e49-host\") pod \"node-ca-mztvs\" (UID: \"5f219618-c523-45c1-9042-7e46c4994e49\") " pod="openshift-image-registry/node-ca-mztvs"
Jul 02 12:38:33.286003 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.285848    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-host-cni-netd\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.286003 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.285876    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-ovnkube-config\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.286003 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.285904    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-ovn-node-metrics-cert\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.286003 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.285904    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-host-cni-netd\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.286723 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.285946    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-ovnkube-script-lib\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.286723 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.285986    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:33.286723 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.285991    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb22eeab-d1d6-47f7-be7f-0b16dc5de6ef-host-slash\") pod \"iptables-alerter-4bj55\" (UID: \"bb22eeab-d1d6-47f7-be7f-0b16dc5de6ef\") " pod="openshift-network-operator/iptables-alerter-4bj55"
Jul 02 12:38:33.286723 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.286016    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7f9139f3-1b94-4ddf-854a-53db4e0693f1-konnectivity-ca\") pod \"konnectivity-agent-t6qwm\" (UID: \"7f9139f3-1b94-4ddf-854a-53db4e0693f1\") " pod="kube-system/konnectivity-agent-t6qwm"
Jul 02 12:38:33.286723 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.286041    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85flv\" (UniqueName: \"kubernetes.io/projected/bb22eeab-d1d6-47f7-be7f-0b16dc5de6ef-kube-api-access-85flv\") pod \"iptables-alerter-4bj55\" (UID: \"bb22eeab-d1d6-47f7-be7f-0b16dc5de6ef\") " pod="openshift-network-operator/iptables-alerter-4bj55"
Jul 02 12:38:33.286723 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.286065    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsrgf\" (UniqueName: \"kubernetes.io/projected/5f219618-c523-45c1-9042-7e46c4994e49-kube-api-access-hsrgf\") pod \"node-ca-mztvs\" (UID: \"5f219618-c523-45c1-9042-7e46c4994e49\") " pod="openshift-image-registry/node-ca-mztvs"
Jul 02 12:38:33.286723 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.286087    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zvwz\" (UniqueName: \"kubernetes.io/projected/ebc69c1d-a909-44d0-bcb1-691e39c5e754-kube-api-access-2zvwz\") pod \"network-check-target-tczr2\" (UID: \"ebc69c1d-a909-44d0-bcb1-691e39c5e754\") " pod="openshift-network-diagnostics/network-check-target-tczr2"
Jul 02 12:38:33.286723 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.286102    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-run-systemd\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.286723 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.286119    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-host-kubelet\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.286723 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.286135    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-run-ovn\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.286723 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.286151    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-log-socket\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.286723 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.286178    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-run-systemd\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.286723 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.286198    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-log-socket\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.286723 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.286238    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-host-kubelet\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.286723 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.286251    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:33.286723 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.286248    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-run-ovn\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.286723 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.286615    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:33.286723 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.286642    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:33.286723 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.286655    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:33.286723 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.286673    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:33.286723 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.286698    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:33.286723 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.286710    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:33.287864 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.286744    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:33.287864 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.286785    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:33.287864 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.286800    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:33.287864 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.287095    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-env-overrides\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.287864 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.287273    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:33.287864 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.287297    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:33.287864 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.287311    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:33.287864 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.287831    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-ovnkube-script-lib\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.287864 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.287848    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-f9pzc"]
Jul 02 12:38:33.288193 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.288067    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:33.288482 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.288442    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-ovnkube-config\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.288730 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.288711    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:33.288796 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.288743    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:33.288796 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.288772    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:33.289179 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.289162    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 02 12:38:33.289256 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.289178    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-88.ec2.internal"
Jul 02 12:38:33.289305 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.289279    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-ovn-node-metrics-cert\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.290771 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.290182    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 02 12:38:33.290771 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.290352    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 02 12:38:33.290771 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.290435    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8twvg\""
Jul 02 12:38:33.291511 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.291493    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.291574 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.291527    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:33.292336 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.292319    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:33.292409 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.292340    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:33.292409 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.292349    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:33.294593 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.294574    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 02 12:38:33.294685 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.294641    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 02 12:38:33.294744 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.294691    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bxtwn\""
Jul 02 12:38:33.302016 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:33.301998    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 12:38:33.302094 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:33.302021    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 12:38:33.302094 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:33.302039    2576 projected.go:196] Error preparing data for projected volume kube-api-access-2zvwz for pod openshift-network-diagnostics/network-check-target-tczr2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 12:38:33.302094 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:33.302081    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ebc69c1d-a909-44d0-bcb1-691e39c5e754-kube-api-access-2zvwz podName:ebc69c1d-a909-44d0-bcb1-691e39c5e754 nodeName:}" failed. No retries permitted until 2026-07-02 12:38:33.802067318 +0000 UTC m=+2.409530319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2zvwz" (UniqueName: "kubernetes.io/projected/ebc69c1d-a909-44d0-bcb1-691e39c5e754-kube-api-access-2zvwz") pod "network-check-target-tczr2" (UID: "ebc69c1d-a909-44d0-bcb1-691e39c5e754") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 12:38:33.302457 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.302440    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkr4n\" (UniqueName: \"kubernetes.io/projected/116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a-kube-api-access-qkr4n\") pod \"ovnkube-node-2c7ks\" (UID: \"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.306006 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.305863    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-fzrpv"]
Jul 02 12:38:33.306085 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.306072    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:33.310746 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:33.310728    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-88.ec2.internal\" not found"
Jul 02 12:38:33.314030 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.314015    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:33.314111 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.314041    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:33.314111 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.314054    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:33.316563 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.316544    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-hh6v9"]
Jul 02 12:38:33.316770 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.316748    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.316867 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.316786    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:33.317061 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.317043    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:33.317581 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.317564    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:33.317669 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.317594    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:33.317669 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.317603    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:33.317669 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.317563    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:33.317812 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.317679    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:33.317812 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.317695    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:33.320119 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.320071    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hh6v9"
Jul 02 12:38:33.320119 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.320109    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:33.320311 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.320191    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 02 12:38:33.320311 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.320222    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 02 12:38:33.320311 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.320280    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 02 12:38:33.320311 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.320308    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-d4r8g\""
Jul 02 12:38:33.320496 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.320307    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 02 12:38:33.320775 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.320761    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:33.320844 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.320790    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:33.320844 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.320804    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:33.323086 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.323032    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 02 12:38:33.323203 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.323158    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 02 12:38:33.323298 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.323283    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lngh4\""
Jul 02 12:38:33.359642 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.359619    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cltnv"
Jul 02 12:38:33.363070 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.363053    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-xl5pq"]
Jul 02 12:38:33.363282 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.363269    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:33.364118 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.364100    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:33.364202 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.364129    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:33.364202 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.364145    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:33.364378 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.364361    2576 predicate.go:218] "Predicate failed on Pod" pod="kube-system/global-pull-secret-syncer-xl5pq" err="Predicate NodeAffinity failed: node(s) didn't match Pod's node affinity/selector"
Jul 02 12:38:33.364435 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.364382    2576 allocation_manager.go:702] "Pod admission denied" podUID="2ba833ad-b709-4ffd-85be-2e2c9cf867e6" pod="kube-system/global-pull-secret-syncer-xl5pq" reason="NodeAffinity" message="Predicate NodeAffinity failed: node(s) didn't match Pod's node affinity/selector"
Jul 02 12:38:33.382133 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.382081    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-hmjl6"]
Jul 02 12:38:33.382248 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.382238    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:33.382978 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.382963    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:33.383067 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.382993    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:33.383067 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.383008    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:33.385527 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.385511    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hmjl6"
Jul 02 12:38:33.385626 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.385542    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:33.386210 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.386196    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:33.386346 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.386223    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:33.386346 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.386238    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:33.386346 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.386320    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85flv\" (UniqueName: \"kubernetes.io/projected/bb22eeab-d1d6-47f7-be7f-0b16dc5de6ef-kube-api-access-85flv\") pod \"iptables-alerter-4bj55\" (UID: \"bb22eeab-d1d6-47f7-be7f-0b16dc5de6ef\") " pod="openshift-network-operator/iptables-alerter-4bj55"
Jul 02 12:38:33.386346 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.386344    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a1138068-d606-49cf-b424-b778436f5ab6-etc-modprobe-d\") pod \"tuned-f9pzc\" (UID: \"a1138068-d606-49cf-b424-b778436f5ab6\") " pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.386540 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.386362    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1138068-d606-49cf-b424-b778436f5ab6-etc-kubernetes\") pod \"tuned-f9pzc\" (UID: \"a1138068-d606-49cf-b424-b778436f5ab6\") " pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.386540 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.386382    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a1138068-d606-49cf-b424-b778436f5ab6-lib-modules\") pod \"tuned-f9pzc\" (UID: \"a1138068-d606-49cf-b424-b778436f5ab6\") " pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.386540 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.386442    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1138068-d606-49cf-b424-b778436f5ab6-tmp\") pod \"tuned-f9pzc\" (UID: \"a1138068-d606-49cf-b424-b778436f5ab6\") " pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.386540 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.386491    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6652ccdf-a3c9-42f8-88fc-39690889dd42-etc-kubernetes\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.386735 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.386529    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:33.386735 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.386534    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a1138068-d606-49cf-b424-b778436f5ab6-etc-sysctl-conf\") pod \"tuned-f9pzc\" (UID: \"a1138068-d606-49cf-b424-b778436f5ab6\") " pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.386735 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.386592    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a1138068-d606-49cf-b424-b778436f5ab6-host\") pod \"tuned-f9pzc\" (UID: \"a1138068-d606-49cf-b424-b778436f5ab6\") " pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.386735 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.386636    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a1138068-d606-49cf-b424-b778436f5ab6-etc-tuned\") pod \"tuned-f9pzc\" (UID: \"a1138068-d606-49cf-b424-b778436f5ab6\") " pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.386735 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.386663    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ba833ad-b709-4ffd-85be-2e2c9cf867e6-original-pull-secret\") pod \"global-pull-secret-syncer-xl5pq\" (UID: \"2ba833ad-b709-4ffd-85be-2e2c9cf867e6\") " pod="kube-system/global-pull-secret-syncer-xl5pq"
Jul 02 12:38:33.386735 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.386714    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7f9139f3-1b94-4ddf-854a-53db4e0693f1-agent-certs\") pod \"konnectivity-agent-t6qwm\" (UID: \"7f9139f3-1b94-4ddf-854a-53db4e0693f1\") " pod="kube-system/konnectivity-agent-t6qwm"
Jul 02 12:38:33.387040 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.386737    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6652ccdf-a3c9-42f8-88fc-39690889dd42-cni-binary-copy\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.387040 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.386760    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9lnc\" (UniqueName: \"kubernetes.io/projected/64db6250-f644-4f6d-856d-f3186f40db09-kube-api-access-s9lnc\") pod \"multus-additional-cni-plugins-hh6v9\" (UID: \"64db6250-f644-4f6d-856d-f3186f40db09\") " pod="openshift-multus/multus-additional-cni-plugins-hh6v9"
Jul 02 12:38:33.387040 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.386794    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5f219618-c523-45c1-9042-7e46c4994e49-serviceca\") pod \"node-ca-mztvs\" (UID: \"5f219618-c523-45c1-9042-7e46c4994e49\") " pod="openshift-image-registry/node-ca-mztvs"
Jul 02 12:38:33.387040 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.386802    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:33.387040 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.386818    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6652ccdf-a3c9-42f8-88fc-39690889dd42-os-release\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.387040 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.386843    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6652ccdf-a3c9-42f8-88fc-39690889dd42-host-run-k8s-cni-cncf-io\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.387040 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.386869    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/64db6250-f644-4f6d-856d-f3186f40db09-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hh6v9\" (UID: \"64db6250-f644-4f6d-856d-f3186f40db09\") " pod="openshift-multus/multus-additional-cni-plugins-hh6v9"
Jul 02 12:38:33.387040 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.387004    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2ba833ad-b709-4ffd-85be-2e2c9cf867e6-kubelet-config\") pod \"global-pull-secret-syncer-xl5pq\" (UID: \"2ba833ad-b709-4ffd-85be-2e2c9cf867e6\") " pod="kube-system/global-pull-secret-syncer-xl5pq"
Jul 02 12:38:33.387040 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.387033    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6652ccdf-a3c9-42f8-88fc-39690889dd42-multus-cni-dir\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.387504 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.387067    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6652ccdf-a3c9-42f8-88fc-39690889dd42-cnibin\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.387504 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.387157    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6652ccdf-a3c9-42f8-88fc-39690889dd42-host-run-netns\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.387504 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.387199    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f219618-c523-45c1-9042-7e46c4994e49-host\") pod \"node-ca-mztvs\" (UID: \"5f219618-c523-45c1-9042-7e46c4994e49\") " pod="openshift-image-registry/node-ca-mztvs"
Jul 02 12:38:33.387504 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.387225    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6652ccdf-a3c9-42f8-88fc-39690889dd42-host-var-lib-cni-bin\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.387504 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.387247    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6652ccdf-a3c9-42f8-88fc-39690889dd42-hostroot\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.387504 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.387256    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f219618-c523-45c1-9042-7e46c4994e49-host\") pod \"node-ca-mztvs\" (UID: \"5f219618-c523-45c1-9042-7e46c4994e49\") " pod="openshift-image-registry/node-ca-mztvs"
Jul 02 12:38:33.387504 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.387269    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6652ccdf-a3c9-42f8-88fc-39690889dd42-multus-conf-dir\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.387504 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.387284    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5f219618-c523-45c1-9042-7e46c4994e49-serviceca\") pod \"node-ca-mztvs\" (UID: \"5f219618-c523-45c1-9042-7e46c4994e49\") " pod="openshift-image-registry/node-ca-mztvs"
Jul 02 12:38:33.387504 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.387327    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7f9139f3-1b94-4ddf-854a-53db4e0693f1-konnectivity-ca\") pod \"konnectivity-agent-t6qwm\" (UID: \"7f9139f3-1b94-4ddf-854a-53db4e0693f1\") " pod="kube-system/konnectivity-agent-t6qwm"
Jul 02 12:38:33.387504 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.387355    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:33.387504 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.387374    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:33.387504 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.387387    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:33.387504 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.387390    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:33.387504 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.387414    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:33.387504 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.387428    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:33.388219 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.387390    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6652ccdf-a3c9-42f8-88fc-39690889dd42-system-cni-dir\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.388219 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.387577    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa04af49-8c6c-46fa-bfcc-500d89b526c9-tmp-dir\") pod \"node-resolver-hmjl6\" (UID: \"fa04af49-8c6c-46fa-bfcc-500d89b526c9\") " pod="openshift-dns/node-resolver-hmjl6"
Jul 02 12:38:33.388219 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.387603    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/64db6250-f644-4f6d-856d-f3186f40db09-cni-binary-copy\") pod \"multus-additional-cni-plugins-hh6v9\" (UID: \"64db6250-f644-4f6d-856d-f3186f40db09\") " pod="openshift-multus/multus-additional-cni-plugins-hh6v9"
Jul 02 12:38:33.388219 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.387631    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsrgf\" (UniqueName: \"kubernetes.io/projected/5f219618-c523-45c1-9042-7e46c4994e49-kube-api-access-hsrgf\") pod \"node-ca-mztvs\" (UID: \"5f219618-c523-45c1-9042-7e46c4994e49\") " pod="openshift-image-registry/node-ca-mztvs"
Jul 02 12:38:33.388219 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.387659    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fa04af49-8c6c-46fa-bfcc-500d89b526c9-hosts-file\") pod \"node-resolver-hmjl6\" (UID: \"fa04af49-8c6c-46fa-bfcc-500d89b526c9\") " pod="openshift-dns/node-resolver-hmjl6"
Jul 02 12:38:33.388219 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.387654    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xl5pq" podStartSLOduration=0.387633096 podStartE2EDuration="387.633096ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 12:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:38:33.387476727 +0000 UTC m=+1.994939755" watchObservedRunningTime="2026-07-02 12:38:33.387633096 +0000 UTC m=+1.995096120"
Jul 02 12:38:33.388219 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.387754    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:33.388219 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.387780    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7f9139f3-1b94-4ddf-854a-53db4e0693f1-konnectivity-ca\") pod \"konnectivity-agent-t6qwm\" (UID: \"7f9139f3-1b94-4ddf-854a-53db4e0693f1\") " pod="kube-system/konnectivity-agent-t6qwm"
Jul 02 12:38:33.388219 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.387683    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64db6250-f644-4f6d-856d-f3186f40db09-cnibin\") pod \"multus-additional-cni-plugins-hh6v9\" (UID: \"64db6250-f644-4f6d-856d-f3186f40db09\") " pod="openshift-multus/multus-additional-cni-plugins-hh6v9"
Jul 02 12:38:33.388219 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.387853    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64db6250-f644-4f6d-856d-f3186f40db09-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hh6v9\" (UID: \"64db6250-f644-4f6d-856d-f3186f40db09\") " pod="openshift-multus/multus-additional-cni-plugins-hh6v9"
Jul 02 12:38:33.388219 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.387887    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcvzh\" (UniqueName: \"kubernetes.io/projected/6652ccdf-a3c9-42f8-88fc-39690889dd42-kube-api-access-qcvzh\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.388219 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.387926    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwsqp\" (UniqueName: \"kubernetes.io/projected/fa04af49-8c6c-46fa-bfcc-500d89b526c9-kube-api-access-lwsqp\") pod \"node-resolver-hmjl6\" (UID: \"fa04af49-8c6c-46fa-bfcc-500d89b526c9\") " pod="openshift-dns/node-resolver-hmjl6"
Jul 02 12:38:33.388219 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.387952    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64db6250-f644-4f6d-856d-f3186f40db09-system-cni-dir\") pod \"multus-additional-cni-plugins-hh6v9\" (UID: \"64db6250-f644-4f6d-856d-f3186f40db09\") " pod="openshift-multus/multus-additional-cni-plugins-hh6v9"
Jul 02 12:38:33.388219 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.387979    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/64db6250-f644-4f6d-856d-f3186f40db09-os-release\") pod \"multus-additional-cni-plugins-hh6v9\" (UID: \"64db6250-f644-4f6d-856d-f3186f40db09\") " pod="openshift-multus/multus-additional-cni-plugins-hh6v9"
Jul 02 12:38:33.388219 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.388005    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1138068-d606-49cf-b424-b778436f5ab6-var-lib-kubelet\") pod \"tuned-f9pzc\" (UID: \"a1138068-d606-49cf-b424-b778436f5ab6\") " pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.388219 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.388032    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1138068-d606-49cf-b424-b778436f5ab6-sys\") pod \"tuned-f9pzc\" (UID: \"a1138068-d606-49cf-b424-b778436f5ab6\") " pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.388219 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.388057    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6652ccdf-a3c9-42f8-88fc-39690889dd42-host-var-lib-cni-multus\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.389062 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.388110    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a1138068-d606-49cf-b424-b778436f5ab6-etc-sysctl-d\") pod \"tuned-f9pzc\" (UID: \"a1138068-d606-49cf-b424-b778436f5ab6\") " pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.389062 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.388142    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a1138068-d606-49cf-b424-b778436f5ab6-etc-systemd\") pod \"tuned-f9pzc\" (UID: \"a1138068-d606-49cf-b424-b778436f5ab6\") " pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.389062 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.388183    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6652ccdf-a3c9-42f8-88fc-39690889dd42-multus-socket-dir-parent\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.389062 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.388227    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2ba833ad-b709-4ffd-85be-2e2c9cf867e6-dbus\") pod \"global-pull-secret-syncer-xl5pq\" (UID: \"2ba833ad-b709-4ffd-85be-2e2c9cf867e6\") " pod="kube-system/global-pull-secret-syncer-xl5pq"
Jul 02 12:38:33.389062 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.388254    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltnld\" (UniqueName: \"kubernetes.io/projected/a1138068-d606-49cf-b424-b778436f5ab6-kube-api-access-ltnld\") pod \"tuned-f9pzc\" (UID: \"a1138068-d606-49cf-b424-b778436f5ab6\") " pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.389062 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.388272    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6652ccdf-a3c9-42f8-88fc-39690889dd42-host-var-lib-kubelet\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.389062 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.388295    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6652ccdf-a3c9-42f8-88fc-39690889dd42-multus-daemon-config\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.389062 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.388323    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bb22eeab-d1d6-47f7-be7f-0b16dc5de6ef-iptables-alerter-script\") pod \"iptables-alerter-4bj55\" (UID: \"bb22eeab-d1d6-47f7-be7f-0b16dc5de6ef\") " pod="openshift-network-operator/iptables-alerter-4bj55"
Jul 02 12:38:33.389062 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.388345    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a1138068-d606-49cf-b424-b778436f5ab6-run\") pod \"tuned-f9pzc\" (UID: \"a1138068-d606-49cf-b424-b778436f5ab6\") " pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.389062 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.388366    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6652ccdf-a3c9-42f8-88fc-39690889dd42-host-run-multus-certs\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.389062 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.388396    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb22eeab-d1d6-47f7-be7f-0b16dc5de6ef-host-slash\") pod \"iptables-alerter-4bj55\" (UID: \"bb22eeab-d1d6-47f7-be7f-0b16dc5de6ef\") " pod="openshift-network-operator/iptables-alerter-4bj55"
Jul 02 12:38:33.389062 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.388424    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a1138068-d606-49cf-b424-b778436f5ab6-etc-sysconfig\") pod \"tuned-f9pzc\" (UID: \"a1138068-d606-49cf-b424-b778436f5ab6\") " pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.389062 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.388477    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:33.389062 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.388500    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:33.389062 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.388515    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:33.389062 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.388537    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb22eeab-d1d6-47f7-be7f-0b16dc5de6ef-host-slash\") pod \"iptables-alerter-4bj55\" (UID: \"bb22eeab-d1d6-47f7-be7f-0b16dc5de6ef\") " pod="openshift-network-operator/iptables-alerter-4bj55"
Jul 02 12:38:33.389062 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.388551    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/64db6250-f644-4f6d-856d-f3186f40db09-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hh6v9\" (UID: \"64db6250-f644-4f6d-856d-f3186f40db09\") " pod="openshift-multus/multus-additional-cni-plugins-hh6v9"
Jul 02 12:38:33.389062 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.388721    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 02 12:38:33.389062 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.388805    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 02 12:38:33.389062 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.388810    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-d2kfx\""
Jul 02 12:38:33.389602 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.388925    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bb22eeab-d1d6-47f7-be7f-0b16dc5de6ef-iptables-alerter-script\") pod \"iptables-alerter-4bj55\" (UID: \"bb22eeab-d1d6-47f7-be7f-0b16dc5de6ef\") " pod="openshift-network-operator/iptables-alerter-4bj55"
Jul 02 12:38:33.389890 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.389869    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7f9139f3-1b94-4ddf-854a-53db4e0693f1-agent-certs\") pod \"konnectivity-agent-t6qwm\" (UID: \"7f9139f3-1b94-4ddf-854a-53db4e0693f1\") " pod="kube-system/konnectivity-agent-t6qwm"
Jul 02 12:38:33.399990 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.399966    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsrgf\" (UniqueName: \"kubernetes.io/projected/5f219618-c523-45c1-9042-7e46c4994e49-kube-api-access-hsrgf\") pod \"node-ca-mztvs\" (UID: \"5f219618-c523-45c1-9042-7e46c4994e49\") " pod="openshift-image-registry/node-ca-mztvs"
Jul 02 12:38:33.400400 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.400385    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85flv\" (UniqueName: \"kubernetes.io/projected/bb22eeab-d1d6-47f7-be7f-0b16dc5de6ef-kube-api-access-85flv\") pod \"iptables-alerter-4bj55\" (UID: \"bb22eeab-d1d6-47f7-be7f-0b16dc5de6ef\") " pod="openshift-network-operator/iptables-alerter-4bj55"
Jul 02 12:38:33.411249 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:33.411233    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-88.ec2.internal\" not found"
Jul 02 12:38:33.433653 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.433635    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:33.488876 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.488847    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6652ccdf-a3c9-42f8-88fc-39690889dd42-host-run-k8s-cni-cncf-io\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.489014 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.488883    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/64db6250-f644-4f6d-856d-f3186f40db09-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hh6v9\" (UID: \"64db6250-f644-4f6d-856d-f3186f40db09\") " pod="openshift-multus/multus-additional-cni-plugins-hh6v9"
Jul 02 12:38:33.489014 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.488926    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2ba833ad-b709-4ffd-85be-2e2c9cf867e6-kubelet-config\") pod \"global-pull-secret-syncer-xl5pq\" (UID: \"2ba833ad-b709-4ffd-85be-2e2c9cf867e6\") " pod="kube-system/global-pull-secret-syncer-xl5pq"
Jul 02 12:38:33.489014 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.488954    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6652ccdf-a3c9-42f8-88fc-39690889dd42-multus-cni-dir\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.489014 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.488978    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6652ccdf-a3c9-42f8-88fc-39690889dd42-cnibin\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.489014 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489001    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6652ccdf-a3c9-42f8-88fc-39690889dd42-host-run-netns\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.489236 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489040    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2ba833ad-b709-4ffd-85be-2e2c9cf867e6-kubelet-config\") pod \"global-pull-secret-syncer-xl5pq\" (UID: \"2ba833ad-b709-4ffd-85be-2e2c9cf867e6\") " pod="kube-system/global-pull-secret-syncer-xl5pq"
Jul 02 12:38:33.489236 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489052    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6652ccdf-a3c9-42f8-88fc-39690889dd42-host-var-lib-cni-bin\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.489236 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489079    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6652ccdf-a3c9-42f8-88fc-39690889dd42-cnibin\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.489236 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489094    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6652ccdf-a3c9-42f8-88fc-39690889dd42-hostroot\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.489236 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489121    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6652ccdf-a3c9-42f8-88fc-39690889dd42-hostroot\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.489236 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489115    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6652ccdf-a3c9-42f8-88fc-39690889dd42-host-run-k8s-cni-cncf-io\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.489236 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489132    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6652ccdf-a3c9-42f8-88fc-39690889dd42-multus-conf-dir\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.489236 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489164    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6652ccdf-a3c9-42f8-88fc-39690889dd42-multus-conf-dir\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.489236 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489174    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6652ccdf-a3c9-42f8-88fc-39690889dd42-system-cni-dir\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.489236 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489138    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6652ccdf-a3c9-42f8-88fc-39690889dd42-multus-cni-dir\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.489236 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489193    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6652ccdf-a3c9-42f8-88fc-39690889dd42-host-var-lib-cni-bin\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.489236 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489200    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa04af49-8c6c-46fa-bfcc-500d89b526c9-tmp-dir\") pod \"node-resolver-hmjl6\" (UID: \"fa04af49-8c6c-46fa-bfcc-500d89b526c9\") " pod="openshift-dns/node-resolver-hmjl6"
Jul 02 12:38:33.489236 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489166    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6652ccdf-a3c9-42f8-88fc-39690889dd42-host-run-netns\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.489236 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489226    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/64db6250-f644-4f6d-856d-f3186f40db09-cni-binary-copy\") pod \"multus-additional-cni-plugins-hh6v9\" (UID: \"64db6250-f644-4f6d-856d-f3186f40db09\") " pod="openshift-multus/multus-additional-cni-plugins-hh6v9"
Jul 02 12:38:33.489820 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489251    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fa04af49-8c6c-46fa-bfcc-500d89b526c9-hosts-file\") pod \"node-resolver-hmjl6\" (UID: \"fa04af49-8c6c-46fa-bfcc-500d89b526c9\") " pod="openshift-dns/node-resolver-hmjl6"
Jul 02 12:38:33.489820 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489228    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6652ccdf-a3c9-42f8-88fc-39690889dd42-system-cni-dir\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.489820 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489274    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64db6250-f644-4f6d-856d-f3186f40db09-cnibin\") pod \"multus-additional-cni-plugins-hh6v9\" (UID: \"64db6250-f644-4f6d-856d-f3186f40db09\") " pod="openshift-multus/multus-additional-cni-plugins-hh6v9"
Jul 02 12:38:33.489820 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489323    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fa04af49-8c6c-46fa-bfcc-500d89b526c9-hosts-file\") pod \"node-resolver-hmjl6\" (UID: \"fa04af49-8c6c-46fa-bfcc-500d89b526c9\") " pod="openshift-dns/node-resolver-hmjl6"
Jul 02 12:38:33.489820 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489352    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64db6250-f644-4f6d-856d-f3186f40db09-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hh6v9\" (UID: \"64db6250-f644-4f6d-856d-f3186f40db09\") " pod="openshift-multus/multus-additional-cni-plugins-hh6v9"
Jul 02 12:38:33.489820 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489383    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcvzh\" (UniqueName: \"kubernetes.io/projected/6652ccdf-a3c9-42f8-88fc-39690889dd42-kube-api-access-qcvzh\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.489820 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489390    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64db6250-f644-4f6d-856d-f3186f40db09-cnibin\") pod \"multus-additional-cni-plugins-hh6v9\" (UID: \"64db6250-f644-4f6d-856d-f3186f40db09\") " pod="openshift-multus/multus-additional-cni-plugins-hh6v9"
Jul 02 12:38:33.489820 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489405    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwsqp\" (UniqueName: \"kubernetes.io/projected/fa04af49-8c6c-46fa-bfcc-500d89b526c9-kube-api-access-lwsqp\") pod \"node-resolver-hmjl6\" (UID: \"fa04af49-8c6c-46fa-bfcc-500d89b526c9\") " pod="openshift-dns/node-resolver-hmjl6"
Jul 02 12:38:33.489820 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489443    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64db6250-f644-4f6d-856d-f3186f40db09-system-cni-dir\") pod \"multus-additional-cni-plugins-hh6v9\" (UID: \"64db6250-f644-4f6d-856d-f3186f40db09\") " pod="openshift-multus/multus-additional-cni-plugins-hh6v9"
Jul 02 12:38:33.489820 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489492    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/64db6250-f644-4f6d-856d-f3186f40db09-os-release\") pod \"multus-additional-cni-plugins-hh6v9\" (UID: \"64db6250-f644-4f6d-856d-f3186f40db09\") " pod="openshift-multus/multus-additional-cni-plugins-hh6v9"
Jul 02 12:38:33.489820 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489500    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:33.489820 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489503    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/64db6250-f644-4f6d-856d-f3186f40db09-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hh6v9\" (UID: \"64db6250-f644-4f6d-856d-f3186f40db09\") " pod="openshift-multus/multus-additional-cni-plugins-hh6v9"
Jul 02 12:38:33.489820 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489530    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64db6250-f644-4f6d-856d-f3186f40db09-system-cni-dir\") pod \"multus-additional-cni-plugins-hh6v9\" (UID: \"64db6250-f644-4f6d-856d-f3186f40db09\") " pod="openshift-multus/multus-additional-cni-plugins-hh6v9"
Jul 02 12:38:33.489820 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489554    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1138068-d606-49cf-b424-b778436f5ab6-var-lib-kubelet\") pod \"tuned-f9pzc\" (UID: \"a1138068-d606-49cf-b424-b778436f5ab6\") " pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.489820 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489558    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/64db6250-f644-4f6d-856d-f3186f40db09-os-release\") pod \"multus-additional-cni-plugins-hh6v9\" (UID: \"64db6250-f644-4f6d-856d-f3186f40db09\") " pod="openshift-multus/multus-additional-cni-plugins-hh6v9"
Jul 02 12:38:33.489820 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489569    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:33.489820 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489498    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa04af49-8c6c-46fa-bfcc-500d89b526c9-tmp-dir\") pod \"node-resolver-hmjl6\" (UID: \"fa04af49-8c6c-46fa-bfcc-500d89b526c9\") " pod="openshift-dns/node-resolver-hmjl6"
Jul 02 12:38:33.489820 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489574    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1138068-d606-49cf-b424-b778436f5ab6-sys\") pod \"tuned-f9pzc\" (UID: \"a1138068-d606-49cf-b424-b778436f5ab6\") " pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.490671 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489582    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64db6250-f644-4f6d-856d-f3186f40db09-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hh6v9\" (UID: \"64db6250-f644-4f6d-856d-f3186f40db09\") " pod="openshift-multus/multus-additional-cni-plugins-hh6v9"
Jul 02 12:38:33.490671 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489602    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6652ccdf-a3c9-42f8-88fc-39690889dd42-host-var-lib-cni-multus\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.490671 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489615    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1138068-d606-49cf-b424-b778436f5ab6-var-lib-kubelet\") pod \"tuned-f9pzc\" (UID: \"a1138068-d606-49cf-b424-b778436f5ab6\") " pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.490671 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489641    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6652ccdf-a3c9-42f8-88fc-39690889dd42-host-var-lib-cni-multus\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.490671 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489648    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a1138068-d606-49cf-b424-b778436f5ab6-etc-sysctl-d\") pod \"tuned-f9pzc\" (UID: \"a1138068-d606-49cf-b424-b778436f5ab6\") " pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.490671 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489665    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/64db6250-f644-4f6d-856d-f3186f40db09-cni-binary-copy\") pod \"multus-additional-cni-plugins-hh6v9\" (UID: \"64db6250-f644-4f6d-856d-f3186f40db09\") " pod="openshift-multus/multus-additional-cni-plugins-hh6v9"
Jul 02 12:38:33.490671 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489675    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a1138068-d606-49cf-b424-b778436f5ab6-etc-systemd\") pod \"tuned-f9pzc\" (UID: \"a1138068-d606-49cf-b424-b778436f5ab6\") " pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.490671 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489694    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6652ccdf-a3c9-42f8-88fc-39690889dd42-multus-socket-dir-parent\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.490671 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489721    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2ba833ad-b709-4ffd-85be-2e2c9cf867e6-dbus\") pod \"global-pull-secret-syncer-xl5pq\" (UID: \"2ba833ad-b709-4ffd-85be-2e2c9cf867e6\") " pod="kube-system/global-pull-secret-syncer-xl5pq"
Jul 02 12:38:33.490671 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489743    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltnld\" (UniqueName: \"kubernetes.io/projected/a1138068-d606-49cf-b424-b778436f5ab6-kube-api-access-ltnld\") pod \"tuned-f9pzc\" (UID: \"a1138068-d606-49cf-b424-b778436f5ab6\") " pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.490671 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489742    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a1138068-d606-49cf-b424-b778436f5ab6-etc-systemd\") pod \"tuned-f9pzc\" (UID: \"a1138068-d606-49cf-b424-b778436f5ab6\") " pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.490671 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489757    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6652ccdf-a3c9-42f8-88fc-39690889dd42-multus-socket-dir-parent\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.490671 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489651    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1138068-d606-49cf-b424-b778436f5ab6-sys\") pod \"tuned-f9pzc\" (UID: \"a1138068-d606-49cf-b424-b778436f5ab6\") " pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.490671 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489773    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a1138068-d606-49cf-b424-b778436f5ab6-etc-sysctl-d\") pod \"tuned-f9pzc\" (UID: \"a1138068-d606-49cf-b424-b778436f5ab6\") " pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.490671 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489777    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2ba833ad-b709-4ffd-85be-2e2c9cf867e6-dbus\") pod \"global-pull-secret-syncer-xl5pq\" (UID: \"2ba833ad-b709-4ffd-85be-2e2c9cf867e6\") " pod="kube-system/global-pull-secret-syncer-xl5pq"
Jul 02 12:38:33.490671 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489802    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:33.490671 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489801    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6652ccdf-a3c9-42f8-88fc-39690889dd42-host-var-lib-kubelet\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.490671 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489820    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6652ccdf-a3c9-42f8-88fc-39690889dd42-host-var-lib-kubelet\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.491522 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489828    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6652ccdf-a3c9-42f8-88fc-39690889dd42-multus-daemon-config\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.491522 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489848    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a1138068-d606-49cf-b424-b778436f5ab6-run\") pod \"tuned-f9pzc\" (UID: \"a1138068-d606-49cf-b424-b778436f5ab6\") " pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.491522 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489866    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6652ccdf-a3c9-42f8-88fc-39690889dd42-host-run-multus-certs\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.491522 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489882    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a1138068-d606-49cf-b424-b778436f5ab6-etc-sysconfig\") pod \"tuned-f9pzc\" (UID: \"a1138068-d606-49cf-b424-b778436f5ab6\") " pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.491522 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489897    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/64db6250-f644-4f6d-856d-f3186f40db09-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hh6v9\" (UID: \"64db6250-f644-4f6d-856d-f3186f40db09\") " pod="openshift-multus/multus-additional-cni-plugins-hh6v9"
Jul 02 12:38:33.491522 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489947    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6652ccdf-a3c9-42f8-88fc-39690889dd42-host-run-multus-certs\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.491522 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489955    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a1138068-d606-49cf-b424-b778436f5ab6-etc-sysconfig\") pod \"tuned-f9pzc\" (UID: \"a1138068-d606-49cf-b424-b778436f5ab6\") " pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.491522 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.489987    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a1138068-d606-49cf-b424-b778436f5ab6-etc-modprobe-d\") pod \"tuned-f9pzc\" (UID: \"a1138068-d606-49cf-b424-b778436f5ab6\") " pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.491522 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.490001    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a1138068-d606-49cf-b424-b778436f5ab6-run\") pod \"tuned-f9pzc\" (UID: \"a1138068-d606-49cf-b424-b778436f5ab6\") " pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.491522 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.490016    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1138068-d606-49cf-b424-b778436f5ab6-etc-kubernetes\") pod \"tuned-f9pzc\" (UID: \"a1138068-d606-49cf-b424-b778436f5ab6\") " pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.491522 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.490037    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a1138068-d606-49cf-b424-b778436f5ab6-lib-modules\") pod \"tuned-f9pzc\" (UID: \"a1138068-d606-49cf-b424-b778436f5ab6\") " pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.491522 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.490055    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1138068-d606-49cf-b424-b778436f5ab6-tmp\") pod \"tuned-f9pzc\" (UID: \"a1138068-d606-49cf-b424-b778436f5ab6\") " pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.491522 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.490077    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6652ccdf-a3c9-42f8-88fc-39690889dd42-etc-kubernetes\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.491522 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.490092    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1138068-d606-49cf-b424-b778436f5ab6-etc-kubernetes\") pod \"tuned-f9pzc\" (UID: \"a1138068-d606-49cf-b424-b778436f5ab6\") " pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.491522 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.490102    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a1138068-d606-49cf-b424-b778436f5ab6-etc-sysctl-conf\") pod \"tuned-f9pzc\" (UID: \"a1138068-d606-49cf-b424-b778436f5ab6\") " pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.491522 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.490096    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a1138068-d606-49cf-b424-b778436f5ab6-etc-modprobe-d\") pod \"tuned-f9pzc\" (UID: \"a1138068-d606-49cf-b424-b778436f5ab6\") " pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.491522 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.490129    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a1138068-d606-49cf-b424-b778436f5ab6-host\") pod \"tuned-f9pzc\" (UID: \"a1138068-d606-49cf-b424-b778436f5ab6\") " pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.491522 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.490123    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:33.492143 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.490131    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6652ccdf-a3c9-42f8-88fc-39690889dd42-etc-kubernetes\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.492143 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.490151    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a1138068-d606-49cf-b424-b778436f5ab6-etc-tuned\") pod \"tuned-f9pzc\" (UID: \"a1138068-d606-49cf-b424-b778436f5ab6\") " pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.492143 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.490151    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a1138068-d606-49cf-b424-b778436f5ab6-lib-modules\") pod \"tuned-f9pzc\" (UID: \"a1138068-d606-49cf-b424-b778436f5ab6\") " pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.492143 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.490192    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:33.492143 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.490191    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ba833ad-b709-4ffd-85be-2e2c9cf867e6-original-pull-secret\") pod \"global-pull-secret-syncer-xl5pq\" (UID: \"2ba833ad-b709-4ffd-85be-2e2c9cf867e6\") " pod="kube-system/global-pull-secret-syncer-xl5pq"
Jul 02 12:38:33.492143 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.490206    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a1138068-d606-49cf-b424-b778436f5ab6-host\") pod \"tuned-f9pzc\" (UID: \"a1138068-d606-49cf-b424-b778436f5ab6\") " pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.492143 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.490240    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a1138068-d606-49cf-b424-b778436f5ab6-etc-sysctl-conf\") pod \"tuned-f9pzc\" (UID: \"a1138068-d606-49cf-b424-b778436f5ab6\") " pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.492143 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.490248    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:33.492143 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.490250    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6652ccdf-a3c9-42f8-88fc-39690889dd42-cni-binary-copy\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.492143 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.490279    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9lnc\" (UniqueName: \"kubernetes.io/projected/64db6250-f644-4f6d-856d-f3186f40db09-kube-api-access-s9lnc\") pod \"multus-additional-cni-plugins-hh6v9\" (UID: \"64db6250-f644-4f6d-856d-f3186f40db09\") " pod="openshift-multus/multus-additional-cni-plugins-hh6v9"
Jul 02 12:38:33.492143 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.490284    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6652ccdf-a3c9-42f8-88fc-39690889dd42-multus-daemon-config\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.492143 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.490327    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/64db6250-f644-4f6d-856d-f3186f40db09-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hh6v9\" (UID: \"64db6250-f644-4f6d-856d-f3186f40db09\") " pod="openshift-multus/multus-additional-cni-plugins-hh6v9"
Jul 02 12:38:33.492143 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.490336    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:33.492143 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.490369    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6652ccdf-a3c9-42f8-88fc-39690889dd42-os-release\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.492143 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.490463    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6652ccdf-a3c9-42f8-88fc-39690889dd42-os-release\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.492143 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.490647    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6652ccdf-a3c9-42f8-88fc-39690889dd42-cni-binary-copy\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.492143 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.491033    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:33.492143 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.491048    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:33.492143 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.491056    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:33.492143 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.491068    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:33.492143 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.491060    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:33.492143 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.491085    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:33.492143 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.491090    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:33.492143 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.491099    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:33.492901 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.491098    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:33.492901 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.491110    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:33.492901 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.491123    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:33.492901 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.491102    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:33.492901 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.491101    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:33.492901 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.491062    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:33.492901 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.491232    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:33.492901 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.491138    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:33.492901 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.491246    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:33.492901 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.491252    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:33.492901 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:33.491231    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 12:38:33.492901 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:33.491421    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ba833ad-b709-4ffd-85be-2e2c9cf867e6-original-pull-secret podName:2ba833ad-b709-4ffd-85be-2e2c9cf867e6 nodeName:}" failed. No retries permitted until 2026-07-02 12:38:33.991408457 +0000 UTC m=+2.598871457 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ba833ad-b709-4ffd-85be-2e2c9cf867e6-original-pull-secret") pod "global-pull-secret-syncer-xl5pq" (UID: "2ba833ad-b709-4ffd-85be-2e2c9cf867e6") : object "kube-system"/"original-pull-secret" not registered
Jul 02 12:38:33.492901 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.491455    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:33.492901 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.491474    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:33.492901 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.491488    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:33.493450 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.493280    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1138068-d606-49cf-b424-b778436f5ab6-tmp\") pod \"tuned-f9pzc\" (UID: \"a1138068-d606-49cf-b424-b778436f5ab6\") " pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.493450 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.493304    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a1138068-d606-49cf-b424-b778436f5ab6-etc-tuned\") pod \"tuned-f9pzc\" (UID: \"a1138068-d606-49cf-b424-b778436f5ab6\") " pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.501008 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.500983    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwsqp\" (UniqueName: \"kubernetes.io/projected/fa04af49-8c6c-46fa-bfcc-500d89b526c9-kube-api-access-lwsqp\") pod \"node-resolver-hmjl6\" (UID: \"fa04af49-8c6c-46fa-bfcc-500d89b526c9\") " pod="openshift-dns/node-resolver-hmjl6"
Jul 02 12:38:33.501107 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.501013    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltnld\" (UniqueName: \"kubernetes.io/projected/a1138068-d606-49cf-b424-b778436f5ab6-kube-api-access-ltnld\") pod \"tuned-f9pzc\" (UID: \"a1138068-d606-49cf-b424-b778436f5ab6\") " pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.501319 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.501297    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcvzh\" (UniqueName: \"kubernetes.io/projected/6652ccdf-a3c9-42f8-88fc-39690889dd42-kube-api-access-qcvzh\") pod \"multus-fzrpv\" (UID: \"6652ccdf-a3c9-42f8-88fc-39690889dd42\") " pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.501923 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.501897    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9lnc\" (UniqueName: \"kubernetes.io/projected/64db6250-f644-4f6d-856d-f3186f40db09-kube-api-access-s9lnc\") pod \"multus-additional-cni-plugins-hh6v9\" (UID: \"64db6250-f644-4f6d-856d-f3186f40db09\") " pod="openshift-multus/multus-additional-cni-plugins-hh6v9"
Jul 02 12:38:33.511801 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:33.511783    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-88.ec2.internal\" not found"
Jul 02 12:38:33.518951 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.518938    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4bj55"
Jul 02 12:38:33.539059 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.539037    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t6qwm"
Jul 02 12:38:33.612112 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:33.612072    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-88.ec2.internal\" not found"
Jul 02 12:38:33.624262 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.624233    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mztvs"
Jul 02 12:38:33.641060 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.641005    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f9pzc"
Jul 02 12:38:33.662694 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.662673    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fzrpv"
Jul 02 12:38:33.667430 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.667413    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hh6v9"
Jul 02 12:38:33.691671 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.691643    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7-metrics-certs\") pod \"network-metrics-daemon-sbnql\" (UID: \"86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7\") " pod="openshift-multus/network-metrics-daemon-sbnql"
Jul 02 12:38:33.691793 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.691779    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:33.692538 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.692523    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:33.692621 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.692557    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:33.692621 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.692583    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:33.692759 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:33.692745    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 12:38:33.692836 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:33.692816    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7-metrics-certs podName:86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7 nodeName:}" failed. No retries permitted until 2026-07-02 12:38:34.692795069 +0000 UTC m=+3.300258084 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7-metrics-certs") pod "network-metrics-daemon-sbnql" (UID: "86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 12:38:33.710143 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.710123    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hmjl6"
Jul 02 12:38:33.712187 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:33.712172    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-88.ec2.internal\" not found"
Jul 02 12:38:33.792352 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.792327    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f76d4\" (UniqueName: \"kubernetes.io/projected/86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7-kube-api-access-f76d4\") pod \"network-metrics-daemon-sbnql\" (UID: \"86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7\") " pod="openshift-multus/network-metrics-daemon-sbnql"
Jul 02 12:38:33.792839 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.792416    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:33.793421 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.793395    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:33.793539 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.793426    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:33.793539 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.793451    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:33.795897 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.795878    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f76d4\" (UniqueName: \"kubernetes.io/projected/86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7-kube-api-access-f76d4\") pod \"network-metrics-daemon-sbnql\" (UID: \"86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7\") " pod="openshift-multus/network-metrics-daemon-sbnql"
Jul 02 12:38:33.812705 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:33.812674    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-88.ec2.internal\" not found"
Jul 02 12:38:33.893695 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.893624    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zvwz\" (UniqueName: \"kubernetes.io/projected/ebc69c1d-a909-44d0-bcb1-691e39c5e754-kube-api-access-2zvwz\") pod \"network-check-target-tczr2\" (UID: \"ebc69c1d-a909-44d0-bcb1-691e39c5e754\") " pod="openshift-network-diagnostics/network-check-target-tczr2"
Jul 02 12:38:33.893905 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.893724    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:33.894674 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.894654    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:33.894791 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.894687    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:33.894791 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.894701    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:33.894891 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:33.894842    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 12:38:33.894891 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:33.894866    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 12:38:33.894891 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:33.894881    2576 projected.go:196] Error preparing data for projected volume kube-api-access-2zvwz for pod openshift-network-diagnostics/network-check-target-tczr2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 12:38:33.895068 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:33.894967    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ebc69c1d-a909-44d0-bcb1-691e39c5e754-kube-api-access-2zvwz podName:ebc69c1d-a909-44d0-bcb1-691e39c5e754 nodeName:}" failed. No retries permitted until 2026-07-02 12:38:34.894944047 +0000 UTC m=+3.502407271 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2zvwz" (UniqueName: "kubernetes.io/projected/ebc69c1d-a909-44d0-bcb1-691e39c5e754-kube-api-access-2zvwz") pod "network-check-target-tczr2" (UID: "ebc69c1d-a909-44d0-bcb1-691e39c5e754") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 12:38:33.913596 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:33.913571    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-88.ec2.internal\" not found"
Jul 02 12:38:33.994556 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.994523    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ba833ad-b709-4ffd-85be-2e2c9cf867e6-original-pull-secret\") pod \"global-pull-secret-syncer-xl5pq\" (UID: \"2ba833ad-b709-4ffd-85be-2e2c9cf867e6\") " pod="kube-system/global-pull-secret-syncer-xl5pq"
Jul 02 12:38:33.994702 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.994641    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 12:38:33.995533 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.995515    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 12:38:33.995646 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.995544    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 12:38:33.995646 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:33.995555    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID"
Jul 02 12:38:33.997995 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:33.996024    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 12:38:33.997995 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:33.996101    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ba833ad-b709-4ffd-85be-2e2c9cf867e6-original-pull-secret podName:2ba833ad-b709-4ffd-85be-2e2c9cf867e6 nodeName:}" failed. No retries permitted until 2026-07-02 12:38:34.99607197 +0000 UTC m=+3.603534972 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ba833ad-b709-4ffd-85be-2e2c9cf867e6-original-pull-secret") pod "global-pull-secret-syncer-xl5pq" (UID: "2ba833ad-b709-4ffd-85be-2e2c9cf867e6") : object "kube-system"/"original-pull-secret" not registered
Jul 02 12:38:34.014148 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:34.014128    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-88.ec2.internal\" not found"
Jul 02 12:38:34.022444 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:34.022409    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1138068_d606_49cf_b424_b778436f5ab6.slice/crio-fcce2a05fee0d6ab9e1dc250ffe0771fe77af4e197d3a682d71441ff4bcb9e30 WatchSource:0}: Error finding container fcce2a05fee0d6ab9e1dc250ffe0771fe77af4e197d3a682d71441ff4bcb9e30: Status 404 returned error can't find the container with id fcce2a05fee0d6ab9e1dc250ffe0771fe77af4e197d3a682d71441ff4bcb9e30
Jul 02 12:38:34.022900 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:34.022881    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f05da72d6558d5d49cf0691bccf02a9.slice/crio-cdde14e78c4ffeb71290e0145a6939c3ae2f821ec7abd8dce05c0ed06af539b4 WatchSource:0}: Error finding container cdde14e78c4ffeb71290e0145a6939c3ae2f821ec7abd8dce05c0ed06af539b4: Status 404 returned error can't find the container with id cdde14e78c4ffeb71290e0145a6939c3ae2f821ec7abd8dce05c0ed06af539b4
Jul 02 12:38:34.023474 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:34.023451    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7c8733428fcf3eb08cd473039be930f.slice/crio-71b20f5922e8a524d560ddfa02b3060065ecdd5355bfb74b58d52409d9f8a8a9 WatchSource:0}: Error finding container 71b20f5922e8a524d560ddfa02b3060065ecdd5355bfb74b58d52409d9f8a8a9: Status 404 returned error can't find the container with id 71b20f5922e8a524d560ddfa02b3060065ecdd5355bfb74b58d52409d9f8a8a9
Jul 02 12:38:34.024364 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:34.024342    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f9139f3_1b94_4ddf_854a_53db4e0693f1.slice/crio-05c81a93b49402e27e99d3f3171a455f05d07b1b703c90ac871b72632f73c3ce WatchSource:0}: Error finding container 05c81a93b49402e27e99d3f3171a455f05d07b1b703c90ac871b72632f73c3ce: Status 404 returned error can't find the container with id 05c81a93b49402e27e99d3f3171a455f05d07b1b703c90ac871b72632f73c3ce
Jul 02 12:38:34.027999 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:34.027810    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 12:38:34.070416 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:34.070393    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d27224c_42a6_4708_a061_1705cbdba994.slice/crio-815c10a77d2c3c4135b5c56cb62b6802be197de0738251da8eb91273d387be92 WatchSource:0}: Error finding container 815c10a77d2c3c4135b5c56cb62b6802be197de0738251da8eb91273d387be92: Status 404 returned error can't find the container with id 815c10a77d2c3c4135b5c56cb62b6802be197de0738251da8eb91273d387be92
Jul 02 12:38:34.079517 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:34.079496    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb22eeab_d1d6_47f7_be7f_0b16dc5de6ef.slice/crio-038749b46e5b90b7aa5e6e952b32cda6fe8f2137e7e9f284019a507cd3b3de7d WatchSource:0}: Error finding container 038749b46e5b90b7aa5e6e952b32cda6fe8f2137e7e9f284019a507cd3b3de7d: Status 404 returned error can't find the container with id 038749b46e5b90b7aa5e6e952b32cda6fe8f2137e7e9f284019a507cd3b3de7d
Jul 02 12:38:34.105691 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:34.105669    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f219618_c523_45c1_9042_7e46c4994e49.slice/crio-e53a1aa8ee03850f3297d9244baa9d566defac10559bdd859aa498e7e8a8e765 WatchSource:0}: Error finding container e53a1aa8ee03850f3297d9244baa9d566defac10559bdd859aa498e7e8a8e765: Status 404 returned error can't find the container with id e53a1aa8ee03850f3297d9244baa9d566defac10559bdd859aa498e7e8a8e765
Jul 02 12:38:34.114986 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:34.114962    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-88.ec2.internal\" not found"
Jul 02 12:38:34.160164 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:34.160136    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod116c84eb_0c3e_4bfc_a1ea_bec6c7fb9a9a.slice/crio-037f68bb0d30aa49a9342c4fa9372c56540e00a579481381e103b054373543f4 WatchSource:0}: Error finding container 037f68bb0d30aa49a9342c4fa9372c56540e00a579481381e103b054373543f4: Status 404 returned error can't find the container with id 037f68bb0d30aa49a9342c4fa9372c56540e00a579481381e103b054373543f4
Jul 02 12:38:34.180338 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:34.180311    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6652ccdf_a3c9_42f8_88fc_39690889dd42.slice/crio-ec73683297e5fcf8e441455d12d424804184eb3a3dfe3961ae56ba366ea30b4a WatchSource:0}: Error finding container ec73683297e5fcf8e441455d12d424804184eb3a3dfe3961ae56ba366ea30b4a: Status 404 returned error can't find the container with id ec73683297e5fcf8e441455d12d424804184eb3a3dfe3961ae56ba366ea30b4a
Jul 02 12:38:34.183145 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:34.183124    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa04af49_8c6c_46fa_bfcc_500d89b526c9.slice/crio-ec9634bd0885fe325f908ba1bc9ce8880c6a59dafa55b57d8d4316df9eb7f4ff WatchSource:0}: Error finding container ec9634bd0885fe325f908ba1bc9ce8880c6a59dafa55b57d8d4316df9eb7f4ff: Status 404 returned error can't find the container with id ec9634bd0885fe325f908ba1bc9ce8880c6a59dafa55b57d8d4316df9eb7f4ff
Jul 02 12:38:34.183802 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:38:34.183727    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64db6250_f644_4f6d_856d_f3186f40db09.slice/crio-e49a727aede70b7aeb1ba3728d096d5004f414cbeb9a6b26828080e68f17bc60 WatchSource:0}: Error finding container e49a727aede70b7aeb1ba3728d096d5004f414cbeb9a6b26828080e68f17bc60: Status 404 returned error can't find the container with id e49a727aede70b7aeb1ba3728d096d5004f414cbeb9a6b26828080e68f17bc60
Jul 02 12:38:34.185609 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:34.185574    2576 kuberuntime_manager.go:1459] "Unhandled Error" err="init container egress-router-binary-copy start failed in pod multus-additional-cni-plugins-hh6v9_openshift-multus(64db6250-f644-4f6d-856d-f3186f40db09): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Jul 02 12:38:34.185709 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:34.185613    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"egress-router-binary-copy\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-multus/multus-additional-cni-plugins-hh6v9" podUID="64db6250-f644-4f6d-856d-f3186f40db09"
Jul 02 12:38:34.192063 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:34.192043    2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 12:38:34.229460 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:34.229441    2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 12:38:34.281896 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:34.281868    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-88.ec2.internal"
Jul 02 12:38:34.296779 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:34.296758    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 02 12:38:34.296893 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:34.296838    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-88.ec2.internal"
Jul 02 12:38:34.297390 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:34.297373    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-88.ec2.internal"]
Jul 02 12:38:34.306797 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:34.306775    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 02 12:38:34.307487 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:34.307470    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-143-88.ec2.internal"]
Jul 02 12:38:34.699950 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:34.699407    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7-metrics-certs\") pod \"network-metrics-daemon-sbnql\" (UID: \"86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7\") " pod="openshift-multus/network-metrics-daemon-sbnql"
Jul 02 12:38:34.699950 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:34.699584    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 12:38:34.699950 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:34.699646    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7-metrics-certs podName:86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7 nodeName:}" failed. No retries permitted until 2026-07-02 12:38:36.699629455 +0000 UTC m=+5.307092461 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7-metrics-certs") pod "network-metrics-daemon-sbnql" (UID: "86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 12:38:34.783180 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:34.782976    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 02 12:38:34.794610 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:34.794572    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 02 12:38:34.817980 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:34.817953    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fzjxs"
Jul 02 12:38:34.826484 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:34.826459    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fzjxs"
Jul 02 12:38:34.857535 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:34.857508    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tczr2"
Jul 02 12:38:34.857690 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:34.857633    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tczr2" podUID="ebc69c1d-a909-44d0-bcb1-691e39c5e754"
Jul 02 12:38:34.858085 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:34.858065    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sbnql"
Jul 02 12:38:34.858190 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:34.858168    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sbnql" podUID="86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7"
Jul 02 12:38:34.884630 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:34.884574    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mztvs" event={"ID":"5f219618-c523-45c1-9042-7e46c4994e49","Type":"ContainerStarted","Data":"e53a1aa8ee03850f3297d9244baa9d566defac10559bdd859aa498e7e8a8e765"}
Jul 02 12:38:34.892472 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:34.892444    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hh6v9" event={"ID":"64db6250-f644-4f6d-856d-f3186f40db09","Type":"ContainerStarted","Data":"e49a727aede70b7aeb1ba3728d096d5004f414cbeb9a6b26828080e68f17bc60"}
Jul 02 12:38:34.897415 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:34.897384    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"egress-router-binary-copy\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3135760793a0f16b958eb7c71615b417dd6b0d707b9f31aa5acc0d0df96ecec\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-multus/multus-additional-cni-plugins-hh6v9" podUID="64db6250-f644-4f6d-856d-f3186f40db09"
Jul 02 12:38:34.901584 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:34.901562    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2ba833ad-b709-4ffd-85be-2e2c9cf867e6-kubelet-config\") pod \"2ba833ad-b709-4ffd-85be-2e2c9cf867e6\" (UID: \"2ba833ad-b709-4ffd-85be-2e2c9cf867e6\") "
Jul 02 12:38:34.901688 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:34.901625    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2ba833ad-b709-4ffd-85be-2e2c9cf867e6-dbus\") pod \"2ba833ad-b709-4ffd-85be-2e2c9cf867e6\" (UID: \"2ba833ad-b709-4ffd-85be-2e2c9cf867e6\") "
Jul 02 12:38:34.901819 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:34.901803    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zvwz\" (UniqueName: \"kubernetes.io/projected/ebc69c1d-a909-44d0-bcb1-691e39c5e754-kube-api-access-2zvwz\") pod \"network-check-target-tczr2\" (UID: \"ebc69c1d-a909-44d0-bcb1-691e39c5e754\") " pod="openshift-network-diagnostics/network-check-target-tczr2"
Jul 02 12:38:34.902171 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:34.902137    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/2ba833ad-b709-4ffd-85be-2e2c9cf867e6-kubelet-config" (OuterVolumeSpecName: "kubelet-config") pod "2ba833ad-b709-4ffd-85be-2e2c9cf867e6" (UID: "2ba833ad-b709-4ffd-85be-2e2c9cf867e6"). InnerVolumeSpecName "kubelet-config". PluginName "kubernetes.io/host-path", VolumeGIDValue ""
Jul 02 12:38:34.902260 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:34.902171    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/2ba833ad-b709-4ffd-85be-2e2c9cf867e6-dbus" (OuterVolumeSpecName: "dbus") pod "2ba833ad-b709-4ffd-85be-2e2c9cf867e6" (UID: "2ba833ad-b709-4ffd-85be-2e2c9cf867e6"). InnerVolumeSpecName "dbus". PluginName "kubernetes.io/host-path", VolumeGIDValue ""
Jul 02 12:38:34.902859 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:34.902842    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 12:38:34.902965 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:34.902865    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 12:38:34.902965 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:34.902876    2576 projected.go:196] Error preparing data for projected volume kube-api-access-2zvwz for pod openshift-network-diagnostics/network-check-target-tczr2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 12:38:34.902965 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:34.902944    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ebc69c1d-a909-44d0-bcb1-691e39c5e754-kube-api-access-2zvwz podName:ebc69c1d-a909-44d0-bcb1-691e39c5e754 nodeName:}" failed. No retries permitted until 2026-07-02 12:38:36.902925647 +0000 UTC m=+5.510388655 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2zvwz" (UniqueName: "kubernetes.io/projected/ebc69c1d-a909-44d0-bcb1-691e39c5e754-kube-api-access-2zvwz") pod "network-check-target-tczr2" (UID: "ebc69c1d-a909-44d0-bcb1-691e39c5e754") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 12:38:34.905489 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:34.905456    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fzrpv" event={"ID":"6652ccdf-a3c9-42f8-88fc-39690889dd42","Type":"ContainerStarted","Data":"ec73683297e5fcf8e441455d12d424804184eb3a3dfe3961ae56ba366ea30b4a"}
Jul 02 12:38:34.909611 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:34.909588    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4bj55" event={"ID":"bb22eeab-d1d6-47f7-be7f-0b16dc5de6ef","Type":"ContainerStarted","Data":"038749b46e5b90b7aa5e6e952b32cda6fe8f2137e7e9f284019a507cd3b3de7d"}
Jul 02 12:38:34.923053 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:34.923029    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cltnv" event={"ID":"3d27224c-42a6-4708-a061-1705cbdba994","Type":"ContainerStarted","Data":"815c10a77d2c3c4135b5c56cb62b6802be197de0738251da8eb91273d387be92"}
Jul 02 12:38:34.938694 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:34.938659    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-88.ec2.internal" event={"ID":"f7c8733428fcf3eb08cd473039be930f","Type":"ContainerStarted","Data":"71b20f5922e8a524d560ddfa02b3060065ecdd5355bfb74b58d52409d9f8a8a9"}
Jul 02 12:38:34.943562 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:34.943527    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f9pzc" event={"ID":"a1138068-d606-49cf-b424-b778436f5ab6","Type":"ContainerStarted","Data":"fcce2a05fee0d6ab9e1dc250ffe0771fe77af4e197d3a682d71441ff4bcb9e30"}
Jul 02 12:38:34.945570 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:34.945539    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t6qwm" event={"ID":"7f9139f3-1b94-4ddf-854a-53db4e0693f1","Type":"ContainerStarted","Data":"05c81a93b49402e27e99d3f3171a455f05d07b1b703c90ac871b72632f73c3ce"}
Jul 02 12:38:34.953615 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:34.953540    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-88.ec2.internal" event={"ID":"6f05da72d6558d5d49cf0691bccf02a9","Type":"ContainerStarted","Data":"cdde14e78c4ffeb71290e0145a6939c3ae2f821ec7abd8dce05c0ed06af539b4"}
Jul 02 12:38:34.985660 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:34.985622    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hmjl6" event={"ID":"fa04af49-8c6c-46fa-bfcc-500d89b526c9","Type":"ContainerStarted","Data":"ec9634bd0885fe325f908ba1bc9ce8880c6a59dafa55b57d8d4316df9eb7f4ff"}
Jul 02 12:38:34.996603 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:34.996480    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks" event={"ID":"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a","Type":"ContainerStarted","Data":"037f68bb0d30aa49a9342c4fa9372c56540e00a579481381e103b054373543f4"}
Jul 02 12:38:35.003146 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:35.002563    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ba833ad-b709-4ffd-85be-2e2c9cf867e6-original-pull-secret\") pod \"global-pull-secret-syncer-xl5pq\" (UID: \"2ba833ad-b709-4ffd-85be-2e2c9cf867e6\") " pod="kube-system/global-pull-secret-syncer-xl5pq"
Jul 02 12:38:35.003146 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:35.002621    2576 reconciler_common.go:299] "Volume detached for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2ba833ad-b709-4ffd-85be-2e2c9cf867e6-dbus\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:38:35.003146 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:35.002638    2576 reconciler_common.go:299] "Volume detached for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2ba833ad-b709-4ffd-85be-2e2c9cf867e6-kubelet-config\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:38:35.003146 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:35.002744    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 12:38:35.003146 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:35.002798    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ba833ad-b709-4ffd-85be-2e2c9cf867e6-original-pull-secret podName:2ba833ad-b709-4ffd-85be-2e2c9cf867e6 nodeName:}" failed. No retries permitted until 2026-07-02 12:38:37.002780884 +0000 UTC m=+5.610243890 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ba833ad-b709-4ffd-85be-2e2c9cf867e6-original-pull-secret") pod "global-pull-secret-syncer-xl5pq" (UID: "2ba833ad-b709-4ffd-85be-2e2c9cf867e6") : object "kube-system"/"original-pull-secret" not registered
Jul 02 12:38:35.830004 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:35.829899    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 12:33:34 +0000 UTC" deadline="2028-01-11 15:09:17.690370851 +0000 UTC"
Jul 02 12:38:35.830004 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:35.829968    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13394h30m41.86040768s"
Jul 02 12:38:36.722418 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:36.722320    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7-metrics-certs\") pod \"network-metrics-daemon-sbnql\" (UID: \"86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7\") " pod="openshift-multus/network-metrics-daemon-sbnql"
Jul 02 12:38:36.722598 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:36.722537    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 12:38:36.722658 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:36.722604    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7-metrics-certs podName:86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7 nodeName:}" failed. No retries permitted until 2026-07-02 12:38:40.722584845 +0000 UTC m=+9.330047859 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7-metrics-certs") pod "network-metrics-daemon-sbnql" (UID: "86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 12:38:36.830858 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:36.830809    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 12:33:34 +0000 UTC" deadline="2028-02-03 05:41:36.795648819 +0000 UTC"
Jul 02 12:38:36.830858 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:36.830850    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13937h2m59.964803128s"
Jul 02 12:38:36.866375 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:36.866341    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tczr2"
Jul 02 12:38:36.866375 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:36.866364    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sbnql"
Jul 02 12:38:36.866656 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:36.866466    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tczr2" podUID="ebc69c1d-a909-44d0-bcb1-691e39c5e754"
Jul 02 12:38:36.866716 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:36.866663    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sbnql" podUID="86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7"
Jul 02 12:38:36.925274 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:36.924614    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zvwz\" (UniqueName: \"kubernetes.io/projected/ebc69c1d-a909-44d0-bcb1-691e39c5e754-kube-api-access-2zvwz\") pod \"network-check-target-tczr2\" (UID: \"ebc69c1d-a909-44d0-bcb1-691e39c5e754\") " pod="openshift-network-diagnostics/network-check-target-tczr2"
Jul 02 12:38:36.925274 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:36.924777    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 12:38:36.925274 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:36.924802    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 12:38:36.925274 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:36.924816    2576 projected.go:196] Error preparing data for projected volume kube-api-access-2zvwz for pod openshift-network-diagnostics/network-check-target-tczr2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 12:38:36.925274 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:36.924900    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ebc69c1d-a909-44d0-bcb1-691e39c5e754-kube-api-access-2zvwz podName:ebc69c1d-a909-44d0-bcb1-691e39c5e754 nodeName:}" failed. No retries permitted until 2026-07-02 12:38:40.924881582 +0000 UTC m=+9.532344587 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2zvwz" (UniqueName: "kubernetes.io/projected/ebc69c1d-a909-44d0-bcb1-691e39c5e754-kube-api-access-2zvwz") pod "network-check-target-tczr2" (UID: "ebc69c1d-a909-44d0-bcb1-691e39c5e754") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 12:38:37.025229 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:37.025130    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ba833ad-b709-4ffd-85be-2e2c9cf867e6-original-pull-secret\") pod \"global-pull-secret-syncer-xl5pq\" (UID: \"2ba833ad-b709-4ffd-85be-2e2c9cf867e6\") " pod="kube-system/global-pull-secret-syncer-xl5pq"
Jul 02 12:38:37.025399 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:37.025327    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 12:38:37.025399 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:37.025393    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ba833ad-b709-4ffd-85be-2e2c9cf867e6-original-pull-secret podName:2ba833ad-b709-4ffd-85be-2e2c9cf867e6 nodeName:}" failed. No retries permitted until 2026-07-02 12:38:41.025374557 +0000 UTC m=+9.632837560 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ba833ad-b709-4ffd-85be-2e2c9cf867e6-original-pull-secret") pod "global-pull-secret-syncer-xl5pq" (UID: "2ba833ad-b709-4ffd-85be-2e2c9cf867e6") : object "kube-system"/"original-pull-secret" not registered
Jul 02 12:38:38.857026 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:38.856997    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sbnql"
Jul 02 12:38:38.857497 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:38.857184    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sbnql" podUID="86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7"
Jul 02 12:38:38.857497 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:38.857255    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tczr2"
Jul 02 12:38:38.857497 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:38.857368    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tczr2" podUID="ebc69c1d-a909-44d0-bcb1-691e39c5e754"
Jul 02 12:38:40.757579 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:40.756898    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7-metrics-certs\") pod \"network-metrics-daemon-sbnql\" (UID: \"86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7\") " pod="openshift-multus/network-metrics-daemon-sbnql"
Jul 02 12:38:40.757579 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:40.757106    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 12:38:40.757579 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:40.757177    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7-metrics-certs podName:86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7 nodeName:}" failed. No retries permitted until 2026-07-02 12:38:48.75715684 +0000 UTC m=+17.364619865 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7-metrics-certs") pod "network-metrics-daemon-sbnql" (UID: "86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 12:38:40.857948 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:40.857657    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sbnql"
Jul 02 12:38:40.857948 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:40.857827    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sbnql" podUID="86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7"
Jul 02 12:38:40.858193 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:40.858174    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tczr2"
Jul 02 12:38:40.858350 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:40.858319    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tczr2" podUID="ebc69c1d-a909-44d0-bcb1-691e39c5e754"
Jul 02 12:38:40.958816 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:40.958780    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zvwz\" (UniqueName: \"kubernetes.io/projected/ebc69c1d-a909-44d0-bcb1-691e39c5e754-kube-api-access-2zvwz\") pod \"network-check-target-tczr2\" (UID: \"ebc69c1d-a909-44d0-bcb1-691e39c5e754\") " pod="openshift-network-diagnostics/network-check-target-tczr2"
Jul 02 12:38:40.959026 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:40.959008    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 12:38:40.959104 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:40.959041    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 12:38:40.959104 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:40.959054    2576 projected.go:196] Error preparing data for projected volume kube-api-access-2zvwz for pod openshift-network-diagnostics/network-check-target-tczr2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 12:38:40.959207 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:40.959113    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ebc69c1d-a909-44d0-bcb1-691e39c5e754-kube-api-access-2zvwz podName:ebc69c1d-a909-44d0-bcb1-691e39c5e754 nodeName:}" failed. No retries permitted until 2026-07-02 12:38:48.959095382 +0000 UTC m=+17.566558384 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2zvwz" (UniqueName: "kubernetes.io/projected/ebc69c1d-a909-44d0-bcb1-691e39c5e754-kube-api-access-2zvwz") pod "network-check-target-tczr2" (UID: "ebc69c1d-a909-44d0-bcb1-691e39c5e754") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 12:38:41.060091 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:41.059971    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ba833ad-b709-4ffd-85be-2e2c9cf867e6-original-pull-secret\") pod \"global-pull-secret-syncer-xl5pq\" (UID: \"2ba833ad-b709-4ffd-85be-2e2c9cf867e6\") " pod="kube-system/global-pull-secret-syncer-xl5pq"
Jul 02 12:38:41.060261 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:41.060142    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 12:38:41.060261 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:41.060216    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ba833ad-b709-4ffd-85be-2e2c9cf867e6-original-pull-secret podName:2ba833ad-b709-4ffd-85be-2e2c9cf867e6 nodeName:}" failed. No retries permitted until 2026-07-02 12:38:49.060196558 +0000 UTC m=+17.667659563 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ba833ad-b709-4ffd-85be-2e2c9cf867e6-original-pull-secret") pod "global-pull-secret-syncer-xl5pq" (UID: "2ba833ad-b709-4ffd-85be-2e2c9cf867e6") : object "kube-system"/"original-pull-secret" not registered
Jul 02 12:38:42.860461 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:42.859975    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sbnql"
Jul 02 12:38:42.860461 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:42.860098    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sbnql" podUID="86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7"
Jul 02 12:38:42.860461 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:42.860226    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tczr2"
Jul 02 12:38:42.860461 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:42.860343    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tczr2" podUID="ebc69c1d-a909-44d0-bcb1-691e39c5e754"
Jul 02 12:38:42.878443 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:42.878419    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kube-system/global-pull-secret-syncer-xl5pq"]
Jul 02 12:38:42.878636 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:42.878614    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl5pq"
Jul 02 12:38:42.882113 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:42.881964    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kube-system/global-pull-secret-syncer-xl5pq"]
Jul 02 12:38:42.884422 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:42.884393    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"global-pull-secret-syncer-xl5pq\" is forbidden: User \"system:node:ip-10-0-143-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kube-system\": no relationship found between node 'ip-10-0-143-88.ec2.internal' and this object" podUID="2ba833ad-b709-4ffd-85be-2e2c9cf867e6" pod="kube-system/global-pull-secret-syncer-xl5pq"
Jul 02 12:38:42.904010 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:42.903335    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-72n2l"]
Jul 02 12:38:42.909593 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:42.909477    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-72n2l"
Jul 02 12:38:42.909593 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:42.909560    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-72n2l" podUID="116aa48e-717b-4d49-897d-5538398f4a2a"
Jul 02 12:38:42.927057 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:42.927005    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"global-pull-secret-syncer-xl5pq\" is forbidden: User \"system:node:ip-10-0-143-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kube-system\": no relationship found between node 'ip-10-0-143-88.ec2.internal' and this object" podUID="2ba833ad-b709-4ffd-85be-2e2c9cf867e6" pod="kube-system/global-pull-secret-syncer-xl5pq"
Jul 02 12:38:42.977709 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:42.977666    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/116aa48e-717b-4d49-897d-5538398f4a2a-original-pull-secret\") pod \"global-pull-secret-syncer-72n2l\" (UID: \"116aa48e-717b-4d49-897d-5538398f4a2a\") " pod="kube-system/global-pull-secret-syncer-72n2l"
Jul 02 12:38:42.977869 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:42.977717    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/116aa48e-717b-4d49-897d-5538398f4a2a-kubelet-config\") pod \"global-pull-secret-syncer-72n2l\" (UID: \"116aa48e-717b-4d49-897d-5538398f4a2a\") " pod="kube-system/global-pull-secret-syncer-72n2l"
Jul 02 12:38:42.977869 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:42.977744    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/116aa48e-717b-4d49-897d-5538398f4a2a-dbus\") pod \"global-pull-secret-syncer-72n2l\" (UID: \"116aa48e-717b-4d49-897d-5538398f4a2a\") " pod="kube-system/global-pull-secret-syncer-72n2l"
Jul 02 12:38:42.977869 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:42.977787    2576 reconciler_common.go:299] "Volume detached for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ba833ad-b709-4ffd-85be-2e2c9cf867e6-original-pull-secret\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:38:43.078301 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:43.078264    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/116aa48e-717b-4d49-897d-5538398f4a2a-original-pull-secret\") pod \"global-pull-secret-syncer-72n2l\" (UID: \"116aa48e-717b-4d49-897d-5538398f4a2a\") " pod="kube-system/global-pull-secret-syncer-72n2l"
Jul 02 12:38:43.078301 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:43.078313    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/116aa48e-717b-4d49-897d-5538398f4a2a-kubelet-config\") pod \"global-pull-secret-syncer-72n2l\" (UID: \"116aa48e-717b-4d49-897d-5538398f4a2a\") " pod="kube-system/global-pull-secret-syncer-72n2l"
Jul 02 12:38:43.078536 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:43.078344    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/116aa48e-717b-4d49-897d-5538398f4a2a-dbus\") pod \"global-pull-secret-syncer-72n2l\" (UID: \"116aa48e-717b-4d49-897d-5538398f4a2a\") " pod="kube-system/global-pull-secret-syncer-72n2l"
Jul 02 12:38:43.078536 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:43.078428    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 12:38:43.078536 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:43.078449    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/116aa48e-717b-4d49-897d-5538398f4a2a-kubelet-config\") pod \"global-pull-secret-syncer-72n2l\" (UID: \"116aa48e-717b-4d49-897d-5538398f4a2a\") " pod="kube-system/global-pull-secret-syncer-72n2l"
Jul 02 12:38:43.078536 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:43.078497    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/116aa48e-717b-4d49-897d-5538398f4a2a-original-pull-secret podName:116aa48e-717b-4d49-897d-5538398f4a2a nodeName:}" failed. No retries permitted until 2026-07-02 12:38:43.578478641 +0000 UTC m=+12.185941648 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/116aa48e-717b-4d49-897d-5538398f4a2a-original-pull-secret") pod "global-pull-secret-syncer-72n2l" (UID: "116aa48e-717b-4d49-897d-5538398f4a2a") : object "kube-system"/"original-pull-secret" not registered
Jul 02 12:38:43.078536 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:43.078509    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/116aa48e-717b-4d49-897d-5538398f4a2a-dbus\") pod \"global-pull-secret-syncer-72n2l\" (UID: \"116aa48e-717b-4d49-897d-5538398f4a2a\") " pod="kube-system/global-pull-secret-syncer-72n2l"
Jul 02 12:38:43.583747 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:43.583709    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/116aa48e-717b-4d49-897d-5538398f4a2a-original-pull-secret\") pod \"global-pull-secret-syncer-72n2l\" (UID: \"116aa48e-717b-4d49-897d-5538398f4a2a\") " pod="kube-system/global-pull-secret-syncer-72n2l"
Jul 02 12:38:43.583953 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:43.583879    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 12:38:43.584031 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:43.583960    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/116aa48e-717b-4d49-897d-5538398f4a2a-original-pull-secret podName:116aa48e-717b-4d49-897d-5538398f4a2a nodeName:}" failed. No retries permitted until 2026-07-02 12:38:44.583940984 +0000 UTC m=+13.191404001 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/116aa48e-717b-4d49-897d-5538398f4a2a-original-pull-secret") pod "global-pull-secret-syncer-72n2l" (UID: "116aa48e-717b-4d49-897d-5538398f4a2a") : object "kube-system"/"original-pull-secret" not registered
Jul 02 12:38:44.593463 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:44.593432    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/116aa48e-717b-4d49-897d-5538398f4a2a-original-pull-secret\") pod \"global-pull-secret-syncer-72n2l\" (UID: \"116aa48e-717b-4d49-897d-5538398f4a2a\") " pod="kube-system/global-pull-secret-syncer-72n2l"
Jul 02 12:38:44.593812 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:44.593535    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 12:38:44.593812 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:44.593583    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/116aa48e-717b-4d49-897d-5538398f4a2a-original-pull-secret podName:116aa48e-717b-4d49-897d-5538398f4a2a nodeName:}" failed. No retries permitted until 2026-07-02 12:38:46.593570465 +0000 UTC m=+15.201033466 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/116aa48e-717b-4d49-897d-5538398f4a2a-original-pull-secret") pod "global-pull-secret-syncer-72n2l" (UID: "116aa48e-717b-4d49-897d-5538398f4a2a") : object "kube-system"/"original-pull-secret" not registered
Jul 02 12:38:44.857700 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:44.857629    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tczr2"
Jul 02 12:38:44.857700 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:44.857660    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-72n2l"
Jul 02 12:38:44.857876 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:44.857630    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sbnql"
Jul 02 12:38:44.857876 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:44.857788    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tczr2" podUID="ebc69c1d-a909-44d0-bcb1-691e39c5e754"
Jul 02 12:38:44.857876 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:44.857858    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sbnql" podUID="86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7"
Jul 02 12:38:44.858034 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:44.858000    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-72n2l" podUID="116aa48e-717b-4d49-897d-5538398f4a2a"
Jul 02 12:38:46.607144 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:46.607110    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/116aa48e-717b-4d49-897d-5538398f4a2a-original-pull-secret\") pod \"global-pull-secret-syncer-72n2l\" (UID: \"116aa48e-717b-4d49-897d-5538398f4a2a\") " pod="kube-system/global-pull-secret-syncer-72n2l"
Jul 02 12:38:46.607541 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:46.607247    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 12:38:46.607541 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:46.607306    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/116aa48e-717b-4d49-897d-5538398f4a2a-original-pull-secret podName:116aa48e-717b-4d49-897d-5538398f4a2a nodeName:}" failed. No retries permitted until 2026-07-02 12:38:50.607289033 +0000 UTC m=+19.214752035 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/116aa48e-717b-4d49-897d-5538398f4a2a-original-pull-secret") pod "global-pull-secret-syncer-72n2l" (UID: "116aa48e-717b-4d49-897d-5538398f4a2a") : object "kube-system"/"original-pull-secret" not registered
Jul 02 12:38:46.856829 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:46.856792    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-72n2l"
Jul 02 12:38:46.856829 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:46.856792    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tczr2"
Jul 02 12:38:46.857088 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:46.856939    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sbnql"
Jul 02 12:38:46.857144 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:46.857093    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-72n2l" podUID="116aa48e-717b-4d49-897d-5538398f4a2a"
Jul 02 12:38:46.857239 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:46.857174    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sbnql" podUID="86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7"
Jul 02 12:38:46.857239 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:46.857214    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tczr2" podUID="ebc69c1d-a909-44d0-bcb1-691e39c5e754"
Jul 02 12:38:48.821802 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:48.821546    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7-metrics-certs\") pod \"network-metrics-daemon-sbnql\" (UID: \"86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7\") " pod="openshift-multus/network-metrics-daemon-sbnql"
Jul 02 12:38:48.822264 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:48.821726    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 12:38:48.822264 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:48.821982    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7-metrics-certs podName:86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:04.821952221 +0000 UTC m=+33.429415241 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7-metrics-certs") pod "network-metrics-daemon-sbnql" (UID: "86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 12:38:48.857063 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:48.857029    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tczr2"
Jul 02 12:38:48.857063 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:48.857066    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-72n2l"
Jul 02 12:38:48.857292 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:48.857029    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sbnql"
Jul 02 12:38:48.857292 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:48.857151    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tczr2" podUID="ebc69c1d-a909-44d0-bcb1-691e39c5e754"
Jul 02 12:38:48.857292 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:48.857242    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sbnql" podUID="86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7"
Jul 02 12:38:48.857451 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:48.857316    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-72n2l" podUID="116aa48e-717b-4d49-897d-5538398f4a2a"
Jul 02 12:38:49.022989 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:49.022955    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zvwz\" (UniqueName: \"kubernetes.io/projected/ebc69c1d-a909-44d0-bcb1-691e39c5e754-kube-api-access-2zvwz\") pod \"network-check-target-tczr2\" (UID: \"ebc69c1d-a909-44d0-bcb1-691e39c5e754\") " pod="openshift-network-diagnostics/network-check-target-tczr2"
Jul 02 12:38:49.023166 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:49.023146    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 12:38:49.023229 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:49.023170    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 12:38:49.023229 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:49.023183    2576 projected.go:196] Error preparing data for projected volume kube-api-access-2zvwz for pod openshift-network-diagnostics/network-check-target-tczr2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 12:38:49.023304 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:49.023245    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ebc69c1d-a909-44d0-bcb1-691e39c5e754-kube-api-access-2zvwz podName:ebc69c1d-a909-44d0-bcb1-691e39c5e754 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:05.023229299 +0000 UTC m=+33.630692303 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2zvwz" (UniqueName: "kubernetes.io/projected/ebc69c1d-a909-44d0-bcb1-691e39c5e754-kube-api-access-2zvwz") pod "network-check-target-tczr2" (UID: "ebc69c1d-a909-44d0-bcb1-691e39c5e754") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 12:38:50.632226 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:50.632193    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/116aa48e-717b-4d49-897d-5538398f4a2a-original-pull-secret\") pod \"global-pull-secret-syncer-72n2l\" (UID: \"116aa48e-717b-4d49-897d-5538398f4a2a\") " pod="kube-system/global-pull-secret-syncer-72n2l"
Jul 02 12:38:50.632692 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:50.632351    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 12:38:50.632692 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:50.632423    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/116aa48e-717b-4d49-897d-5538398f4a2a-original-pull-secret podName:116aa48e-717b-4d49-897d-5538398f4a2a nodeName:}" failed. No retries permitted until 2026-07-02 12:38:58.632402764 +0000 UTC m=+27.239865765 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/116aa48e-717b-4d49-897d-5538398f4a2a-original-pull-secret") pod "global-pull-secret-syncer-72n2l" (UID: "116aa48e-717b-4d49-897d-5538398f4a2a") : object "kube-system"/"original-pull-secret" not registered
Jul 02 12:38:50.857686 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:50.857648    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sbnql"
Jul 02 12:38:50.857868 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:50.857782    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sbnql" podUID="86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7"
Jul 02 12:38:50.857868 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:50.857826    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tczr2"
Jul 02 12:38:50.857868 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:50.857830    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-72n2l"
Jul 02 12:38:50.858057 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:50.857981    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tczr2" podUID="ebc69c1d-a909-44d0-bcb1-691e39c5e754"
Jul 02 12:38:50.858119 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:50.858060    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-72n2l" podUID="116aa48e-717b-4d49-897d-5538398f4a2a"
Jul 02 12:38:52.857348 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:52.857315    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-72n2l"
Jul 02 12:38:52.857704 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:52.857439    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-72n2l" podUID="116aa48e-717b-4d49-897d-5538398f4a2a"
Jul 02 12:38:52.857704 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:52.857514    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sbnql"
Jul 02 12:38:52.857704 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:52.857607    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sbnql" podUID="86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7"
Jul 02 12:38:52.857704 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:52.857638    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tczr2"
Jul 02 12:38:52.857849 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:52.857713    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tczr2" podUID="ebc69c1d-a909-44d0-bcb1-691e39c5e754"
Jul 02 12:38:54.052094 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:54.051857    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cltnv" event={"ID":"3d27224c-42a6-4708-a061-1705cbdba994","Type":"ContainerStarted","Data":"6b8b7502ad046baa5c81034a442a129c60b958f01d235143f98c6b116a69fba8"}
Jul 02 12:38:54.053557 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:54.053527    2576 generic.go:356] "Generic (PLEG): container finished" podID="f7c8733428fcf3eb08cd473039be930f" containerID="586b21b39556f62c2cbe3aa82c501c89f0a2ffaba59bd0f77dede032d5448b4f" exitCode=0
Jul 02 12:38:54.053674 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:54.053562    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-88.ec2.internal" event={"ID":"f7c8733428fcf3eb08cd473039be930f","Type":"ContainerDied","Data":"586b21b39556f62c2cbe3aa82c501c89f0a2ffaba59bd0f77dede032d5448b4f"}
Jul 02 12:38:54.055324 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:54.055301    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f9pzc" event={"ID":"a1138068-d606-49cf-b424-b778436f5ab6","Type":"ContainerStarted","Data":"e57fa4934b8c3b3462c2723f401c24d10943dd39014f4d6215363023156a7b61"}
Jul 02 12:38:54.056859 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:54.056827    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t6qwm" event={"ID":"7f9139f3-1b94-4ddf-854a-53db4e0693f1","Type":"ContainerStarted","Data":"839d44debcce3c433aa5c753ef8c2287e4ae6f237176c1c23fd083258b7051fa"}
Jul 02 12:38:54.059221 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:54.059194    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-88.ec2.internal" event={"ID":"6f05da72d6558d5d49cf0691bccf02a9","Type":"ContainerStarted","Data":"b3e9af6b998bb2934eb483b98fb132d441dc12a208248185b99a4cc1b8ff15b9"}
Jul 02 12:38:54.063221 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:54.063195    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hmjl6" event={"ID":"fa04af49-8c6c-46fa-bfcc-500d89b526c9","Type":"ContainerStarted","Data":"c5665d5748c900c99bab94cb942cc4429365b64aaf80aaec042a8748b39557b6"}
Jul 02 12:38:54.066832 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:54.066806    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2c7ks_116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a/ovn-acl-logging/0.log"
Jul 02 12:38:54.067496 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:54.067475    2576 generic.go:356] "Generic (PLEG): container finished" podID="116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a" containerID="6b0fb1a929a00f1611b127d4184ed30790de48957d6a700680aa4a5aae990616" exitCode=1
Jul 02 12:38:54.067602 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:54.067543    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks" event={"ID":"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a","Type":"ContainerStarted","Data":"d45da723a269e70b58c0dfb01e278f448ff6f9d8da477e30b079add645a78cf0"}
Jul 02 12:38:54.067672 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:54.067603    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks" event={"ID":"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a","Type":"ContainerStarted","Data":"0c57155f6a81778d1fab3cbb31a48256b3c735af9bc28ef66b975c8b450f4c4c"}
Jul 02 12:38:54.067672 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:54.067652    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks" event={"ID":"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a","Type":"ContainerStarted","Data":"30ff708b3d9620ac189658ec8ece30db97e97629a289f71f0762063c6349a1df"}
Jul 02 12:38:54.067768 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:54.067698    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks" event={"ID":"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a","Type":"ContainerStarted","Data":"c7a447d37427fd9d255fc1bc698331f09a62c4a629d784188cf170740c1d60cd"}
Jul 02 12:38:54.067768 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:54.067743    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks" event={"ID":"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a","Type":"ContainerDied","Data":"6b0fb1a929a00f1611b127d4184ed30790de48957d6a700680aa4a5aae990616"}
Jul 02 12:38:54.067828 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:54.067785    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks" event={"ID":"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a","Type":"ContainerStarted","Data":"8a73940a1ecaa1b410584e0fb7b30b0de838f45ec4a587a993fc57dfc96f360c"}
Jul 02 12:38:54.069714 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:54.069690    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mztvs" event={"ID":"5f219618-c523-45c1-9042-7e46c4994e49","Type":"ContainerStarted","Data":"bdf417e8964d18303ae6cce856b11a292477fc5622267673a85c5f02da9edfb4"}
Jul 02 12:38:54.071236 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:54.071215    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fzrpv" event={"ID":"6652ccdf-a3c9-42f8-88fc-39690889dd42","Type":"ContainerStarted","Data":"5e49f6ce41b3e35ea6bc1fdf4ba4ff98b6aef5c4495a72542bded76a2347e00e"}
Jul 02 12:38:54.073560 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:54.073518    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-f9pzc" podStartSLOduration=2.148257318 podStartE2EDuration="21.073508012s" totalImagesPullingTime="18.925250694s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 12:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:38:54.073364938 +0000 UTC m=+22.680827963" watchObservedRunningTime="2026-07-02 12:38:54.073508012 +0000 UTC m=+22.680971068"
Jul 02 12:38:54.087679 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:54.087632    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-t6qwm" podStartSLOduration=10.458867831 podStartE2EDuration="21.087615863s" totalImagesPullingTime="10.628748032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:38:54.087555716 +0000 UTC m=+22.695018742" watchObservedRunningTime="2026-07-02 12:38:54.087615863 +0000 UTC m=+22.695078888"
Jul 02 12:38:54.116084 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:54.116029    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fzrpv" podStartSLOduration=2.103024153 podStartE2EDuration="21.116010419s" totalImagesPullingTime="19.012986266s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 12:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:38:54.115901535 +0000 UTC m=+22.723364561" watchObservedRunningTime="2026-07-02 12:38:54.116010419 +0000 UTC m=+22.723473446"
Jul 02 12:38:54.136210 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:54.136167    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mztvs" podStartSLOduration=2.2915784280000002 podStartE2EDuration="21.136152453s" totalImagesPullingTime="18.844574025s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 12:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:38:54.135829906 +0000 UTC m=+22.743292931" watchObservedRunningTime="2026-07-02 12:38:54.136152453 +0000 UTC m=+22.743615475"
Jul 02 12:38:54.152107 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:54.152059    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hmjl6" podStartSLOduration=2.3854666079999998 podStartE2EDuration="21.15204524s" totalImagesPullingTime="18.766578632s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 12:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:38:54.151415505 +0000 UTC m=+22.758878529" watchObservedRunningTime="2026-07-02 12:38:54.15204524 +0000 UTC m=+22.759508263"
Jul 02 12:38:54.166261 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:54.166221    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-88.ec2.internal" podStartSLOduration=20.166209039 podStartE2EDuration="20.166209039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 12:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:38:54.165947987 +0000 UTC m=+22.773411011" watchObservedRunningTime="2026-07-02 12:38:54.166209039 +0000 UTC m=+22.773672062"
Jul 02 12:38:54.571268 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:54.571234    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-t6qwm"
Jul 02 12:38:54.571902 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:54.571882    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-t6qwm"
Jul 02 12:38:54.857368 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:54.857339    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sbnql"
Jul 02 12:38:54.857506 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:54.857369    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tczr2"
Jul 02 12:38:54.857506 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:54.857339    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-72n2l"
Jul 02 12:38:54.857506 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:54.857472    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sbnql" podUID="86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7"
Jul 02 12:38:54.857681 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:54.857531    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-72n2l" podUID="116aa48e-717b-4d49-897d-5538398f4a2a"
Jul 02 12:38:54.857681 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:54.857588    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tczr2" podUID="ebc69c1d-a909-44d0-bcb1-691e39c5e754"
Jul 02 12:38:55.075345 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:55.075261    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hh6v9" event={"ID":"64db6250-f644-4f6d-856d-f3186f40db09","Type":"ContainerStarted","Data":"eb231c65d0329327669edd6a29baff4390df323bfa85e8f1b14b1c684627fb69"}
Jul 02 12:38:55.076901 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:55.076851    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4bj55" event={"ID":"bb22eeab-d1d6-47f7-be7f-0b16dc5de6ef","Type":"ContainerStarted","Data":"b21b64cd73f1ff13c903bcab4a00988136c7c7e84d1ec92065ad55e7bdea9358"}
Jul 02 12:38:55.522173 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:55.522152    2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 02 12:38:55.860116 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:55.860032    2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-02T12:38:55.522167925Z","UUID":"50c87aac-0e2c-4038-aa37-602cb95fa381","Handler":null,"Name":"","Endpoint":""}
Jul 02 12:38:55.861422 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:55.861403    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 02 12:38:55.861530 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:55.861435    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 02 12:38:56.080954 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:56.080865    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cltnv" event={"ID":"3d27224c-42a6-4708-a061-1705cbdba994","Type":"ContainerStarted","Data":"b9e736123a6bcdf5444ef32e75ac91f3beabaa14c40afd414a911619763d30e2"}
Jul 02 12:38:56.083237 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:56.083204    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-88.ec2.internal" event={"ID":"f7c8733428fcf3eb08cd473039be930f","Type":"ContainerStarted","Data":"2c28ff221be4c3b35e95f96832515b0a6f9df2e940f84fb5e96f6a74eb34f99f"}
Jul 02 12:38:56.086855 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:56.086836    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2c7ks_116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a/ovn-acl-logging/0.log"
Jul 02 12:38:56.087347 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:56.087310    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks" event={"ID":"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a","Type":"ContainerStarted","Data":"7523c913d0baa22115fc5bb518214f1838d60fb3cb38502d336343ab7535609d"}
Jul 02 12:38:56.089054 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:56.089033    2576 generic.go:356] "Generic (PLEG): container finished" podID="64db6250-f644-4f6d-856d-f3186f40db09" containerID="eb231c65d0329327669edd6a29baff4390df323bfa85e8f1b14b1c684627fb69" exitCode=0
Jul 02 12:38:56.089159 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:56.089108    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hh6v9" event={"ID":"64db6250-f644-4f6d-856d-f3186f40db09","Type":"ContainerDied","Data":"eb231c65d0329327669edd6a29baff4390df323bfa85e8f1b14b1c684627fb69"}
Jul 02 12:38:56.089159 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:56.089121    2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 02 12:38:56.100121 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:56.100074    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4bj55" podStartSLOduration=4.22931995 podStartE2EDuration="23.100056357s" totalImagesPullingTime="18.870736407s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 12:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:38:55.128251087 +0000 UTC m=+23.735714112" watchObservedRunningTime="2026-07-02 12:38:56.100056357 +0000 UTC m=+24.707519383"
Jul 02 12:38:56.100616 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:56.100586    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-88.ec2.internal" podStartSLOduration=22.100574191 podStartE2EDuration="22.100574191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 12:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:38:56.099959063 +0000 UTC m=+24.707422087" watchObservedRunningTime="2026-07-02 12:38:56.100574191 +0000 UTC m=+24.708037215"
Jul 02 12:38:56.857748 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:56.857660    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tczr2"
Jul 02 12:38:56.857945 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:56.857769    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tczr2" podUID="ebc69c1d-a909-44d0-bcb1-691e39c5e754"
Jul 02 12:38:56.857945 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:56.857660    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sbnql"
Jul 02 12:38:56.857945 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:56.857795    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-72n2l"
Jul 02 12:38:56.858113 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:56.857905    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-72n2l" podUID="116aa48e-717b-4d49-897d-5538398f4a2a"
Jul 02 12:38:56.858113 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:56.858059    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sbnql" podUID="86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7"
Jul 02 12:38:57.093739 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:57.093699    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cltnv" event={"ID":"3d27224c-42a6-4708-a061-1705cbdba994","Type":"ContainerStarted","Data":"1f379fe94a63f0a6633bab09ee9ddb4d638fe8a0b7c8b53ae79a990110332d83"}
Jul 02 12:38:57.111832 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:57.111736    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cltnv" podStartSLOduration=2.453353698 podStartE2EDuration="24.111718191s" totalImagesPullingTime="21.658364493s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 12:38:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:38:57.111148424 +0000 UTC m=+25.718611447" watchObservedRunningTime="2026-07-02 12:38:57.111718191 +0000 UTC m=+25.719181215"
Jul 02 12:38:58.101020 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:58.100836    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2c7ks_116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a/ovn-acl-logging/0.log"
Jul 02 12:38:58.101638 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:58.101589    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks" event={"ID":"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a","Type":"ContainerStarted","Data":"f16deca91625a4777e6048ec549e87b24a524bf91e265487964e29dc92b0611e"}
Jul 02 12:38:58.101954 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:58.101906    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:58.102068 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:58.102032    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:58.102111 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:58.102081    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:58.102158 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:58.102131    2576 scope.go:117] "RemoveContainer" containerID="6b0fb1a929a00f1611b127d4184ed30790de48957d6a700680aa4a5aae990616"
Jul 02 12:38:58.122122 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:58.122062    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:58.122450 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:58.122430    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:38:58.690322 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:58.690292    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/116aa48e-717b-4d49-897d-5538398f4a2a-original-pull-secret\") pod \"global-pull-secret-syncer-72n2l\" (UID: \"116aa48e-717b-4d49-897d-5538398f4a2a\") " pod="kube-system/global-pull-secret-syncer-72n2l"
Jul 02 12:38:58.690505 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:58.690426    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 12:38:58.690505 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:58.690492    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/116aa48e-717b-4d49-897d-5538398f4a2a-original-pull-secret podName:116aa48e-717b-4d49-897d-5538398f4a2a nodeName:}" failed. No retries permitted until 2026-07-02 12:39:14.690474031 +0000 UTC m=+43.297937036 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/116aa48e-717b-4d49-897d-5538398f4a2a-original-pull-secret") pod "global-pull-secret-syncer-72n2l" (UID: "116aa48e-717b-4d49-897d-5538398f4a2a") : object "kube-system"/"original-pull-secret" not registered
Jul 02 12:38:58.856858 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:58.856826    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sbnql"
Jul 02 12:38:58.856858 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:58.856858    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-72n2l"
Jul 02 12:38:58.857045 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:58.856826    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tczr2"
Jul 02 12:38:58.857045 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:58.857004    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sbnql" podUID="86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7"
Jul 02 12:38:58.857140 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:58.857122    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-72n2l" podUID="116aa48e-717b-4d49-897d-5538398f4a2a"
Jul 02 12:38:58.857216 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:58.857200    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tczr2" podUID="ebc69c1d-a909-44d0-bcb1-691e39c5e754"
Jul 02 12:38:59.109798 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:59.109717    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2c7ks_116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a/ovn-acl-logging/0.log"
Jul 02 12:38:59.110462 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:59.110172    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks" event={"ID":"116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a","Type":"ContainerStarted","Data":"0184fc7181d01dccad218707e19953d965cdd6a483f417376fee6747ec1cc102"}
Jul 02 12:38:59.160969 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:59.160894    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks" podStartSLOduration=7.319103975 podStartE2EDuration="26.160874905s" totalImagesPullingTime="18.84177093s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 12:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:38:59.158296306 +0000 UTC m=+27.765759330" watchObservedRunningTime="2026-07-02 12:38:59.160874905 +0000 UTC m=+27.768337929"
Jul 02 12:38:59.898746 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:59.898713    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-72n2l"]
Jul 02 12:38:59.898928 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:59.898891    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-72n2l"
Jul 02 12:38:59.899049 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:59.899022    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-72n2l" podUID="116aa48e-717b-4d49-897d-5538398f4a2a"
Jul 02 12:38:59.904639 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:59.904611    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tczr2"]
Jul 02 12:38:59.904764 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:59.904722    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tczr2"
Jul 02 12:38:59.904851 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:59.904816    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tczr2" podUID="ebc69c1d-a909-44d0-bcb1-691e39c5e754"
Jul 02 12:38:59.905371 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:59.905348    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sbnql"]
Jul 02 12:38:59.905470 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:38:59.905451    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sbnql"
Jul 02 12:38:59.905568 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:38:59.905544    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sbnql" podUID="86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7"
Jul 02 12:39:01.116244 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:01.116079    2576 generic.go:356] "Generic (PLEG): container finished" podID="64db6250-f644-4f6d-856d-f3186f40db09" containerID="03be4c1129036df70208796479f8d7a9e82445448a018e28e5531b2af9806e09" exitCode=0
Jul 02 12:39:01.116527 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:01.116148    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hh6v9" event={"ID":"64db6250-f644-4f6d-856d-f3186f40db09","Type":"ContainerDied","Data":"03be4c1129036df70208796479f8d7a9e82445448a018e28e5531b2af9806e09"}
Jul 02 12:39:01.373201 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:01.373173    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-t6qwm"
Jul 02 12:39:01.373362 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:01.373289    2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 02 12:39:01.373776 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:01.373762    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-t6qwm"
Jul 02 12:39:01.857706 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:01.857683    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tczr2"
Jul 02 12:39:01.857864 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:01.857717    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-72n2l"
Jul 02 12:39:01.857864 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:01.857684    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sbnql"
Jul 02 12:39:01.857864 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:01.857795    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tczr2" podUID="ebc69c1d-a909-44d0-bcb1-691e39c5e754"
Jul 02 12:39:01.858041 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:01.857873    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sbnql" podUID="86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7"
Jul 02 12:39:01.858041 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:01.858002    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-72n2l" podUID="116aa48e-717b-4d49-897d-5538398f4a2a"
Jul 02 12:39:02.120068 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:02.120037    2576 generic.go:356] "Generic (PLEG): container finished" podID="64db6250-f644-4f6d-856d-f3186f40db09" containerID="0da231b441956df056884aa4b16126a989af1b59759f6e905d781d6ba2bed290" exitCode=0
Jul 02 12:39:02.120428 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:02.120115    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hh6v9" event={"ID":"64db6250-f644-4f6d-856d-f3186f40db09","Type":"ContainerDied","Data":"0da231b441956df056884aa4b16126a989af1b59759f6e905d781d6ba2bed290"}
Jul 02 12:39:03.857105 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:03.857075    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sbnql"
Jul 02 12:39:03.857509 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:03.857170    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sbnql" podUID="86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7"
Jul 02 12:39:03.857509 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:03.857083    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-72n2l"
Jul 02 12:39:03.857509 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:03.857202    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tczr2"
Jul 02 12:39:03.857509 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:03.857273    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-72n2l" podUID="116aa48e-717b-4d49-897d-5538398f4a2a"
Jul 02 12:39:03.857509 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:03.857332    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tczr2" podUID="ebc69c1d-a909-44d0-bcb1-691e39c5e754"
Jul 02 12:39:04.127151 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:04.127057    2576 generic.go:356] "Generic (PLEG): container finished" podID="64db6250-f644-4f6d-856d-f3186f40db09" containerID="e635dff4bc44ab6b57a527e97388ae7274da428decc694f63b260680183fb6ef" exitCode=0
Jul 02 12:39:04.127151 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:04.127119    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hh6v9" event={"ID":"64db6250-f644-4f6d-856d-f3186f40db09","Type":"ContainerDied","Data":"e635dff4bc44ab6b57a527e97388ae7274da428decc694f63b260680183fb6ef"}
Jul 02 12:39:04.832290 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:04.832257    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7-metrics-certs\") pod \"network-metrics-daemon-sbnql\" (UID: \"86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7\") " pod="openshift-multus/network-metrics-daemon-sbnql"
Jul 02 12:39:04.832465 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:04.832445    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 12:39:04.832535 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:04.832524    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7-metrics-certs podName:86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:36.832504233 +0000 UTC m=+65.439967243 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7-metrics-certs") pod "network-metrics-daemon-sbnql" (UID: "86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 12:39:05.033852 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:05.033814    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zvwz\" (UniqueName: \"kubernetes.io/projected/ebc69c1d-a909-44d0-bcb1-691e39c5e754-kube-api-access-2zvwz\") pod \"network-check-target-tczr2\" (UID: \"ebc69c1d-a909-44d0-bcb1-691e39c5e754\") " pod="openshift-network-diagnostics/network-check-target-tczr2"
Jul 02 12:39:05.034319 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:05.034018    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 12:39:05.034319 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:05.034037    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 12:39:05.034319 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:05.034047    2576 projected.go:196] Error preparing data for projected volume kube-api-access-2zvwz for pod openshift-network-diagnostics/network-check-target-tczr2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 12:39:05.034319 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:05.034110    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ebc69c1d-a909-44d0-bcb1-691e39c5e754-kube-api-access-2zvwz podName:ebc69c1d-a909-44d0-bcb1-691e39c5e754 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:37.034094984 +0000 UTC m=+65.641557999 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2zvwz" (UniqueName: "kubernetes.io/projected/ebc69c1d-a909-44d0-bcb1-691e39c5e754-kube-api-access-2zvwz") pod "network-check-target-tczr2" (UID: "ebc69c1d-a909-44d0-bcb1-691e39c5e754") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 12:39:05.857005 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:05.856974    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-72n2l"
Jul 02 12:39:05.857183 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:05.856974    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sbnql"
Jul 02 12:39:05.857183 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:05.857071    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-72n2l" podUID="116aa48e-717b-4d49-897d-5538398f4a2a"
Jul 02 12:39:05.857183 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:05.856974    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tczr2"
Jul 02 12:39:05.857342 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:05.857158    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sbnql" podUID="86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7"
Jul 02 12:39:05.857342 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:05.857272    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tczr2" podUID="ebc69c1d-a909-44d0-bcb1-691e39c5e754"
Jul 02 12:39:07.186619 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.186384    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeReady"
Jul 02 12:39:07.187075 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.186754    2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 02 12:39:07.233866 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.233834    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-847db68bb4-fgghh"]
Jul 02 12:39:07.261716 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.261682    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-s82bz"]
Jul 02 12:39:07.261873 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.261814    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-847db68bb4-fgghh"
Jul 02 12:39:07.268142 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.268118    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 02 12:39:07.268292 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.268198    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 02 12:39:07.268292 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.268253    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-58r25\""
Jul 02 12:39:07.268550 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.268528    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 02 12:39:07.272858 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.272819    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 02 12:39:07.281717 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.281698    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-847db68bb4-fgghh"]
Jul 02 12:39:07.281805 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.281733    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-shm9z"]
Jul 02 12:39:07.281847 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.281821    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-s82bz"
Jul 02 12:39:07.285505 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.285041    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 02 12:39:07.285505 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.285326    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-l95ls\""
Jul 02 12:39:07.285697 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.285678    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 02 12:39:07.307896 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.307863    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-s82bz"]
Jul 02 12:39:07.308017 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.307907    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-shm9z"]
Jul 02 12:39:07.308017 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.307962    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-shm9z"
Jul 02 12:39:07.311746 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.311723    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 02 12:39:07.312001 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.311986    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8zllz\""
Jul 02 12:39:07.312121 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.312103    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 02 12:39:07.312258 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.312242    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 02 12:39:07.350276 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.350244    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0c646b36-0375-43c3-a0eb-8e08a2821414-config-volume\") pod \"dns-default-s82bz\" (UID: \"0c646b36-0375-43c3-a0eb-8e08a2821414\") " pod="openshift-dns/dns-default-s82bz"
Jul 02 12:39:07.350443 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.350290    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/69ae490b-bd41-4ab2-b05c-3393fea68351-registry-certificates\") pod \"image-registry-847db68bb4-fgghh\" (UID: \"69ae490b-bd41-4ab2-b05c-3393fea68351\") " pod="openshift-image-registry/image-registry-847db68bb4-fgghh"
Jul 02 12:39:07.350443 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.350311    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/69ae490b-bd41-4ab2-b05c-3393fea68351-installation-pull-secrets\") pod \"image-registry-847db68bb4-fgghh\" (UID: \"69ae490b-bd41-4ab2-b05c-3393fea68351\") " pod="openshift-image-registry/image-registry-847db68bb4-fgghh"
Jul 02 12:39:07.350443 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.350346    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/69ae490b-bd41-4ab2-b05c-3393fea68351-image-registry-private-configuration\") pod \"image-registry-847db68bb4-fgghh\" (UID: \"69ae490b-bd41-4ab2-b05c-3393fea68351\") " pod="openshift-image-registry/image-registry-847db68bb4-fgghh"
Jul 02 12:39:07.350443 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.350373    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/69ae490b-bd41-4ab2-b05c-3393fea68351-ca-trust-extracted\") pod \"image-registry-847db68bb4-fgghh\" (UID: \"69ae490b-bd41-4ab2-b05c-3393fea68351\") " pod="openshift-image-registry/image-registry-847db68bb4-fgghh"
Jul 02 12:39:07.350443 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.350402    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c646b36-0375-43c3-a0eb-8e08a2821414-tmp-dir\") pod \"dns-default-s82bz\" (UID: \"0c646b36-0375-43c3-a0eb-8e08a2821414\") " pod="openshift-dns/dns-default-s82bz"
Jul 02 12:39:07.350443 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.350422    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hh97\" (UniqueName: \"kubernetes.io/projected/0c646b36-0375-43c3-a0eb-8e08a2821414-kube-api-access-2hh97\") pod \"dns-default-s82bz\" (UID: \"0c646b36-0375-43c3-a0eb-8e08a2821414\") " pod="openshift-dns/dns-default-s82bz"
Jul 02 12:39:07.350706 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.350471    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzpmp\" (UniqueName: \"kubernetes.io/projected/69ae490b-bd41-4ab2-b05c-3393fea68351-kube-api-access-wzpmp\") pod \"image-registry-847db68bb4-fgghh\" (UID: \"69ae490b-bd41-4ab2-b05c-3393fea68351\") " pod="openshift-image-registry/image-registry-847db68bb4-fgghh"
Jul 02 12:39:07.350706 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.350508    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69ae490b-bd41-4ab2-b05c-3393fea68351-bound-sa-token\") pod \"image-registry-847db68bb4-fgghh\" (UID: \"69ae490b-bd41-4ab2-b05c-3393fea68351\") " pod="openshift-image-registry/image-registry-847db68bb4-fgghh"
Jul 02 12:39:07.350706 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.350534    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c646b36-0375-43c3-a0eb-8e08a2821414-metrics-tls\") pod \"dns-default-s82bz\" (UID: \"0c646b36-0375-43c3-a0eb-8e08a2821414\") " pod="openshift-dns/dns-default-s82bz"
Jul 02 12:39:07.350706 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.350555    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69ae490b-bd41-4ab2-b05c-3393fea68351-registry-tls\") pod \"image-registry-847db68bb4-fgghh\" (UID: \"69ae490b-bd41-4ab2-b05c-3393fea68351\") " pod="openshift-image-registry/image-registry-847db68bb4-fgghh"
Jul 02 12:39:07.350706 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.350604    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/69ae490b-bd41-4ab2-b05c-3393fea68351-trusted-ca\") pod \"image-registry-847db68bb4-fgghh\" (UID: \"69ae490b-bd41-4ab2-b05c-3393fea68351\") " pod="openshift-image-registry/image-registry-847db68bb4-fgghh"
Jul 02 12:39:07.451790 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.451710    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/69ae490b-bd41-4ab2-b05c-3393fea68351-trusted-ca\") pod \"image-registry-847db68bb4-fgghh\" (UID: \"69ae490b-bd41-4ab2-b05c-3393fea68351\") " pod="openshift-image-registry/image-registry-847db68bb4-fgghh"
Jul 02 12:39:07.451790 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.451773    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0c646b36-0375-43c3-a0eb-8e08a2821414-config-volume\") pod \"dns-default-s82bz\" (UID: \"0c646b36-0375-43c3-a0eb-8e08a2821414\") " pod="openshift-dns/dns-default-s82bz"
Jul 02 12:39:07.451790 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.451793    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/69ae490b-bd41-4ab2-b05c-3393fea68351-registry-certificates\") pod \"image-registry-847db68bb4-fgghh\" (UID: \"69ae490b-bd41-4ab2-b05c-3393fea68351\") " pod="openshift-image-registry/image-registry-847db68bb4-fgghh"
Jul 02 12:39:07.452091 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.451810    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/69ae490b-bd41-4ab2-b05c-3393fea68351-installation-pull-secrets\") pod \"image-registry-847db68bb4-fgghh\" (UID: \"69ae490b-bd41-4ab2-b05c-3393fea68351\") " pod="openshift-image-registry/image-registry-847db68bb4-fgghh"
Jul 02 12:39:07.452091 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.451850    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/69ae490b-bd41-4ab2-b05c-3393fea68351-image-registry-private-configuration\") pod \"image-registry-847db68bb4-fgghh\" (UID: \"69ae490b-bd41-4ab2-b05c-3393fea68351\") " pod="openshift-image-registry/image-registry-847db68bb4-fgghh"
Jul 02 12:39:07.452091 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.451877    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/69ae490b-bd41-4ab2-b05c-3393fea68351-ca-trust-extracted\") pod \"image-registry-847db68bb4-fgghh\" (UID: \"69ae490b-bd41-4ab2-b05c-3393fea68351\") " pod="openshift-image-registry/image-registry-847db68bb4-fgghh"
Jul 02 12:39:07.452091 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.451928    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c646b36-0375-43c3-a0eb-8e08a2821414-tmp-dir\") pod \"dns-default-s82bz\" (UID: \"0c646b36-0375-43c3-a0eb-8e08a2821414\") " pod="openshift-dns/dns-default-s82bz"
Jul 02 12:39:07.452299 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.452196    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hh97\" (UniqueName: \"kubernetes.io/projected/0c646b36-0375-43c3-a0eb-8e08a2821414-kube-api-access-2hh97\") pod \"dns-default-s82bz\" (UID: \"0c646b36-0375-43c3-a0eb-8e08a2821414\") " pod="openshift-dns/dns-default-s82bz"
Jul 02 12:39:07.452299 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.452262    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzpmp\" (UniqueName: \"kubernetes.io/projected/69ae490b-bd41-4ab2-b05c-3393fea68351-kube-api-access-wzpmp\") pod \"image-registry-847db68bb4-fgghh\" (UID: \"69ae490b-bd41-4ab2-b05c-3393fea68351\") " pod="openshift-image-registry/image-registry-847db68bb4-fgghh"
Jul 02 12:39:07.452404 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.452314    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bflzp\" (UniqueName: \"kubernetes.io/projected/8d276bfd-8b47-48ae-a99c-6e4f069e0fd5-kube-api-access-bflzp\") pod \"ingress-canary-shm9z\" (UID: \"8d276bfd-8b47-48ae-a99c-6e4f069e0fd5\") " pod="openshift-ingress-canary/ingress-canary-shm9z"
Jul 02 12:39:07.452404 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.452347    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69ae490b-bd41-4ab2-b05c-3393fea68351-bound-sa-token\") pod \"image-registry-847db68bb4-fgghh\" (UID: \"69ae490b-bd41-4ab2-b05c-3393fea68351\") " pod="openshift-image-registry/image-registry-847db68bb4-fgghh"
Jul 02 12:39:07.452404 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.452382    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c646b36-0375-43c3-a0eb-8e08a2821414-metrics-tls\") pod \"dns-default-s82bz\" (UID: \"0c646b36-0375-43c3-a0eb-8e08a2821414\") " pod="openshift-dns/dns-default-s82bz"
Jul 02 12:39:07.452540 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.452409    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69ae490b-bd41-4ab2-b05c-3393fea68351-registry-tls\") pod \"image-registry-847db68bb4-fgghh\" (UID: \"69ae490b-bd41-4ab2-b05c-3393fea68351\") " pod="openshift-image-registry/image-registry-847db68bb4-fgghh"
Jul 02 12:39:07.452540 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.452432    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d276bfd-8b47-48ae-a99c-6e4f069e0fd5-cert\") pod \"ingress-canary-shm9z\" (UID: \"8d276bfd-8b47-48ae-a99c-6e4f069e0fd5\") " pod="openshift-ingress-canary/ingress-canary-shm9z"
Jul 02 12:39:07.452540 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.452459    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0c646b36-0375-43c3-a0eb-8e08a2821414-config-volume\") pod \"dns-default-s82bz\" (UID: \"0c646b36-0375-43c3-a0eb-8e08a2821414\") " pod="openshift-dns/dns-default-s82bz"
Jul 02 12:39:07.453019 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:07.452727    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 12:39:07.453019 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:07.452749    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-847db68bb4-fgghh: secret "image-registry-tls" not found
Jul 02 12:39:07.453019 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:07.452823    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69ae490b-bd41-4ab2-b05c-3393fea68351-registry-tls podName:69ae490b-bd41-4ab2-b05c-3393fea68351 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:07.952803961 +0000 UTC m=+36.560266973 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/69ae490b-bd41-4ab2-b05c-3393fea68351-registry-tls") pod "image-registry-847db68bb4-fgghh" (UID: "69ae490b-bd41-4ab2-b05c-3393fea68351") : secret "image-registry-tls" not found
Jul 02 12:39:07.453019 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.452843    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/69ae490b-bd41-4ab2-b05c-3393fea68351-trusted-ca\") pod \"image-registry-847db68bb4-fgghh\" (UID: \"69ae490b-bd41-4ab2-b05c-3393fea68351\") " pod="openshift-image-registry/image-registry-847db68bb4-fgghh"
Jul 02 12:39:07.453019 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:07.452906    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 12:39:07.453019 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.452945    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c646b36-0375-43c3-a0eb-8e08a2821414-tmp-dir\") pod \"dns-default-s82bz\" (UID: \"0c646b36-0375-43c3-a0eb-8e08a2821414\") " pod="openshift-dns/dns-default-s82bz"
Jul 02 12:39:07.453019 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:07.452985    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c646b36-0375-43c3-a0eb-8e08a2821414-metrics-tls podName:0c646b36-0375-43c3-a0eb-8e08a2821414 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:07.952970164 +0000 UTC m=+36.560433177 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c646b36-0375-43c3-a0eb-8e08a2821414-metrics-tls") pod "dns-default-s82bz" (UID: "0c646b36-0375-43c3-a0eb-8e08a2821414") : secret "dns-default-metrics-tls" not found
Jul 02 12:39:07.453356 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.453056    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/69ae490b-bd41-4ab2-b05c-3393fea68351-ca-trust-extracted\") pod \"image-registry-847db68bb4-fgghh\" (UID: \"69ae490b-bd41-4ab2-b05c-3393fea68351\") " pod="openshift-image-registry/image-registry-847db68bb4-fgghh"
Jul 02 12:39:07.453356 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.453065    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/69ae490b-bd41-4ab2-b05c-3393fea68351-registry-certificates\") pod \"image-registry-847db68bb4-fgghh\" (UID: \"69ae490b-bd41-4ab2-b05c-3393fea68351\") " pod="openshift-image-registry/image-registry-847db68bb4-fgghh"
Jul 02 12:39:07.456575 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.456547    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/69ae490b-bd41-4ab2-b05c-3393fea68351-installation-pull-secrets\") pod \"image-registry-847db68bb4-fgghh\" (UID: \"69ae490b-bd41-4ab2-b05c-3393fea68351\") " pod="openshift-image-registry/image-registry-847db68bb4-fgghh"
Jul 02 12:39:07.456575 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.456553    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/69ae490b-bd41-4ab2-b05c-3393fea68351-image-registry-private-configuration\") pod \"image-registry-847db68bb4-fgghh\" (UID: \"69ae490b-bd41-4ab2-b05c-3393fea68351\") " pod="openshift-image-registry/image-registry-847db68bb4-fgghh"
Jul 02 12:39:07.464707 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.464679    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzpmp\" (UniqueName: \"kubernetes.io/projected/69ae490b-bd41-4ab2-b05c-3393fea68351-kube-api-access-wzpmp\") pod \"image-registry-847db68bb4-fgghh\" (UID: \"69ae490b-bd41-4ab2-b05c-3393fea68351\") " pod="openshift-image-registry/image-registry-847db68bb4-fgghh"
Jul 02 12:39:07.468382 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.468359    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hh97\" (UniqueName: \"kubernetes.io/projected/0c646b36-0375-43c3-a0eb-8e08a2821414-kube-api-access-2hh97\") pod \"dns-default-s82bz\" (UID: \"0c646b36-0375-43c3-a0eb-8e08a2821414\") " pod="openshift-dns/dns-default-s82bz"
Jul 02 12:39:07.472616 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.472591    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69ae490b-bd41-4ab2-b05c-3393fea68351-bound-sa-token\") pod \"image-registry-847db68bb4-fgghh\" (UID: \"69ae490b-bd41-4ab2-b05c-3393fea68351\") " pod="openshift-image-registry/image-registry-847db68bb4-fgghh"
Jul 02 12:39:07.553820 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.553782    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bflzp\" (UniqueName: \"kubernetes.io/projected/8d276bfd-8b47-48ae-a99c-6e4f069e0fd5-kube-api-access-bflzp\") pod \"ingress-canary-shm9z\" (UID: \"8d276bfd-8b47-48ae-a99c-6e4f069e0fd5\") " pod="openshift-ingress-canary/ingress-canary-shm9z"
Jul 02 12:39:07.554049 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.553866    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d276bfd-8b47-48ae-a99c-6e4f069e0fd5-cert\") pod \"ingress-canary-shm9z\" (UID: \"8d276bfd-8b47-48ae-a99c-6e4f069e0fd5\") " pod="openshift-ingress-canary/ingress-canary-shm9z"
Jul 02 12:39:07.556559 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.556535    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d276bfd-8b47-48ae-a99c-6e4f069e0fd5-cert\") pod \"ingress-canary-shm9z\" (UID: \"8d276bfd-8b47-48ae-a99c-6e4f069e0fd5\") " pod="openshift-ingress-canary/ingress-canary-shm9z"
Jul 02 12:39:07.563382 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.563362    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bflzp\" (UniqueName: \"kubernetes.io/projected/8d276bfd-8b47-48ae-a99c-6e4f069e0fd5-kube-api-access-bflzp\") pod \"ingress-canary-shm9z\" (UID: \"8d276bfd-8b47-48ae-a99c-6e4f069e0fd5\") " pod="openshift-ingress-canary/ingress-canary-shm9z"
Jul 02 12:39:07.617109 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.617083    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-shm9z"
Jul 02 12:39:07.857421 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.857335    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tczr2"
Jul 02 12:39:07.857421 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.857398    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sbnql"
Jul 02 12:39:07.857614 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.857429    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-72n2l"
Jul 02 12:39:07.860804 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.860663    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 02 12:39:07.860804 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.860677    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 02 12:39:07.860804 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.860670    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 02 12:39:07.860804 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.860776    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bnkmz\""
Jul 02 12:39:07.861470 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.861449    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hkdpl\""
Jul 02 12:39:07.861574 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.861478    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 02 12:39:07.956942 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.956903    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c646b36-0375-43c3-a0eb-8e08a2821414-metrics-tls\") pod \"dns-default-s82bz\" (UID: \"0c646b36-0375-43c3-a0eb-8e08a2821414\") " pod="openshift-dns/dns-default-s82bz"
Jul 02 12:39:07.957090 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:07.956948    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69ae490b-bd41-4ab2-b05c-3393fea68351-registry-tls\") pod \"image-registry-847db68bb4-fgghh\" (UID: \"69ae490b-bd41-4ab2-b05c-3393fea68351\") " pod="openshift-image-registry/image-registry-847db68bb4-fgghh"
Jul 02 12:39:07.957090 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:07.957053    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 12:39:07.957181 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:07.957099    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 12:39:07.957181 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:07.957114    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-847db68bb4-fgghh: secret "image-registry-tls" not found
Jul 02 12:39:07.957181 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:07.957131    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c646b36-0375-43c3-a0eb-8e08a2821414-metrics-tls podName:0c646b36-0375-43c3-a0eb-8e08a2821414 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:08.957114008 +0000 UTC m=+37.564577008 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c646b36-0375-43c3-a0eb-8e08a2821414-metrics-tls") pod "dns-default-s82bz" (UID: "0c646b36-0375-43c3-a0eb-8e08a2821414") : secret "dns-default-metrics-tls" not found
Jul 02 12:39:07.957181 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:07.957161    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69ae490b-bd41-4ab2-b05c-3393fea68351-registry-tls podName:69ae490b-bd41-4ab2-b05c-3393fea68351 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:08.957144789 +0000 UTC m=+37.564607794 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/69ae490b-bd41-4ab2-b05c-3393fea68351-registry-tls") pod "image-registry-847db68bb4-fgghh" (UID: "69ae490b-bd41-4ab2-b05c-3393fea68351") : secret "image-registry-tls" not found
Jul 02 12:39:08.433823 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:08.433792    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-shm9z"]
Jul 02 12:39:08.501302 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:39:08.501275    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d276bfd_8b47_48ae_a99c_6e4f069e0fd5.slice/crio-82475d00135effe096f1e2d9996882d2c8957e1a8e52d9547736ccda20fc21c8 WatchSource:0}: Error finding container 82475d00135effe096f1e2d9996882d2c8957e1a8e52d9547736ccda20fc21c8: Status 404 returned error can't find the container with id 82475d00135effe096f1e2d9996882d2c8957e1a8e52d9547736ccda20fc21c8
Jul 02 12:39:08.965273 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:08.965096    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c646b36-0375-43c3-a0eb-8e08a2821414-metrics-tls\") pod \"dns-default-s82bz\" (UID: \"0c646b36-0375-43c3-a0eb-8e08a2821414\") " pod="openshift-dns/dns-default-s82bz"
Jul 02 12:39:08.965273 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:08.965278    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69ae490b-bd41-4ab2-b05c-3393fea68351-registry-tls\") pod \"image-registry-847db68bb4-fgghh\" (UID: \"69ae490b-bd41-4ab2-b05c-3393fea68351\") " pod="openshift-image-registry/image-registry-847db68bb4-fgghh"
Jul 02 12:39:08.965508 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:08.965243    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 12:39:08.965508 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:08.965368    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 12:39:08.965508 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:08.965379    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-847db68bb4-fgghh: secret "image-registry-tls" not found
Jul 02 12:39:08.965508 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:08.965409    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c646b36-0375-43c3-a0eb-8e08a2821414-metrics-tls podName:0c646b36-0375-43c3-a0eb-8e08a2821414 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:10.965387621 +0000 UTC m=+39.572850624 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c646b36-0375-43c3-a0eb-8e08a2821414-metrics-tls") pod "dns-default-s82bz" (UID: "0c646b36-0375-43c3-a0eb-8e08a2821414") : secret "dns-default-metrics-tls" not found
Jul 02 12:39:08.965508 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:08.965427    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69ae490b-bd41-4ab2-b05c-3393fea68351-registry-tls podName:69ae490b-bd41-4ab2-b05c-3393fea68351 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:10.9654188 +0000 UTC m=+39.572881802 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/69ae490b-bd41-4ab2-b05c-3393fea68351-registry-tls") pod "image-registry-847db68bb4-fgghh" (UID: "69ae490b-bd41-4ab2-b05c-3393fea68351") : secret "image-registry-tls" not found
Jul 02 12:39:09.140653 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:09.140614    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-shm9z" event={"ID":"8d276bfd-8b47-48ae-a99c-6e4f069e0fd5","Type":"ContainerStarted","Data":"82475d00135effe096f1e2d9996882d2c8957e1a8e52d9547736ccda20fc21c8"}
Jul 02 12:39:09.143865 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:09.143808    2576 generic.go:356] "Generic (PLEG): container finished" podID="64db6250-f644-4f6d-856d-f3186f40db09" containerID="ba6072bb16dba755067a32dfa63b919829876b095e4914f71d82864138160676" exitCode=0
Jul 02 12:39:09.144005 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:09.143877    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hh6v9" event={"ID":"64db6250-f644-4f6d-856d-f3186f40db09","Type":"ContainerDied","Data":"ba6072bb16dba755067a32dfa63b919829876b095e4914f71d82864138160676"}
Jul 02 12:39:10.149031 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:10.149000    2576 generic.go:356] "Generic (PLEG): container finished" podID="64db6250-f644-4f6d-856d-f3186f40db09" containerID="5dced82017071e9d0f3e635af3e1998ae5f9e90b91f17423059f2d1e21845e9e" exitCode=0
Jul 02 12:39:10.149396 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:10.149051    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hh6v9" event={"ID":"64db6250-f644-4f6d-856d-f3186f40db09","Type":"ContainerDied","Data":"5dced82017071e9d0f3e635af3e1998ae5f9e90b91f17423059f2d1e21845e9e"}
Jul 02 12:39:10.983255 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:10.983212    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c646b36-0375-43c3-a0eb-8e08a2821414-metrics-tls\") pod \"dns-default-s82bz\" (UID: \"0c646b36-0375-43c3-a0eb-8e08a2821414\") " pod="openshift-dns/dns-default-s82bz"
Jul 02 12:39:10.983255 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:10.983259    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69ae490b-bd41-4ab2-b05c-3393fea68351-registry-tls\") pod \"image-registry-847db68bb4-fgghh\" (UID: \"69ae490b-bd41-4ab2-b05c-3393fea68351\") " pod="openshift-image-registry/image-registry-847db68bb4-fgghh"
Jul 02 12:39:10.983517 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:10.983381    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 12:39:10.983517 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:10.983444    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c646b36-0375-43c3-a0eb-8e08a2821414-metrics-tls podName:0c646b36-0375-43c3-a0eb-8e08a2821414 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:14.983428548 +0000 UTC m=+43.590891549 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c646b36-0375-43c3-a0eb-8e08a2821414-metrics-tls") pod "dns-default-s82bz" (UID: "0c646b36-0375-43c3-a0eb-8e08a2821414") : secret "dns-default-metrics-tls" not found
Jul 02 12:39:10.983517 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:10.983383    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 12:39:10.983517 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:10.983465    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-847db68bb4-fgghh: secret "image-registry-tls" not found
Jul 02 12:39:10.983517 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:10.983496    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69ae490b-bd41-4ab2-b05c-3393fea68351-registry-tls podName:69ae490b-bd41-4ab2-b05c-3393fea68351 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:14.983487883 +0000 UTC m=+43.590950884 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/69ae490b-bd41-4ab2-b05c-3393fea68351-registry-tls") pod "image-registry-847db68bb4-fgghh" (UID: "69ae490b-bd41-4ab2-b05c-3393fea68351") : secret "image-registry-tls" not found
Jul 02 12:39:11.155392 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:11.155352    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hh6v9" event={"ID":"64db6250-f644-4f6d-856d-f3186f40db09","Type":"ContainerStarted","Data":"12cbb8cb532f30889c5e9ac6cf88b17ca7951d69d3bc5515f94471d2f5b51cd7"}
Jul 02 12:39:11.183981 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:11.183929    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hh6v9" podStartSLOduration=3.846964756 podStartE2EDuration="38.183897068s" totalImagesPullingTime="34.336932312s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 12:38:33 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-02 12:39:11.183691834 +0000 UTC m=+39.791154857" watchObservedRunningTime="2026-07-02 12:39:11.183897068 +0000 UTC m=+39.791360092"
Jul 02 12:39:12.158745 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.158703    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-shm9z" event={"ID":"8d276bfd-8b47-48ae-a99c-6e4f069e0fd5","Type":"ContainerStarted","Data":"c03541b03a98b1aba8a9140ca2b3f2ff890f495caabf2fca532114f4a2d7c16f"}
Jul 02 12:39:12.177077 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.177025    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-shm9z" podStartSLOduration=2.345683594 podStartE2EDuration="5.177010357s" totalImagesPullingTime="2.831326763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:39:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:39:12.175955327 +0000 UTC m=+40.783418349" watchObservedRunningTime="2026-07-02 12:39:12.177010357 +0000 UTC m=+40.784473371"
Jul 02 12:39:12.404182 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.404152    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-656bc596cc-m2fg7"]
Jul 02 12:39:12.430521 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.430449    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-656bc596cc-m2fg7"]
Jul 02 12:39:12.430648 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.430542    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-656bc596cc-m2fg7"
Jul 02 12:39:12.432985 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.432963    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 02 12:39:12.433163 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.433148    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 02 12:39:12.433206 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.433168    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 02 12:39:12.433252 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.433207    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 02 12:39:12.433477 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.433456    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 02 12:39:12.433582 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.433486    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 02 12:39:12.433582 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.433540    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-vxtl6\""
Jul 02 12:39:12.508498 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.508471    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-54cb84584-dz8tm"]
Jul 02 12:39:12.525036 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.525014    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5f67b55555-r4pwp"]
Jul 02 12:39:12.525158 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.525144    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-dz8tm"
Jul 02 12:39:12.528993 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.528972    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 02 12:39:12.530233 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.530151    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 02 12:39:12.530233 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.530167    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 02 12:39:12.530233 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.530176    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-fsbd6\""
Jul 02 12:39:12.530233 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.530167    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 02 12:39:12.543452 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.543429    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-54cb84584-dz8tm"]
Jul 02 12:39:12.543552 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.543435    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5f67b55555-r4pwp"
Jul 02 12:39:12.547827 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.547810    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 02 12:39:12.548048 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.547884    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 02 12:39:12.548048 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.547975    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 02 12:39:12.548048 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.547978    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 02 12:39:12.548048 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.547992    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-s4qb2\""
Jul 02 12:39:12.551168 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.551145    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5f67b55555-r4pwp"]
Jul 02 12:39:12.565433 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.565415    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 02 12:39:12.594652 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.594631    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02f1bfff-40d6-4805-8579-9e3f5a390e25-service-ca-bundle\") pod \"router-default-656bc596cc-m2fg7\" (UID: \"02f1bfff-40d6-4805-8579-9e3f5a390e25\") " pod="openshift-ingress/router-default-656bc596cc-m2fg7"
Jul 02 12:39:12.594743 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.594656    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2gnb\" (UniqueName: \"kubernetes.io/projected/02f1bfff-40d6-4805-8579-9e3f5a390e25-kube-api-access-j2gnb\") pod \"router-default-656bc596cc-m2fg7\" (UID: \"02f1bfff-40d6-4805-8579-9e3f5a390e25\") " pod="openshift-ingress/router-default-656bc596cc-m2fg7"
Jul 02 12:39:12.594743 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.594686    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02f1bfff-40d6-4805-8579-9e3f5a390e25-metrics-certs\") pod \"router-default-656bc596cc-m2fg7\" (UID: \"02f1bfff-40d6-4805-8579-9e3f5a390e25\") " pod="openshift-ingress/router-default-656bc596cc-m2fg7"
Jul 02 12:39:12.594809 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.594751    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/02f1bfff-40d6-4805-8579-9e3f5a390e25-default-certificate\") pod \"router-default-656bc596cc-m2fg7\" (UID: \"02f1bfff-40d6-4805-8579-9e3f5a390e25\") " pod="openshift-ingress/router-default-656bc596cc-m2fg7"
Jul 02 12:39:12.594840 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.594813    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/02f1bfff-40d6-4805-8579-9e3f5a390e25-stats-auth\") pod \"router-default-656bc596cc-m2fg7\" (UID: \"02f1bfff-40d6-4805-8579-9e3f5a390e25\") " pod="openshift-ingress/router-default-656bc596cc-m2fg7"
Jul 02 12:39:12.695329 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.695243    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dabc8a5d-7710-42b1-93e2-bf1a0f5c94e5-service-ca-bundle\") pod \"insights-operator-5f67b55555-r4pwp\" (UID: \"dabc8a5d-7710-42b1-93e2-bf1a0f5c94e5\") " pod="openshift-insights/insights-operator-5f67b55555-r4pwp"
Jul 02 12:39:12.695329 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.695283    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dabc8a5d-7710-42b1-93e2-bf1a0f5c94e5-serving-cert\") pod \"insights-operator-5f67b55555-r4pwp\" (UID: \"dabc8a5d-7710-42b1-93e2-bf1a0f5c94e5\") " pod="openshift-insights/insights-operator-5f67b55555-r4pwp"
Jul 02 12:39:12.695329 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.695327    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dabc8a5d-7710-42b1-93e2-bf1a0f5c94e5-trusted-ca-bundle\") pod \"insights-operator-5f67b55555-r4pwp\" (UID: \"dabc8a5d-7710-42b1-93e2-bf1a0f5c94e5\") " pod="openshift-insights/insights-operator-5f67b55555-r4pwp"
Jul 02 12:39:12.695684 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.695362    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6klkc\" (UniqueName: \"kubernetes.io/projected/ac58c65c-c830-4ba0-aef5-c5745ed26d93-kube-api-access-6klkc\") pod \"cluster-monitoring-operator-54cb84584-dz8tm\" (UID: \"ac58c65c-c830-4ba0-aef5-c5745ed26d93\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-dz8tm"
Jul 02 12:39:12.695684 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.695389    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/dabc8a5d-7710-42b1-93e2-bf1a0f5c94e5-snapshots\") pod \"insights-operator-5f67b55555-r4pwp\" (UID: \"dabc8a5d-7710-42b1-93e2-bf1a0f5c94e5\") " pod="openshift-insights/insights-operator-5f67b55555-r4pwp"
Jul 02 12:39:12.695684 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.695427    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02f1bfff-40d6-4805-8579-9e3f5a390e25-metrics-certs\") pod \"router-default-656bc596cc-m2fg7\" (UID: \"02f1bfff-40d6-4805-8579-9e3f5a390e25\") " pod="openshift-ingress/router-default-656bc596cc-m2fg7"
Jul 02 12:39:12.695684 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.695446    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/02f1bfff-40d6-4805-8579-9e3f5a390e25-default-certificate\") pod \"router-default-656bc596cc-m2fg7\" (UID: \"02f1bfff-40d6-4805-8579-9e3f5a390e25\") " pod="openshift-ingress/router-default-656bc596cc-m2fg7"
Jul 02 12:39:12.695684 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.695468    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/02f1bfff-40d6-4805-8579-9e3f5a390e25-stats-auth\") pod \"router-default-656bc596cc-m2fg7\" (UID: \"02f1bfff-40d6-4805-8579-9e3f5a390e25\") " pod="openshift-ingress/router-default-656bc596cc-m2fg7"
Jul 02 12:39:12.695684 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.695492    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chs5b\" (UniqueName: \"kubernetes.io/projected/dabc8a5d-7710-42b1-93e2-bf1a0f5c94e5-kube-api-access-chs5b\") pod \"insights-operator-5f67b55555-r4pwp\" (UID: \"dabc8a5d-7710-42b1-93e2-bf1a0f5c94e5\") " pod="openshift-insights/insights-operator-5f67b55555-r4pwp"
Jul 02 12:39:12.695684 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.695520    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ac58c65c-c830-4ba0-aef5-c5745ed26d93-telemetry-config\") pod \"cluster-monitoring-operator-54cb84584-dz8tm\" (UID: \"ac58c65c-c830-4ba0-aef5-c5745ed26d93\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-dz8tm"
Jul 02 12:39:12.695684 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.695548    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ac58c65c-c830-4ba0-aef5-c5745ed26d93-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-dz8tm\" (UID: \"ac58c65c-c830-4ba0-aef5-c5745ed26d93\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-dz8tm"
Jul 02 12:39:12.695684 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:12.695560    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 02 12:39:12.695684 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:12.695626    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02f1bfff-40d6-4805-8579-9e3f5a390e25-metrics-certs podName:02f1bfff-40d6-4805-8579-9e3f5a390e25 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:13.195607827 +0000 UTC m=+41.803070829 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02f1bfff-40d6-4805-8579-9e3f5a390e25-metrics-certs") pod "router-default-656bc596cc-m2fg7" (UID: "02f1bfff-40d6-4805-8579-9e3f5a390e25") : secret "router-metrics-certs-default" not found
Jul 02 12:39:12.695684 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.695677    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dabc8a5d-7710-42b1-93e2-bf1a0f5c94e5-tmp\") pod \"insights-operator-5f67b55555-r4pwp\" (UID: \"dabc8a5d-7710-42b1-93e2-bf1a0f5c94e5\") " pod="openshift-insights/insights-operator-5f67b55555-r4pwp"
Jul 02 12:39:12.696178 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.695722    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02f1bfff-40d6-4805-8579-9e3f5a390e25-service-ca-bundle\") pod \"router-default-656bc596cc-m2fg7\" (UID: \"02f1bfff-40d6-4805-8579-9e3f5a390e25\") " pod="openshift-ingress/router-default-656bc596cc-m2fg7"
Jul 02 12:39:12.696178 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.695749    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2gnb\" (UniqueName: \"kubernetes.io/projected/02f1bfff-40d6-4805-8579-9e3f5a390e25-kube-api-access-j2gnb\") pod \"router-default-656bc596cc-m2fg7\" (UID: \"02f1bfff-40d6-4805-8579-9e3f5a390e25\") " pod="openshift-ingress/router-default-656bc596cc-m2fg7"
Jul 02 12:39:12.696178 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:12.695833    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/02f1bfff-40d6-4805-8579-9e3f5a390e25-service-ca-bundle podName:02f1bfff-40d6-4805-8579-9e3f5a390e25 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:13.195808884 +0000 UTC m=+41.803271903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/02f1bfff-40d6-4805-8579-9e3f5a390e25-service-ca-bundle") pod "router-default-656bc596cc-m2fg7" (UID: "02f1bfff-40d6-4805-8579-9e3f5a390e25") : configmap references non-existent config key: service-ca.crt
Jul 02 12:39:12.699328 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.699300    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/02f1bfff-40d6-4805-8579-9e3f5a390e25-default-certificate\") pod \"router-default-656bc596cc-m2fg7\" (UID: \"02f1bfff-40d6-4805-8579-9e3f5a390e25\") " pod="openshift-ingress/router-default-656bc596cc-m2fg7"
Jul 02 12:39:12.699424 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.699333    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/02f1bfff-40d6-4805-8579-9e3f5a390e25-stats-auth\") pod \"router-default-656bc596cc-m2fg7\" (UID: \"02f1bfff-40d6-4805-8579-9e3f5a390e25\") " pod="openshift-ingress/router-default-656bc596cc-m2fg7"
Jul 02 12:39:12.705776 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.705752    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2gnb\" (UniqueName: \"kubernetes.io/projected/02f1bfff-40d6-4805-8579-9e3f5a390e25-kube-api-access-j2gnb\") pod \"router-default-656bc596cc-m2fg7\" (UID: \"02f1bfff-40d6-4805-8579-9e3f5a390e25\") " pod="openshift-ingress/router-default-656bc596cc-m2fg7"
Jul 02 12:39:12.796305 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.796265    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chs5b\" (UniqueName: \"kubernetes.io/projected/dabc8a5d-7710-42b1-93e2-bf1a0f5c94e5-kube-api-access-chs5b\") pod \"insights-operator-5f67b55555-r4pwp\" (UID: \"dabc8a5d-7710-42b1-93e2-bf1a0f5c94e5\") " pod="openshift-insights/insights-operator-5f67b55555-r4pwp"
Jul 02 12:39:12.796305 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.796310    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ac58c65c-c830-4ba0-aef5-c5745ed26d93-telemetry-config\") pod \"cluster-monitoring-operator-54cb84584-dz8tm\" (UID: \"ac58c65c-c830-4ba0-aef5-c5745ed26d93\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-dz8tm"
Jul 02 12:39:12.796542 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.796337    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ac58c65c-c830-4ba0-aef5-c5745ed26d93-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-dz8tm\" (UID: \"ac58c65c-c830-4ba0-aef5-c5745ed26d93\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-dz8tm"
Jul 02 12:39:12.796542 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:12.796444    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 02 12:39:12.796542 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:12.796503    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac58c65c-c830-4ba0-aef5-c5745ed26d93-cluster-monitoring-operator-tls podName:ac58c65c-c830-4ba0-aef5-c5745ed26d93 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:13.296484482 +0000 UTC m=+41.903947503 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ac58c65c-c830-4ba0-aef5-c5745ed26d93-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-dz8tm" (UID: "ac58c65c-c830-4ba0-aef5-c5745ed26d93") : secret "cluster-monitoring-operator-tls" not found
Jul 02 12:39:12.796542 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.796522    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dabc8a5d-7710-42b1-93e2-bf1a0f5c94e5-tmp\") pod \"insights-operator-5f67b55555-r4pwp\" (UID: \"dabc8a5d-7710-42b1-93e2-bf1a0f5c94e5\") " pod="openshift-insights/insights-operator-5f67b55555-r4pwp"
Jul 02 12:39:12.796760 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.796627    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dabc8a5d-7710-42b1-93e2-bf1a0f5c94e5-service-ca-bundle\") pod \"insights-operator-5f67b55555-r4pwp\" (UID: \"dabc8a5d-7710-42b1-93e2-bf1a0f5c94e5\") " pod="openshift-insights/insights-operator-5f67b55555-r4pwp"
Jul 02 12:39:12.796760 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.796654    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dabc8a5d-7710-42b1-93e2-bf1a0f5c94e5-serving-cert\") pod \"insights-operator-5f67b55555-r4pwp\" (UID: \"dabc8a5d-7710-42b1-93e2-bf1a0f5c94e5\") " pod="openshift-insights/insights-operator-5f67b55555-r4pwp"
Jul 02 12:39:12.796760 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.796699    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dabc8a5d-7710-42b1-93e2-bf1a0f5c94e5-trusted-ca-bundle\") pod \"insights-operator-5f67b55555-r4pwp\" (UID: \"dabc8a5d-7710-42b1-93e2-bf1a0f5c94e5\") " pod="openshift-insights/insights-operator-5f67b55555-r4pwp"
Jul 02 12:39:12.796760 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.796729    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6klkc\" (UniqueName: \"kubernetes.io/projected/ac58c65c-c830-4ba0-aef5-c5745ed26d93-kube-api-access-6klkc\") pod \"cluster-monitoring-operator-54cb84584-dz8tm\" (UID: \"ac58c65c-c830-4ba0-aef5-c5745ed26d93\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-dz8tm"
Jul 02 12:39:12.796760 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.796753    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/dabc8a5d-7710-42b1-93e2-bf1a0f5c94e5-snapshots\") pod \"insights-operator-5f67b55555-r4pwp\" (UID: \"dabc8a5d-7710-42b1-93e2-bf1a0f5c94e5\") " pod="openshift-insights/insights-operator-5f67b55555-r4pwp"
Jul 02 12:39:12.797025 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.796990    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dabc8a5d-7710-42b1-93e2-bf1a0f5c94e5-tmp\") pod \"insights-operator-5f67b55555-r4pwp\" (UID: \"dabc8a5d-7710-42b1-93e2-bf1a0f5c94e5\") " pod="openshift-insights/insights-operator-5f67b55555-r4pwp"
Jul 02 12:39:12.797324 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.797300    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dabc8a5d-7710-42b1-93e2-bf1a0f5c94e5-service-ca-bundle\") pod \"insights-operator-5f67b55555-r4pwp\" (UID: \"dabc8a5d-7710-42b1-93e2-bf1a0f5c94e5\") " pod="openshift-insights/insights-operator-5f67b55555-r4pwp"
Jul 02 12:39:12.797397 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.797326    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/dabc8a5d-7710-42b1-93e2-bf1a0f5c94e5-snapshots\") pod \"insights-operator-5f67b55555-r4pwp\" (UID: \"dabc8a5d-7710-42b1-93e2-bf1a0f5c94e5\") " pod="openshift-insights/insights-operator-5f67b55555-r4pwp"
Jul 02 12:39:12.797397 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.797342    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ac58c65c-c830-4ba0-aef5-c5745ed26d93-telemetry-config\") pod \"cluster-monitoring-operator-54cb84584-dz8tm\" (UID: \"ac58c65c-c830-4ba0-aef5-c5745ed26d93\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-dz8tm"
Jul 02 12:39:12.797528 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.797511    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dabc8a5d-7710-42b1-93e2-bf1a0f5c94e5-trusted-ca-bundle\") pod \"insights-operator-5f67b55555-r4pwp\" (UID: \"dabc8a5d-7710-42b1-93e2-bf1a0f5c94e5\") " pod="openshift-insights/insights-operator-5f67b55555-r4pwp"
Jul 02 12:39:12.799056 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.799004    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dabc8a5d-7710-42b1-93e2-bf1a0f5c94e5-serving-cert\") pod \"insights-operator-5f67b55555-r4pwp\" (UID: \"dabc8a5d-7710-42b1-93e2-bf1a0f5c94e5\") " pod="openshift-insights/insights-operator-5f67b55555-r4pwp"
Jul 02 12:39:12.827976 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.827950    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chs5b\" (UniqueName: \"kubernetes.io/projected/dabc8a5d-7710-42b1-93e2-bf1a0f5c94e5-kube-api-access-chs5b\") pod \"insights-operator-5f67b55555-r4pwp\" (UID: \"dabc8a5d-7710-42b1-93e2-bf1a0f5c94e5\") " pod="openshift-insights/insights-operator-5f67b55555-r4pwp"
Jul 02 12:39:12.828606 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.828588    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6klkc\" (UniqueName: \"kubernetes.io/projected/ac58c65c-c830-4ba0-aef5-c5745ed26d93-kube-api-access-6klkc\") pod \"cluster-monitoring-operator-54cb84584-dz8tm\" (UID: \"ac58c65c-c830-4ba0-aef5-c5745ed26d93\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-dz8tm"
Jul 02 12:39:12.851403 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:12.851380    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5f67b55555-r4pwp"
Jul 02 12:39:13.021837 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:13.021811    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5f67b55555-r4pwp"]
Jul 02 12:39:13.036718 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:39:13.036687    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddabc8a5d_7710_42b1_93e2_bf1a0f5c94e5.slice/crio-96a9c3b2859f2beb475fa0ba93bc5a7cb580d8c34d7580a9892654d00e4af955 WatchSource:0}: Error finding container 96a9c3b2859f2beb475fa0ba93bc5a7cb580d8c34d7580a9892654d00e4af955: Status 404 returned error can't find the container with id 96a9c3b2859f2beb475fa0ba93bc5a7cb580d8c34d7580a9892654d00e4af955
Jul 02 12:39:13.162317 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:13.162287    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5f67b55555-r4pwp" event={"ID":"dabc8a5d-7710-42b1-93e2-bf1a0f5c94e5","Type":"ContainerStarted","Data":"96a9c3b2859f2beb475fa0ba93bc5a7cb580d8c34d7580a9892654d00e4af955"}
Jul 02 12:39:13.198831 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:13.198812    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02f1bfff-40d6-4805-8579-9e3f5a390e25-service-ca-bundle\") pod \"router-default-656bc596cc-m2fg7\" (UID: \"02f1bfff-40d6-4805-8579-9e3f5a390e25\") " pod="openshift-ingress/router-default-656bc596cc-m2fg7"
Jul 02 12:39:13.198949 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:13.198886    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02f1bfff-40d6-4805-8579-9e3f5a390e25-metrics-certs\") pod \"router-default-656bc596cc-m2fg7\" (UID: \"02f1bfff-40d6-4805-8579-9e3f5a390e25\") " pod="openshift-ingress/router-default-656bc596cc-m2fg7"
Jul 02 12:39:13.199030 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:13.199012    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 02 12:39:13.199066 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:13.199016    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/02f1bfff-40d6-4805-8579-9e3f5a390e25-service-ca-bundle podName:02f1bfff-40d6-4805-8579-9e3f5a390e25 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:14.198996857 +0000 UTC m=+42.806459878 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/02f1bfff-40d6-4805-8579-9e3f5a390e25-service-ca-bundle") pod "router-default-656bc596cc-m2fg7" (UID: "02f1bfff-40d6-4805-8579-9e3f5a390e25") : configmap references non-existent config key: service-ca.crt
Jul 02 12:39:13.199104 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:13.199068    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02f1bfff-40d6-4805-8579-9e3f5a390e25-metrics-certs podName:02f1bfff-40d6-4805-8579-9e3f5a390e25 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:14.199057242 +0000 UTC m=+42.806520243 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02f1bfff-40d6-4805-8579-9e3f5a390e25-metrics-certs") pod "router-default-656bc596cc-m2fg7" (UID: "02f1bfff-40d6-4805-8579-9e3f5a390e25") : secret "router-metrics-certs-default" not found
Jul 02 12:39:13.299791 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:13.299722    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ac58c65c-c830-4ba0-aef5-c5745ed26d93-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-dz8tm\" (UID: \"ac58c65c-c830-4ba0-aef5-c5745ed26d93\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-dz8tm"
Jul 02 12:39:13.299892 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:13.299854    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 02 12:39:13.299978 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:13.299950    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac58c65c-c830-4ba0-aef5-c5745ed26d93-cluster-monitoring-operator-tls podName:ac58c65c-c830-4ba0-aef5-c5745ed26d93 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:14.299930114 +0000 UTC m=+42.907393129 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ac58c65c-c830-4ba0-aef5-c5745ed26d93-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-dz8tm" (UID: "ac58c65c-c830-4ba0-aef5-c5745ed26d93") : secret "cluster-monitoring-operator-tls" not found
Jul 02 12:39:14.206511 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:14.206477    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02f1bfff-40d6-4805-8579-9e3f5a390e25-metrics-certs\") pod \"router-default-656bc596cc-m2fg7\" (UID: \"02f1bfff-40d6-4805-8579-9e3f5a390e25\") " pod="openshift-ingress/router-default-656bc596cc-m2fg7"
Jul 02 12:39:14.206941 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:14.206544    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02f1bfff-40d6-4805-8579-9e3f5a390e25-service-ca-bundle\") pod \"router-default-656bc596cc-m2fg7\" (UID: \"02f1bfff-40d6-4805-8579-9e3f5a390e25\") " pod="openshift-ingress/router-default-656bc596cc-m2fg7"
Jul 02 12:39:14.206941 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:14.206642    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 02 12:39:14.206941 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:14.206713    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02f1bfff-40d6-4805-8579-9e3f5a390e25-metrics-certs podName:02f1bfff-40d6-4805-8579-9e3f5a390e25 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:16.206696175 +0000 UTC m=+44.814159176 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02f1bfff-40d6-4805-8579-9e3f5a390e25-metrics-certs") pod "router-default-656bc596cc-m2fg7" (UID: "02f1bfff-40d6-4805-8579-9e3f5a390e25") : secret "router-metrics-certs-default" not found
Jul 02 12:39:14.206941 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:14.206729    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/02f1bfff-40d6-4805-8579-9e3f5a390e25-service-ca-bundle podName:02f1bfff-40d6-4805-8579-9e3f5a390e25 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:16.206722887 +0000 UTC m=+44.814185889 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/02f1bfff-40d6-4805-8579-9e3f5a390e25-service-ca-bundle") pod "router-default-656bc596cc-m2fg7" (UID: "02f1bfff-40d6-4805-8579-9e3f5a390e25") : configmap references non-existent config key: service-ca.crt
Jul 02 12:39:14.307526 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:14.307494    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ac58c65c-c830-4ba0-aef5-c5745ed26d93-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-dz8tm\" (UID: \"ac58c65c-c830-4ba0-aef5-c5745ed26d93\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-dz8tm"
Jul 02 12:39:14.307701 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:14.307638    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 02 12:39:14.307758 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:14.307707    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac58c65c-c830-4ba0-aef5-c5745ed26d93-cluster-monitoring-operator-tls podName:ac58c65c-c830-4ba0-aef5-c5745ed26d93 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:16.307691768 +0000 UTC m=+44.915154769 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ac58c65c-c830-4ba0-aef5-c5745ed26d93-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-dz8tm" (UID: "ac58c65c-c830-4ba0-aef5-c5745ed26d93") : secret "cluster-monitoring-operator-tls" not found
Jul 02 12:39:14.711491 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:14.711453    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/116aa48e-717b-4d49-897d-5538398f4a2a-original-pull-secret\") pod \"global-pull-secret-syncer-72n2l\" (UID: \"116aa48e-717b-4d49-897d-5538398f4a2a\") " pod="kube-system/global-pull-secret-syncer-72n2l"
Jul 02 12:39:14.714155 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:14.714126    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/116aa48e-717b-4d49-897d-5538398f4a2a-original-pull-secret\") pod \"global-pull-secret-syncer-72n2l\" (UID: \"116aa48e-717b-4d49-897d-5538398f4a2a\") " pod="kube-system/global-pull-secret-syncer-72n2l"
Jul 02 12:39:14.782784 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:14.782741    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-72n2l"
Jul 02 12:39:14.926039 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:14.926009    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-72n2l"]
Jul 02 12:39:14.928802 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:39:14.928765    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod116aa48e_717b_4d49_897d_5538398f4a2a.slice/crio-4bff976d128eece2e0b1dbb5de72b7d60f9bcb52b2d938221b0242971b578938 WatchSource:0}: Error finding container 4bff976d128eece2e0b1dbb5de72b7d60f9bcb52b2d938221b0242971b578938: Status 404 returned error can't find the container with id 4bff976d128eece2e0b1dbb5de72b7d60f9bcb52b2d938221b0242971b578938
Jul 02 12:39:15.013796 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:15.013711    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69ae490b-bd41-4ab2-b05c-3393fea68351-registry-tls\") pod \"image-registry-847db68bb4-fgghh\" (UID: \"69ae490b-bd41-4ab2-b05c-3393fea68351\") " pod="openshift-image-registry/image-registry-847db68bb4-fgghh"
Jul 02 12:39:15.013796 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:15.013781    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c646b36-0375-43c3-a0eb-8e08a2821414-metrics-tls\") pod \"dns-default-s82bz\" (UID: \"0c646b36-0375-43c3-a0eb-8e08a2821414\") " pod="openshift-dns/dns-default-s82bz"
Jul 02 12:39:15.014040 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:15.013891    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 12:39:15.014040 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:15.013902    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 12:39:15.014040 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:15.013945    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-847db68bb4-fgghh: secret "image-registry-tls" not found
Jul 02 12:39:15.014040 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:15.013965    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c646b36-0375-43c3-a0eb-8e08a2821414-metrics-tls podName:0c646b36-0375-43c3-a0eb-8e08a2821414 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:23.013950192 +0000 UTC m=+51.621413193 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c646b36-0375-43c3-a0eb-8e08a2821414-metrics-tls") pod "dns-default-s82bz" (UID: "0c646b36-0375-43c3-a0eb-8e08a2821414") : secret "dns-default-metrics-tls" not found
Jul 02 12:39:15.014040 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:15.013996    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69ae490b-bd41-4ab2-b05c-3393fea68351-registry-tls podName:69ae490b-bd41-4ab2-b05c-3393fea68351 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:23.013979218 +0000 UTC m=+51.621442236 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/69ae490b-bd41-4ab2-b05c-3393fea68351-registry-tls") pod "image-registry-847db68bb4-fgghh" (UID: "69ae490b-bd41-4ab2-b05c-3393fea68351") : secret "image-registry-tls" not found
Jul 02 12:39:15.168246 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:15.168206    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-72n2l" event={"ID":"116aa48e-717b-4d49-897d-5538398f4a2a","Type":"ContainerStarted","Data":"4bff976d128eece2e0b1dbb5de72b7d60f9bcb52b2d938221b0242971b578938"}
Jul 02 12:39:16.225023 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:16.224252    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02f1bfff-40d6-4805-8579-9e3f5a390e25-metrics-certs\") pod \"router-default-656bc596cc-m2fg7\" (UID: \"02f1bfff-40d6-4805-8579-9e3f5a390e25\") " pod="openshift-ingress/router-default-656bc596cc-m2fg7"
Jul 02 12:39:16.225023 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:16.224345    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02f1bfff-40d6-4805-8579-9e3f5a390e25-service-ca-bundle\") pod \"router-default-656bc596cc-m2fg7\" (UID: \"02f1bfff-40d6-4805-8579-9e3f5a390e25\") " pod="openshift-ingress/router-default-656bc596cc-m2fg7"
Jul 02 12:39:16.225023 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:16.224508    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/02f1bfff-40d6-4805-8579-9e3f5a390e25-service-ca-bundle podName:02f1bfff-40d6-4805-8579-9e3f5a390e25 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:20.224490448 +0000 UTC m=+48.831953454 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/02f1bfff-40d6-4805-8579-9e3f5a390e25-service-ca-bundle") pod "router-default-656bc596cc-m2fg7" (UID: "02f1bfff-40d6-4805-8579-9e3f5a390e25") : configmap references non-existent config key: service-ca.crt
Jul 02 12:39:16.225023 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:16.224945    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 02 12:39:16.225023 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:16.225000    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02f1bfff-40d6-4805-8579-9e3f5a390e25-metrics-certs podName:02f1bfff-40d6-4805-8579-9e3f5a390e25 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:20.224982158 +0000 UTC m=+48.832445173 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02f1bfff-40d6-4805-8579-9e3f5a390e25-metrics-certs") pod "router-default-656bc596cc-m2fg7" (UID: "02f1bfff-40d6-4805-8579-9e3f5a390e25") : secret "router-metrics-certs-default" not found
Jul 02 12:39:16.325595 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:16.325566    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ac58c65c-c830-4ba0-aef5-c5745ed26d93-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-dz8tm\" (UID: \"ac58c65c-c830-4ba0-aef5-c5745ed26d93\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-dz8tm"
Jul 02 12:39:16.325733 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:16.325715    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 02 12:39:16.325809 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:16.325781    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac58c65c-c830-4ba0-aef5-c5745ed26d93-cluster-monitoring-operator-tls podName:ac58c65c-c830-4ba0-aef5-c5745ed26d93 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:20.325765567 +0000 UTC m=+48.933228569 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ac58c65c-c830-4ba0-aef5-c5745ed26d93-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-dz8tm" (UID: "ac58c65c-c830-4ba0-aef5-c5745ed26d93") : secret "cluster-monitoring-operator-tls" not found
Jul 02 12:39:17.174766 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:17.174726    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5f67b55555-r4pwp" event={"ID":"dabc8a5d-7710-42b1-93e2-bf1a0f5c94e5","Type":"ContainerStarted","Data":"79d2898796f8d9c8a5b8b8d61cc47d3ebe040adcd915389a3485d6dda6dcfb13"}
Jul 02 12:39:18.917959 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:18.917879    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5f67b55555-r4pwp" podStartSLOduration=3.754979031 podStartE2EDuration="6.917855343s" totalImagesPullingTime="3.162876312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:39:17.192698921 +0000 UTC m=+45.800161945" watchObservedRunningTime="2026-07-02 12:39:18.917855343 +0000 UTC m=+47.525318368"
Jul 02 12:39:18.919088 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:18.919057    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xn552"]
Jul 02 12:39:18.942329 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:18.942300    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xn552"]
Jul 02 12:39:18.942471 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:18.942413    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xn552"
Jul 02 12:39:18.944944 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:18.944900    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 02 12:39:18.945081 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:18.945061    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 02 12:39:18.945157 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:18.945104    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qjvpt\""
Jul 02 12:39:19.049412 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:19.049375    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/880228b8-bf07-4d91-802b-f05146179c2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xn552\" (UID: \"880228b8-bf07-4d91-802b-f05146179c2a\") " pod="openshift-insights/insights-runtime-extractor-xn552"
Jul 02 12:39:19.049585 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:19.049438    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/880228b8-bf07-4d91-802b-f05146179c2a-crio-socket\") pod \"insights-runtime-extractor-xn552\" (UID: \"880228b8-bf07-4d91-802b-f05146179c2a\") " pod="openshift-insights/insights-runtime-extractor-xn552"
Jul 02 12:39:19.049585 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:19.049463    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwt9t\" (UniqueName: \"kubernetes.io/projected/880228b8-bf07-4d91-802b-f05146179c2a-kube-api-access-lwt9t\") pod \"insights-runtime-extractor-xn552\" (UID: \"880228b8-bf07-4d91-802b-f05146179c2a\") " pod="openshift-insights/insights-runtime-extractor-xn552"
Jul 02 12:39:19.049585 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:19.049558    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/880228b8-bf07-4d91-802b-f05146179c2a-data-volume\") pod \"insights-runtime-extractor-xn552\" (UID: \"880228b8-bf07-4d91-802b-f05146179c2a\") " pod="openshift-insights/insights-runtime-extractor-xn552"
Jul 02 12:39:19.049721 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:19.049604    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/880228b8-bf07-4d91-802b-f05146179c2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xn552\" (UID: \"880228b8-bf07-4d91-802b-f05146179c2a\") " pod="openshift-insights/insights-runtime-extractor-xn552"
Jul 02 12:39:19.150351 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:19.150319    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/880228b8-bf07-4d91-802b-f05146179c2a-crio-socket\") pod \"insights-runtime-extractor-xn552\" (UID: \"880228b8-bf07-4d91-802b-f05146179c2a\") " pod="openshift-insights/insights-runtime-extractor-xn552"
Jul 02 12:39:19.150524 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:19.150368    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwt9t\" (UniqueName: \"kubernetes.io/projected/880228b8-bf07-4d91-802b-f05146179c2a-kube-api-access-lwt9t\") pod \"insights-runtime-extractor-xn552\" (UID: \"880228b8-bf07-4d91-802b-f05146179c2a\") " pod="openshift-insights/insights-runtime-extractor-xn552"
Jul 02 12:39:19.150524 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:19.150414    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/880228b8-bf07-4d91-802b-f05146179c2a-data-volume\") pod \"insights-runtime-extractor-xn552\" (UID: \"880228b8-bf07-4d91-802b-f05146179c2a\") " pod="openshift-insights/insights-runtime-extractor-xn552"
Jul 02 12:39:19.150524 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:19.150442    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/880228b8-bf07-4d91-802b-f05146179c2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xn552\" (UID: \"880228b8-bf07-4d91-802b-f05146179c2a\") " pod="openshift-insights/insights-runtime-extractor-xn552"
Jul 02 12:39:19.150648 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:19.150526    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/880228b8-bf07-4d91-802b-f05146179c2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xn552\" (UID: \"880228b8-bf07-4d91-802b-f05146179c2a\") " pod="openshift-insights/insights-runtime-extractor-xn552"
Jul 02 12:39:19.150648 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:19.150569    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/880228b8-bf07-4d91-802b-f05146179c2a-crio-socket\") pod \"insights-runtime-extractor-xn552\" (UID: \"880228b8-bf07-4d91-802b-f05146179c2a\") " pod="openshift-insights/insights-runtime-extractor-xn552"
Jul 02 12:39:19.150648 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:19.150612    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 12:39:19.150792 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:19.150671    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 12:39:19.150792 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:19.150739    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/880228b8-bf07-4d91-802b-f05146179c2a-kube-rbac-proxy-cm podName:880228b8-bf07-4d91-802b-f05146179c2a nodeName:}" failed. No retries permitted until 2026-07-02 12:39:19.650716229 +0000 UTC m=+48.258179241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/880228b8-bf07-4d91-802b-f05146179c2a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xn552" (UID: "880228b8-bf07-4d91-802b-f05146179c2a") : configmap "kube-rbac-proxy" not found
Jul 02 12:39:19.150792 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:19.150761    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/880228b8-bf07-4d91-802b-f05146179c2a-insights-runtime-extractor-tls podName:880228b8-bf07-4d91-802b-f05146179c2a nodeName:}" failed. No retries permitted until 2026-07-02 12:39:19.650750355 +0000 UTC m=+48.258213374 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/880228b8-bf07-4d91-802b-f05146179c2a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xn552" (UID: "880228b8-bf07-4d91-802b-f05146179c2a") : secret "insights-runtime-extractor-tls" not found
Jul 02 12:39:19.150980 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:19.150822    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/880228b8-bf07-4d91-802b-f05146179c2a-data-volume\") pod \"insights-runtime-extractor-xn552\" (UID: \"880228b8-bf07-4d91-802b-f05146179c2a\") " pod="openshift-insights/insights-runtime-extractor-xn552"
Jul 02 12:39:19.159399 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:19.159379    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwt9t\" (UniqueName: \"kubernetes.io/projected/880228b8-bf07-4d91-802b-f05146179c2a-kube-api-access-lwt9t\") pod \"insights-runtime-extractor-xn552\" (UID: \"880228b8-bf07-4d91-802b-f05146179c2a\") " pod="openshift-insights/insights-runtime-extractor-xn552"
Jul 02 12:39:19.655551 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:19.655507    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/880228b8-bf07-4d91-802b-f05146179c2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xn552\" (UID: \"880228b8-bf07-4d91-802b-f05146179c2a\") " pod="openshift-insights/insights-runtime-extractor-xn552"
Jul 02 12:39:19.655744 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:19.655612    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/880228b8-bf07-4d91-802b-f05146179c2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xn552\" (UID: \"880228b8-bf07-4d91-802b-f05146179c2a\") " pod="openshift-insights/insights-runtime-extractor-xn552"
Jul 02 12:39:19.655744 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:19.655687    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 12:39:19.655744 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:19.655708    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 12:39:19.655948 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:19.655765    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/880228b8-bf07-4d91-802b-f05146179c2a-insights-runtime-extractor-tls podName:880228b8-bf07-4d91-802b-f05146179c2a nodeName:}" failed. No retries permitted until 2026-07-02 12:39:20.655743846 +0000 UTC m=+49.263206848 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/880228b8-bf07-4d91-802b-f05146179c2a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xn552" (UID: "880228b8-bf07-4d91-802b-f05146179c2a") : secret "insights-runtime-extractor-tls" not found
Jul 02 12:39:19.655948 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:19.655784    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/880228b8-bf07-4d91-802b-f05146179c2a-kube-rbac-proxy-cm podName:880228b8-bf07-4d91-802b-f05146179c2a nodeName:}" failed. No retries permitted until 2026-07-02 12:39:20.65577444 +0000 UTC m=+49.263237441 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/880228b8-bf07-4d91-802b-f05146179c2a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xn552" (UID: "880228b8-bf07-4d91-802b-f05146179c2a") : configmap "kube-rbac-proxy" not found
Jul 02 12:39:20.260860 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:20.260820    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02f1bfff-40d6-4805-8579-9e3f5a390e25-metrics-certs\") pod \"router-default-656bc596cc-m2fg7\" (UID: \"02f1bfff-40d6-4805-8579-9e3f5a390e25\") " pod="openshift-ingress/router-default-656bc596cc-m2fg7"
Jul 02 12:39:20.261301 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:20.260968    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02f1bfff-40d6-4805-8579-9e3f5a390e25-service-ca-bundle\") pod \"router-default-656bc596cc-m2fg7\" (UID: \"02f1bfff-40d6-4805-8579-9e3f5a390e25\") " pod="openshift-ingress/router-default-656bc596cc-m2fg7"
Jul 02 12:39:20.261301 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:20.261004    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 02 12:39:20.261301 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:20.261079    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02f1bfff-40d6-4805-8579-9e3f5a390e25-metrics-certs podName:02f1bfff-40d6-4805-8579-9e3f5a390e25 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:28.261058943 +0000 UTC m=+56.868521945 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02f1bfff-40d6-4805-8579-9e3f5a390e25-metrics-certs") pod "router-default-656bc596cc-m2fg7" (UID: "02f1bfff-40d6-4805-8579-9e3f5a390e25") : secret "router-metrics-certs-default" not found
Jul 02 12:39:20.261301 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:20.261125    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/02f1bfff-40d6-4805-8579-9e3f5a390e25-service-ca-bundle podName:02f1bfff-40d6-4805-8579-9e3f5a390e25 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:28.261113462 +0000 UTC m=+56.868576466 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/02f1bfff-40d6-4805-8579-9e3f5a390e25-service-ca-bundle") pod "router-default-656bc596cc-m2fg7" (UID: "02f1bfff-40d6-4805-8579-9e3f5a390e25") : configmap references non-existent config key: service-ca.crt
Jul 02 12:39:20.361988 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:20.361957    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ac58c65c-c830-4ba0-aef5-c5745ed26d93-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-dz8tm\" (UID: \"ac58c65c-c830-4ba0-aef5-c5745ed26d93\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-dz8tm"
Jul 02 12:39:20.362134 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:20.362110    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 02 12:39:20.362187 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:20.362177    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac58c65c-c830-4ba0-aef5-c5745ed26d93-cluster-monitoring-operator-tls podName:ac58c65c-c830-4ba0-aef5-c5745ed26d93 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:28.362160752 +0000 UTC m=+56.969623754 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ac58c65c-c830-4ba0-aef5-c5745ed26d93-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-dz8tm" (UID: "ac58c65c-c830-4ba0-aef5-c5745ed26d93") : secret "cluster-monitoring-operator-tls" not found
Jul 02 12:39:20.520419 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:20.520360    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hmjl6_fa04af49-8c6c-46fa-bfcc-500d89b526c9/dns-node-resolver/0.log"
Jul 02 12:39:20.663337 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:20.663308    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/880228b8-bf07-4d91-802b-f05146179c2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xn552\" (UID: \"880228b8-bf07-4d91-802b-f05146179c2a\") " pod="openshift-insights/insights-runtime-extractor-xn552"
Jul 02 12:39:20.663485 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:20.663391    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/880228b8-bf07-4d91-802b-f05146179c2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xn552\" (UID: \"880228b8-bf07-4d91-802b-f05146179c2a\") " pod="openshift-insights/insights-runtime-extractor-xn552"
Jul 02 12:39:20.663485 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:20.663464    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 12:39:20.663559 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:20.663522    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/880228b8-bf07-4d91-802b-f05146179c2a-insights-runtime-extractor-tls podName:880228b8-bf07-4d91-802b-f05146179c2a nodeName:}" failed. No retries permitted until 2026-07-02 12:39:22.663507868 +0000 UTC m=+51.270970872 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/880228b8-bf07-4d91-802b-f05146179c2a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xn552" (UID: "880228b8-bf07-4d91-802b-f05146179c2a") : secret "insights-runtime-extractor-tls" not found
Jul 02 12:39:20.663559 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:20.663467    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 12:39:20.663628 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:20.663603    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/880228b8-bf07-4d91-802b-f05146179c2a-kube-rbac-proxy-cm podName:880228b8-bf07-4d91-802b-f05146179c2a nodeName:}" failed. No retries permitted until 2026-07-02 12:39:22.663587351 +0000 UTC m=+51.271050351 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/880228b8-bf07-4d91-802b-f05146179c2a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xn552" (UID: "880228b8-bf07-4d91-802b-f05146179c2a") : configmap "kube-rbac-proxy" not found
Jul 02 12:39:21.186453 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:21.186421    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-72n2l" event={"ID":"116aa48e-717b-4d49-897d-5538398f4a2a","Type":"ContainerStarted","Data":"c31ad515e77147eff50511c1a292435de4ce16c643639e8eff79cb3aa8fc2a2c"}
Jul 02 12:39:21.202466 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:21.202424    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-72n2l" podStartSLOduration=33.343503196 podStartE2EDuration="39.202411378s" totalImagesPullingTime="5.858908182s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 12:38:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:39:21.201788357 +0000 UTC m=+49.809251377" watchObservedRunningTime="2026-07-02 12:39:21.202411378 +0000 UTC m=+49.809874400"
Jul 02 12:39:21.519295 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:21.519224    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mztvs_5f219618-c523-45c1-9042-7e46c4994e49/node-ca/0.log"
Jul 02 12:39:22.119448 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:22.119420    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-shm9z_8d276bfd-8b47-48ae-a99c-6e4f069e0fd5/serve-healthcheck-canary/0.log"
Jul 02 12:39:22.680249 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:22.680212    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/880228b8-bf07-4d91-802b-f05146179c2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xn552\" (UID: \"880228b8-bf07-4d91-802b-f05146179c2a\") " pod="openshift-insights/insights-runtime-extractor-xn552"
Jul 02 12:39:22.680732 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:22.680317    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/880228b8-bf07-4d91-802b-f05146179c2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xn552\" (UID: \"880228b8-bf07-4d91-802b-f05146179c2a\") " pod="openshift-insights/insights-runtime-extractor-xn552"
Jul 02 12:39:22.680732 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:22.680435    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 12:39:22.680732 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:22.680494    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/880228b8-bf07-4d91-802b-f05146179c2a-insights-runtime-extractor-tls podName:880228b8-bf07-4d91-802b-f05146179c2a nodeName:}" failed. No retries permitted until 2026-07-02 12:39:26.680476506 +0000 UTC m=+55.287939530 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/880228b8-bf07-4d91-802b-f05146179c2a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xn552" (UID: "880228b8-bf07-4d91-802b-f05146179c2a") : secret "insights-runtime-extractor-tls" not found
Jul 02 12:39:22.680982 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:22.680938    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/880228b8-bf07-4d91-802b-f05146179c2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xn552\" (UID: \"880228b8-bf07-4d91-802b-f05146179c2a\") " pod="openshift-insights/insights-runtime-extractor-xn552"
Jul 02 12:39:23.083561 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:23.083483    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c646b36-0375-43c3-a0eb-8e08a2821414-metrics-tls\") pod \"dns-default-s82bz\" (UID: \"0c646b36-0375-43c3-a0eb-8e08a2821414\") " pod="openshift-dns/dns-default-s82bz"
Jul 02 12:39:23.083561 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:23.083555    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69ae490b-bd41-4ab2-b05c-3393fea68351-registry-tls\") pod \"image-registry-847db68bb4-fgghh\" (UID: \"69ae490b-bd41-4ab2-b05c-3393fea68351\") " pod="openshift-image-registry/image-registry-847db68bb4-fgghh"
Jul 02 12:39:23.083740 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:23.083641    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 12:39:23.083740 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:23.083707    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c646b36-0375-43c3-a0eb-8e08a2821414-metrics-tls podName:0c646b36-0375-43c3-a0eb-8e08a2821414 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:39.083691205 +0000 UTC m=+67.691154205 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c646b36-0375-43c3-a0eb-8e08a2821414-metrics-tls") pod "dns-default-s82bz" (UID: "0c646b36-0375-43c3-a0eb-8e08a2821414") : secret "dns-default-metrics-tls" not found
Jul 02 12:39:23.083740 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:23.083709    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 12:39:23.083740 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:23.083726    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-847db68bb4-fgghh: secret "image-registry-tls" not found
Jul 02 12:39:23.083867 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:23.083777    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69ae490b-bd41-4ab2-b05c-3393fea68351-registry-tls podName:69ae490b-bd41-4ab2-b05c-3393fea68351 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:39.083764013 +0000 UTC m=+67.691227015 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/69ae490b-bd41-4ab2-b05c-3393fea68351-registry-tls") pod "image-registry-847db68bb4-fgghh" (UID: "69ae490b-bd41-4ab2-b05c-3393fea68351") : secret "image-registry-tls" not found
Jul 02 12:39:26.708515 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:26.708475    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/880228b8-bf07-4d91-802b-f05146179c2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xn552\" (UID: \"880228b8-bf07-4d91-802b-f05146179c2a\") " pod="openshift-insights/insights-runtime-extractor-xn552"
Jul 02 12:39:26.708990 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:26.708625    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 12:39:26.708990 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:26.708697    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/880228b8-bf07-4d91-802b-f05146179c2a-insights-runtime-extractor-tls podName:880228b8-bf07-4d91-802b-f05146179c2a nodeName:}" failed. No retries permitted until 2026-07-02 12:39:34.708679999 +0000 UTC m=+63.316143009 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/880228b8-bf07-4d91-802b-f05146179c2a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xn552" (UID: "880228b8-bf07-4d91-802b-f05146179c2a") : secret "insights-runtime-extractor-tls" not found
Jul 02 12:39:28.321278 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:28.321246    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02f1bfff-40d6-4805-8579-9e3f5a390e25-metrics-certs\") pod \"router-default-656bc596cc-m2fg7\" (UID: \"02f1bfff-40d6-4805-8579-9e3f5a390e25\") " pod="openshift-ingress/router-default-656bc596cc-m2fg7"
Jul 02 12:39:28.321632 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:28.321309    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02f1bfff-40d6-4805-8579-9e3f5a390e25-service-ca-bundle\") pod \"router-default-656bc596cc-m2fg7\" (UID: \"02f1bfff-40d6-4805-8579-9e3f5a390e25\") " pod="openshift-ingress/router-default-656bc596cc-m2fg7"
Jul 02 12:39:28.321632 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:28.321392    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 02 12:39:28.321632 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:28.321422    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/02f1bfff-40d6-4805-8579-9e3f5a390e25-service-ca-bundle podName:02f1bfff-40d6-4805-8579-9e3f5a390e25 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:44.321408814 +0000 UTC m=+72.928871815 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/02f1bfff-40d6-4805-8579-9e3f5a390e25-service-ca-bundle") pod "router-default-656bc596cc-m2fg7" (UID: "02f1bfff-40d6-4805-8579-9e3f5a390e25") : configmap references non-existent config key: service-ca.crt
Jul 02 12:39:28.321632 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:28.321442    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02f1bfff-40d6-4805-8579-9e3f5a390e25-metrics-certs podName:02f1bfff-40d6-4805-8579-9e3f5a390e25 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:44.321429362 +0000 UTC m=+72.928892363 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02f1bfff-40d6-4805-8579-9e3f5a390e25-metrics-certs") pod "router-default-656bc596cc-m2fg7" (UID: "02f1bfff-40d6-4805-8579-9e3f5a390e25") : secret "router-metrics-certs-default" not found
Jul 02 12:39:28.422078 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:28.422048    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ac58c65c-c830-4ba0-aef5-c5745ed26d93-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-dz8tm\" (UID: \"ac58c65c-c830-4ba0-aef5-c5745ed26d93\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-dz8tm"
Jul 02 12:39:28.422214 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:28.422193    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 02 12:39:28.422255 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:28.422249    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac58c65c-c830-4ba0-aef5-c5745ed26d93-cluster-monitoring-operator-tls podName:ac58c65c-c830-4ba0-aef5-c5745ed26d93 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:44.422235853 +0000 UTC m=+73.029698854 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ac58c65c-c830-4ba0-aef5-c5745ed26d93-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-dz8tm" (UID: "ac58c65c-c830-4ba0-aef5-c5745ed26d93") : secret "cluster-monitoring-operator-tls" not found
Jul 02 12:39:30.122978 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:30.122950    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2c7ks"
Jul 02 12:39:34.771644 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:34.771605    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/880228b8-bf07-4d91-802b-f05146179c2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xn552\" (UID: \"880228b8-bf07-4d91-802b-f05146179c2a\") " pod="openshift-insights/insights-runtime-extractor-xn552"
Jul 02 12:39:34.772123 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:34.771724    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 12:39:34.772123 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:34.771794    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/880228b8-bf07-4d91-802b-f05146179c2a-insights-runtime-extractor-tls podName:880228b8-bf07-4d91-802b-f05146179c2a nodeName:}" failed. No retries permitted until 2026-07-02 12:39:50.77177792 +0000 UTC m=+79.379240921 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/880228b8-bf07-4d91-802b-f05146179c2a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xn552" (UID: "880228b8-bf07-4d91-802b-f05146179c2a") : secret "insights-runtime-extractor-tls" not found
Jul 02 12:39:36.887881 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:36.887837    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7-metrics-certs\") pod \"network-metrics-daemon-sbnql\" (UID: \"86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7\") " pod="openshift-multus/network-metrics-daemon-sbnql"
Jul 02 12:39:36.894243 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:36.894223    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 02 12:39:36.898629 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:36.898612    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 02 12:39:36.898680 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:36.898671    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7-metrics-certs podName:86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7 nodeName:}" failed. No retries permitted until 2026-07-02 12:40:40.898652901 +0000 UTC m=+129.506115901 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7-metrics-certs") pod "network-metrics-daemon-sbnql" (UID: "86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7") : secret "metrics-daemon-secret" not found
Jul 02 12:39:37.089375 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:37.089326    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zvwz\" (UniqueName: \"kubernetes.io/projected/ebc69c1d-a909-44d0-bcb1-691e39c5e754-kube-api-access-2zvwz\") pod \"network-check-target-tczr2\" (UID: \"ebc69c1d-a909-44d0-bcb1-691e39c5e754\") " pod="openshift-network-diagnostics/network-check-target-tczr2"
Jul 02 12:39:37.093175 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:37.093158    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 02 12:39:37.102989 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:37.102970    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 02 12:39:37.112595 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:37.112567    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zvwz\" (UniqueName: \"kubernetes.io/projected/ebc69c1d-a909-44d0-bcb1-691e39c5e754-kube-api-access-2zvwz\") pod \"network-check-target-tczr2\" (UID: \"ebc69c1d-a909-44d0-bcb1-691e39c5e754\") " pod="openshift-network-diagnostics/network-check-target-tczr2"
Jul 02 12:39:37.280020 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:37.279992    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hkdpl\""
Jul 02 12:39:37.286862 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:37.286840    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tczr2"
Jul 02 12:39:37.416016 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:37.415987    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tczr2"]
Jul 02 12:39:37.419201 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:39:37.419176    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebc69c1d_a909_44d0_bcb1_691e39c5e754.slice/crio-0a6611d0cad413a9ab462179a6ab30e83911d86de386ab775557babd4c5911d3 WatchSource:0}: Error finding container 0a6611d0cad413a9ab462179a6ab30e83911d86de386ab775557babd4c5911d3: Status 404 returned error can't find the container with id 0a6611d0cad413a9ab462179a6ab30e83911d86de386ab775557babd4c5911d3
Jul 02 12:39:38.226961 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:38.226905    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tczr2" event={"ID":"ebc69c1d-a909-44d0-bcb1-691e39c5e754","Type":"ContainerStarted","Data":"0a6611d0cad413a9ab462179a6ab30e83911d86de386ab775557babd4c5911d3"}
Jul 02 12:39:39.106872 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:39.106833    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c646b36-0375-43c3-a0eb-8e08a2821414-metrics-tls\") pod \"dns-default-s82bz\" (UID: \"0c646b36-0375-43c3-a0eb-8e08a2821414\") " pod="openshift-dns/dns-default-s82bz"
Jul 02 12:39:39.107079 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:39.106947    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69ae490b-bd41-4ab2-b05c-3393fea68351-registry-tls\") pod \"image-registry-847db68bb4-fgghh\" (UID: \"69ae490b-bd41-4ab2-b05c-3393fea68351\") " pod="openshift-image-registry/image-registry-847db68bb4-fgghh"
Jul 02 12:39:39.109495 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:39.109467    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c646b36-0375-43c3-a0eb-8e08a2821414-metrics-tls\") pod \"dns-default-s82bz\" (UID: \"0c646b36-0375-43c3-a0eb-8e08a2821414\") " pod="openshift-dns/dns-default-s82bz"
Jul 02 12:39:39.127126 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:39.127097    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69ae490b-bd41-4ab2-b05c-3393fea68351-registry-tls\") pod \"image-registry-847db68bb4-fgghh\" (UID: \"69ae490b-bd41-4ab2-b05c-3393fea68351\") " pod="openshift-image-registry/image-registry-847db68bb4-fgghh"
Jul 02 12:39:39.376416 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:39.376345    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-58r25\""
Jul 02 12:39:39.384287 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:39.384262    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-847db68bb4-fgghh"
Jul 02 12:39:39.394383 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:39.394363    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-l95ls\""
Jul 02 12:39:39.402216 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:39.402198    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-s82bz"
Jul 02 12:39:39.555299 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:39.555265    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-847db68bb4-fgghh"]
Jul 02 12:39:39.558502 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:39:39.558470    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69ae490b_bd41_4ab2_b05c_3393fea68351.slice/crio-3acee6408a31cb6ec7d6a712b470473d564b8524586f99d7b7e846ea92653741 WatchSource:0}: Error finding container 3acee6408a31cb6ec7d6a712b470473d564b8524586f99d7b7e846ea92653741: Status 404 returned error can't find the container with id 3acee6408a31cb6ec7d6a712b470473d564b8524586f99d7b7e846ea92653741
Jul 02 12:39:39.584725 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:39.584673    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-s82bz"]
Jul 02 12:39:39.588148 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:39:39.588121    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c646b36_0375_43c3_a0eb_8e08a2821414.slice/crio-fd267a2292fcba0ac76191d622e2c970a2b097a0fa15756da055c2925a4d17fd WatchSource:0}: Error finding container fd267a2292fcba0ac76191d622e2c970a2b097a0fa15756da055c2925a4d17fd: Status 404 returned error can't find the container with id fd267a2292fcba0ac76191d622e2c970a2b097a0fa15756da055c2925a4d17fd
Jul 02 12:39:40.233902 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:40.233866    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-s82bz" event={"ID":"0c646b36-0375-43c3-a0eb-8e08a2821414","Type":"ContainerStarted","Data":"fd267a2292fcba0ac76191d622e2c970a2b097a0fa15756da055c2925a4d17fd"}
Jul 02 12:39:40.235505 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:40.235477    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-847db68bb4-fgghh" event={"ID":"69ae490b-bd41-4ab2-b05c-3393fea68351","Type":"ContainerStarted","Data":"c911f71aa4fdbca6c227b897a68ad926809fe144d43c6d16f013a9d0b7204cdb"}
Jul 02 12:39:40.235647 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:40.235523    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-847db68bb4-fgghh" event={"ID":"69ae490b-bd41-4ab2-b05c-3393fea68351","Type":"ContainerStarted","Data":"3acee6408a31cb6ec7d6a712b470473d564b8524586f99d7b7e846ea92653741"}
Jul 02 12:39:40.235647 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:40.235562    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-847db68bb4-fgghh"
Jul 02 12:39:40.271952 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:40.271891    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-847db68bb4-fgghh" podStartSLOduration=67.271876661 podStartE2EDuration="1m7.271876661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:39:40.270267481 +0000 UTC m=+68.877730529" watchObservedRunningTime="2026-07-02 12:39:40.271876661 +0000 UTC m=+68.879339678"
Jul 02 12:39:41.240929 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:41.240868    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tczr2" event={"ID":"ebc69c1d-a909-44d0-bcb1-691e39c5e754","Type":"ContainerStarted","Data":"b06ee9d7cb7bad37391c1c148467d7f5c3fdaddd9917c9f5795bfb2967217189"}
Jul 02 12:39:41.241376 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:41.241135    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tczr2"
Jul 02 12:39:41.261993 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:41.261947    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tczr2" podStartSLOduration=64.952186137 podStartE2EDuration="1m8.261932505s" totalImagesPullingTime="3.309746368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:39:41.260887102 +0000 UTC m=+69.868350129" watchObservedRunningTime="2026-07-02 12:39:41.261932505 +0000 UTC m=+69.869395518"
Jul 02 12:39:42.246819 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:42.246783    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-s82bz" event={"ID":"0c646b36-0375-43c3-a0eb-8e08a2821414","Type":"ContainerStarted","Data":"0168a00ece48216166bccde9367462227259b0b7b77f59e03276bd7ee6b3a105"}
Jul 02 12:39:42.247220 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:42.246867    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-s82bz" event={"ID":"0c646b36-0375-43c3-a0eb-8e08a2821414","Type":"ContainerStarted","Data":"a4988f0d9b24b7a08bfa1b4bcc183d1fa8444729e4ec20c4f5e626e71e338b97"}
Jul 02 12:39:42.247220 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:42.246895    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-s82bz"
Jul 02 12:39:42.282797 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:42.282747    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-s82bz" podStartSLOduration=33.081280112 podStartE2EDuration="35.282732389s" totalImagesPullingTime="2.201452277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:39:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:39:42.282680931 +0000 UTC m=+70.890143955" watchObservedRunningTime="2026-07-02 12:39:42.282732389 +0000 UTC m=+70.890195411"
Jul 02 12:39:44.350209 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:44.350172    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02f1bfff-40d6-4805-8579-9e3f5a390e25-service-ca-bundle\") pod \"router-default-656bc596cc-m2fg7\" (UID: \"02f1bfff-40d6-4805-8579-9e3f5a390e25\") " pod="openshift-ingress/router-default-656bc596cc-m2fg7"
Jul 02 12:39:44.350582 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:44.350286    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02f1bfff-40d6-4805-8579-9e3f5a390e25-metrics-certs\") pod \"router-default-656bc596cc-m2fg7\" (UID: \"02f1bfff-40d6-4805-8579-9e3f5a390e25\") " pod="openshift-ingress/router-default-656bc596cc-m2fg7"
Jul 02 12:39:44.350756 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:44.350734    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02f1bfff-40d6-4805-8579-9e3f5a390e25-service-ca-bundle\") pod \"router-default-656bc596cc-m2fg7\" (UID: \"02f1bfff-40d6-4805-8579-9e3f5a390e25\") " pod="openshift-ingress/router-default-656bc596cc-m2fg7"
Jul 02 12:39:44.352566 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:44.352544    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02f1bfff-40d6-4805-8579-9e3f5a390e25-metrics-certs\") pod \"router-default-656bc596cc-m2fg7\" (UID: \"02f1bfff-40d6-4805-8579-9e3f5a390e25\") " pod="openshift-ingress/router-default-656bc596cc-m2fg7"
Jul 02 12:39:44.451547 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:44.451510    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ac58c65c-c830-4ba0-aef5-c5745ed26d93-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-dz8tm\" (UID: \"ac58c65c-c830-4ba0-aef5-c5745ed26d93\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-dz8tm"
Jul 02 12:39:44.453873 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:44.453847    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ac58c65c-c830-4ba0-aef5-c5745ed26d93-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-dz8tm\" (UID: \"ac58c65c-c830-4ba0-aef5-c5745ed26d93\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-dz8tm"
Jul 02 12:39:44.542595 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:44.542568    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-vxtl6\""
Jul 02 12:39:44.550545 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:44.550526    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-656bc596cc-m2fg7"
Jul 02 12:39:44.637129 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:44.637103    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-fsbd6\""
Jul 02 12:39:44.644726 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:44.644690    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-dz8tm"
Jul 02 12:39:44.689174 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:44.689088    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-656bc596cc-m2fg7"]
Jul 02 12:39:44.694505 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:39:44.694472    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02f1bfff_40d6_4805_8579_9e3f5a390e25.slice/crio-bd52b48f6707bd9bb0b578e45d861fff7d670892ebdfc230741ce24877bf3d3e WatchSource:0}: Error finding container bd52b48f6707bd9bb0b578e45d861fff7d670892ebdfc230741ce24877bf3d3e: Status 404 returned error can't find the container with id bd52b48f6707bd9bb0b578e45d861fff7d670892ebdfc230741ce24877bf3d3e
Jul 02 12:39:44.775161 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:44.771938    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-54cb84584-dz8tm"]
Jul 02 12:39:45.256504 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:45.256466    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-dz8tm" event={"ID":"ac58c65c-c830-4ba0-aef5-c5745ed26d93","Type":"ContainerStarted","Data":"0894aa157998bd574627bbc8e570bebca5c89450c6fcf6efede2a8403135bb38"}
Jul 02 12:39:45.257945 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:45.257902    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-656bc596cc-m2fg7" event={"ID":"02f1bfff-40d6-4805-8579-9e3f5a390e25","Type":"ContainerStarted","Data":"36c428ec8edd8b4c723278069e216787501eb5464afdd4e5d1e770f1ae8f3117"}
Jul 02 12:39:45.258099 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:45.257966    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-656bc596cc-m2fg7" event={"ID":"02f1bfff-40d6-4805-8579-9e3f5a390e25","Type":"ContainerStarted","Data":"bd52b48f6707bd9bb0b578e45d861fff7d670892ebdfc230741ce24877bf3d3e"}
Jul 02 12:39:45.278939 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:45.278878    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-656bc596cc-m2fg7" podStartSLOduration=33.278865872 podStartE2EDuration="33.278865872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:39:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:39:45.278676103 +0000 UTC m=+73.886139137" watchObservedRunningTime="2026-07-02 12:39:45.278865872 +0000 UTC m=+73.886328894"
Jul 02 12:39:45.551159 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:45.551083    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-656bc596cc-m2fg7"
Jul 02 12:39:45.555224 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:45.555200    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-656bc596cc-m2fg7"
Jul 02 12:39:46.261208 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:46.261179    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-656bc596cc-m2fg7"
Jul 02 12:39:46.262653 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:46.262629    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-656bc596cc-m2fg7"
Jul 02 12:39:47.265903 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:47.265861    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-dz8tm" event={"ID":"ac58c65c-c830-4ba0-aef5-c5745ed26d93","Type":"ContainerStarted","Data":"9040890aa31237061f885d7bb4e7b3e6cc47ac6f411292662a24eb1cd67ff7b8"}
Jul 02 12:39:47.329864 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:47.329807    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-dz8tm" podStartSLOduration=33.421177784 podStartE2EDuration="35.329785249s" totalImagesPullingTime="1.908607465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:39:47.328946747 +0000 UTC m=+75.936409761" watchObservedRunningTime="2026-07-02 12:39:47.329785249 +0000 UTC m=+75.937248275"
Jul 02 12:39:50.802783 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:50.802734    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/880228b8-bf07-4d91-802b-f05146179c2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xn552\" (UID: \"880228b8-bf07-4d91-802b-f05146179c2a\") " pod="openshift-insights/insights-runtime-extractor-xn552"
Jul 02 12:39:50.805033 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:50.805005    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/880228b8-bf07-4d91-802b-f05146179c2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xn552\" (UID: \"880228b8-bf07-4d91-802b-f05146179c2a\") " pod="openshift-insights/insights-runtime-extractor-xn552"
Jul 02 12:39:51.057206 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:51.057123    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qjvpt\""
Jul 02 12:39:51.065421 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:51.065403    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xn552"
Jul 02 12:39:51.192523 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:51.192456    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xn552"]
Jul 02 12:39:51.195964 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:39:51.195936    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod880228b8_bf07_4d91_802b_f05146179c2a.slice/crio-f2bc53e29d0dc80fededa6375055719740c10bff17dfa84e87cf558c169fdfe0 WatchSource:0}: Error finding container f2bc53e29d0dc80fededa6375055719740c10bff17dfa84e87cf558c169fdfe0: Status 404 returned error can't find the container with id f2bc53e29d0dc80fededa6375055719740c10bff17dfa84e87cf558c169fdfe0
Jul 02 12:39:51.279197 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:51.279160    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xn552" event={"ID":"880228b8-bf07-4d91-802b-f05146179c2a","Type":"ContainerStarted","Data":"f2bc53e29d0dc80fededa6375055719740c10bff17dfa84e87cf558c169fdfe0"}
Jul 02 12:39:52.252795 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:52.252760    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-s82bz"
Jul 02 12:39:54.963240 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:54.963202    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-56499946bc-9fqqb"]
Jul 02 12:39:54.966145 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:54.966127    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-56499946bc-9fqqb"
Jul 02 12:39:54.982023 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:54.981994    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"network-check-source-56499946bc-9fqqb\" is forbidden: User \"system:node:ip-10-0-143-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-network-diagnostics\": no relationship found between node 'ip-10-0-143-88.ec2.internal' and this object" podUID="afc2b803-2e88-4ea8-ac6c-05ac215e33e9" pod="openshift-network-diagnostics/network-check-source-56499946bc-9fqqb"
Jul 02 12:39:54.982023 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:54.982008    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"network-diagnostics-dockercfg-czd5z\" is forbidden: User \"system:node:ip-10-0-143-88.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-network-diagnostics\": no relationship found between node 'ip-10-0-143-88.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-czd5z\"" type="*v1.Secret"
Jul 02 12:39:54.999333 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:54.999308    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-nfq4q"]
Jul 02 12:39:55.003701 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.003680    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-nfq4q"
Jul 02 12:39:55.003817 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.003797    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-56499946bc-9fqqb"]
Jul 02 12:39:55.007584 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.007560    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 02 12:39:55.008283 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.008201    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 02 12:39:55.008553 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.008535    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-5dx9x\""
Jul 02 12:39:55.034471 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.034446    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-nfq4q"]
Jul 02 12:39:55.038824 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.038791    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz8pj\" (UniqueName: \"kubernetes.io/projected/afc2b803-2e88-4ea8-ac6c-05ac215e33e9-kube-api-access-nz8pj\") pod \"network-check-source-56499946bc-9fqqb\" (UID: \"afc2b803-2e88-4ea8-ac6c-05ac215e33e9\") " pod="openshift-network-diagnostics/network-check-source-56499946bc-9fqqb"
Jul 02 12:39:55.056850 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.056657    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-lrwvv"]
Jul 02 12:39:55.060068 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.060050    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-f8c8cfdd6-t28hq"]
Jul 02 12:39:55.060177 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.060161    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-lrwvv"
Jul 02 12:39:55.062703 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.062688    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5499b6fb8f-2c2lq"]
Jul 02 12:39:55.062837 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.062821    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f8c8cfdd6-t28hq"
Jul 02 12:39:55.063365 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.063281    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-p4vdp\""
Jul 02 12:39:55.063472 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.063460    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 02 12:39:55.065978 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.065903    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5499b6fb8f-2c2lq"
Jul 02 12:39:55.067650 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.067637    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 02 12:39:55.068119 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.068099    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-zbh6r\""
Jul 02 12:39:55.068202 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.068103    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 02 12:39:55.068707 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.068688    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 02 12:39:55.069137 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.069120    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 02 12:39:55.069205 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.069187    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 02 12:39:55.069372 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.069353    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 02 12:39:55.069500 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.069484    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 02 12:39:55.078728 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.078708    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-p9dkk\""
Jul 02 12:39:55.085246 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.085222    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-lrwvv"]
Jul 02 12:39:55.089907 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.089888    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5499b6fb8f-2c2lq"]
Jul 02 12:39:55.117770 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.117750    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f8c8cfdd6-t28hq"]
Jul 02 12:39:55.140513 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.140219    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e1593a5a-d0b6-4d6d-8a47-9a653e2778db-oauth-serving-cert\") pod \"console-f8c8cfdd6-t28hq\" (UID: \"e1593a5a-d0b6-4d6d-8a47-9a653e2778db\") " pod="openshift-console/console-f8c8cfdd6-t28hq"
Jul 02 12:39:55.140513 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.140297    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz8pj\" (UniqueName: \"kubernetes.io/projected/afc2b803-2e88-4ea8-ac6c-05ac215e33e9-kube-api-access-nz8pj\") pod \"network-check-source-56499946bc-9fqqb\" (UID: \"afc2b803-2e88-4ea8-ac6c-05ac215e33e9\") " pod="openshift-network-diagnostics/network-check-source-56499946bc-9fqqb"
Jul 02 12:39:55.140513 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.140339    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lbhb\" (UniqueName: \"kubernetes.io/projected/5afef10c-ce2e-4a9d-8e49-e385b6ed71a2-kube-api-access-8lbhb\") pod \"downloads-5499b6fb8f-2c2lq\" (UID: \"5afef10c-ce2e-4a9d-8e49-e385b6ed71a2\") " pod="openshift-console/downloads-5499b6fb8f-2c2lq"
Jul 02 12:39:55.140513 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.140384    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ffc27c9b-735f-444c-8b69-433ef52cf365-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-nfq4q\" (UID: \"ffc27c9b-735f-444c-8b69-433ef52cf365\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-nfq4q"
Jul 02 12:39:55.140513 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.140432    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e1593a5a-d0b6-4d6d-8a47-9a653e2778db-service-ca\") pod \"console-f8c8cfdd6-t28hq\" (UID: \"e1593a5a-d0b6-4d6d-8a47-9a653e2778db\") " pod="openshift-console/console-f8c8cfdd6-t28hq"
Jul 02 12:39:55.140513 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.140458    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j9xj\" (UniqueName: \"kubernetes.io/projected/e1593a5a-d0b6-4d6d-8a47-9a653e2778db-kube-api-access-8j9xj\") pod \"console-f8c8cfdd6-t28hq\" (UID: \"e1593a5a-d0b6-4d6d-8a47-9a653e2778db\") " pod="openshift-console/console-f8c8cfdd6-t28hq"
Jul 02 12:39:55.140778 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.140513    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e1593a5a-d0b6-4d6d-8a47-9a653e2778db-console-config\") pod \"console-f8c8cfdd6-t28hq\" (UID: \"e1593a5a-d0b6-4d6d-8a47-9a653e2778db\") " pod="openshift-console/console-f8c8cfdd6-t28hq"
Jul 02 12:39:55.140778 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.140547    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e1593a5a-d0b6-4d6d-8a47-9a653e2778db-console-oauth-config\") pod \"console-f8c8cfdd6-t28hq\" (UID: \"e1593a5a-d0b6-4d6d-8a47-9a653e2778db\") " pod="openshift-console/console-f8c8cfdd6-t28hq"
Jul 02 12:39:55.140778 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.140610    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f185005f-3257-447b-9488-d15974db3138-tls-certificates\") pod \"prometheus-operator-admission-webhook-7ddc8784d4-lrwvv\" (UID: \"f185005f-3257-447b-9488-d15974db3138\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-lrwvv"
Jul 02 12:39:55.140778 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.140633    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e1593a5a-d0b6-4d6d-8a47-9a653e2778db-console-serving-cert\") pod \"console-f8c8cfdd6-t28hq\" (UID: \"e1593a5a-d0b6-4d6d-8a47-9a653e2778db\") " pod="openshift-console/console-f8c8cfdd6-t28hq"
Jul 02 12:39:55.140778 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.140650    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ffc27c9b-735f-444c-8b69-433ef52cf365-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-nfq4q\" (UID: \"ffc27c9b-735f-444c-8b69-433ef52cf365\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-nfq4q"
Jul 02 12:39:55.171421 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.171369    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz8pj\" (UniqueName: \"kubernetes.io/projected/afc2b803-2e88-4ea8-ac6c-05ac215e33e9-kube-api-access-nz8pj\") pod \"network-check-source-56499946bc-9fqqb\" (UID: \"afc2b803-2e88-4ea8-ac6c-05ac215e33e9\") " pod="openshift-network-diagnostics/network-check-source-56499946bc-9fqqb"
Jul 02 12:39:55.241521 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.241499    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e1593a5a-d0b6-4d6d-8a47-9a653e2778db-service-ca\") pod \"console-f8c8cfdd6-t28hq\" (UID: \"e1593a5a-d0b6-4d6d-8a47-9a653e2778db\") " pod="openshift-console/console-f8c8cfdd6-t28hq"
Jul 02 12:39:55.241677 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.241533    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j9xj\" (UniqueName: \"kubernetes.io/projected/e1593a5a-d0b6-4d6d-8a47-9a653e2778db-kube-api-access-8j9xj\") pod \"console-f8c8cfdd6-t28hq\" (UID: \"e1593a5a-d0b6-4d6d-8a47-9a653e2778db\") " pod="openshift-console/console-f8c8cfdd6-t28hq"
Jul 02 12:39:55.241677 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.241554    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e1593a5a-d0b6-4d6d-8a47-9a653e2778db-console-config\") pod \"console-f8c8cfdd6-t28hq\" (UID: \"e1593a5a-d0b6-4d6d-8a47-9a653e2778db\") " pod="openshift-console/console-f8c8cfdd6-t28hq"
Jul 02 12:39:55.241677 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.241650    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e1593a5a-d0b6-4d6d-8a47-9a653e2778db-console-oauth-config\") pod \"console-f8c8cfdd6-t28hq\" (UID: \"e1593a5a-d0b6-4d6d-8a47-9a653e2778db\") " pod="openshift-console/console-f8c8cfdd6-t28hq"
Jul 02 12:39:55.241851 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.241703    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f185005f-3257-447b-9488-d15974db3138-tls-certificates\") pod \"prometheus-operator-admission-webhook-7ddc8784d4-lrwvv\" (UID: \"f185005f-3257-447b-9488-d15974db3138\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-lrwvv"
Jul 02 12:39:55.241851 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.241752    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e1593a5a-d0b6-4d6d-8a47-9a653e2778db-console-serving-cert\") pod \"console-f8c8cfdd6-t28hq\" (UID: \"e1593a5a-d0b6-4d6d-8a47-9a653e2778db\") " pod="openshift-console/console-f8c8cfdd6-t28hq"
Jul 02 12:39:55.241851 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.241770    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ffc27c9b-735f-444c-8b69-433ef52cf365-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-nfq4q\" (UID: \"ffc27c9b-735f-444c-8b69-433ef52cf365\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-nfq4q"
Jul 02 12:39:55.241851 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.241817    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e1593a5a-d0b6-4d6d-8a47-9a653e2778db-oauth-serving-cert\") pod \"console-f8c8cfdd6-t28hq\" (UID: \"e1593a5a-d0b6-4d6d-8a47-9a653e2778db\") " pod="openshift-console/console-f8c8cfdd6-t28hq"
Jul 02 12:39:55.242076 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.241852    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lbhb\" (UniqueName: \"kubernetes.io/projected/5afef10c-ce2e-4a9d-8e49-e385b6ed71a2-kube-api-access-8lbhb\") pod \"downloads-5499b6fb8f-2c2lq\" (UID: \"5afef10c-ce2e-4a9d-8e49-e385b6ed71a2\") " pod="openshift-console/downloads-5499b6fb8f-2c2lq"
Jul 02 12:39:55.242076 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.241905    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ffc27c9b-735f-444c-8b69-433ef52cf365-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-nfq4q\" (UID: \"ffc27c9b-735f-444c-8b69-433ef52cf365\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-nfq4q"
Jul 02 12:39:55.242629 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.242398    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e1593a5a-d0b6-4d6d-8a47-9a653e2778db-service-ca\") pod \"console-f8c8cfdd6-t28hq\" (UID: \"e1593a5a-d0b6-4d6d-8a47-9a653e2778db\") " pod="openshift-console/console-f8c8cfdd6-t28hq"
Jul 02 12:39:55.243416 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.243261    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e1593a5a-d0b6-4d6d-8a47-9a653e2778db-console-config\") pod \"console-f8c8cfdd6-t28hq\" (UID: \"e1593a5a-d0b6-4d6d-8a47-9a653e2778db\") " pod="openshift-console/console-f8c8cfdd6-t28hq"
Jul 02 12:39:55.244432 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.243563    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ffc27c9b-735f-444c-8b69-433ef52cf365-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-nfq4q\" (UID: \"ffc27c9b-735f-444c-8b69-433ef52cf365\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-nfq4q"
Jul 02 12:39:55.244432 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.244282    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e1593a5a-d0b6-4d6d-8a47-9a653e2778db-oauth-serving-cert\") pod \"console-f8c8cfdd6-t28hq\" (UID: \"e1593a5a-d0b6-4d6d-8a47-9a653e2778db\") " pod="openshift-console/console-f8c8cfdd6-t28hq"
Jul 02 12:39:55.244890 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.244849    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ffc27c9b-735f-444c-8b69-433ef52cf365-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-nfq4q\" (UID: \"ffc27c9b-735f-444c-8b69-433ef52cf365\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-nfq4q"
Jul 02 12:39:55.244890 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.244853    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e1593a5a-d0b6-4d6d-8a47-9a653e2778db-console-serving-cert\") pod \"console-f8c8cfdd6-t28hq\" (UID: \"e1593a5a-d0b6-4d6d-8a47-9a653e2778db\") " pod="openshift-console/console-f8c8cfdd6-t28hq"
Jul 02 12:39:55.245100 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.244964    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e1593a5a-d0b6-4d6d-8a47-9a653e2778db-console-oauth-config\") pod \"console-f8c8cfdd6-t28hq\" (UID: \"e1593a5a-d0b6-4d6d-8a47-9a653e2778db\") " pod="openshift-console/console-f8c8cfdd6-t28hq"
Jul 02 12:39:55.245100 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.245088    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f185005f-3257-447b-9488-d15974db3138-tls-certificates\") pod \"prometheus-operator-admission-webhook-7ddc8784d4-lrwvv\" (UID: \"f185005f-3257-447b-9488-d15974db3138\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-lrwvv"
Jul 02 12:39:55.251807 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.251788    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j9xj\" (UniqueName: \"kubernetes.io/projected/e1593a5a-d0b6-4d6d-8a47-9a653e2778db-kube-api-access-8j9xj\") pod \"console-f8c8cfdd6-t28hq\" (UID: \"e1593a5a-d0b6-4d6d-8a47-9a653e2778db\") " pod="openshift-console/console-f8c8cfdd6-t28hq"
Jul 02 12:39:55.258835 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.258815    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lbhb\" (UniqueName: \"kubernetes.io/projected/5afef10c-ce2e-4a9d-8e49-e385b6ed71a2-kube-api-access-8lbhb\") pod \"downloads-5499b6fb8f-2c2lq\" (UID: \"5afef10c-ce2e-4a9d-8e49-e385b6ed71a2\") " pod="openshift-console/downloads-5499b6fb8f-2c2lq"
Jul 02 12:39:55.293285 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.293256    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xn552" event={"ID":"880228b8-bf07-4d91-802b-f05146179c2a","Type":"ContainerStarted","Data":"c1ad391a0f454a9f9ab7b40f1dccc0048ea59f67d7f340b232b98a57d4118689"}
Jul 02 12:39:55.337132 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.337113    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-nfq4q"
Jul 02 12:39:55.371482 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.371456    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-lrwvv"
Jul 02 12:39:55.379028 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.379001    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f8c8cfdd6-t28hq"
Jul 02 12:39:55.384247 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.384129    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5499b6fb8f-2c2lq"
Jul 02 12:39:55.510986 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.510957    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-nfq4q"]
Jul 02 12:39:55.514567 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:39:55.514528    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffc27c9b_735f_444c_8b69_433ef52cf365.slice/crio-c22b7351fc53fc4871e51e58e7f232dad10a96d97b0c2e2f0963cd24930847fb WatchSource:0}: Error finding container c22b7351fc53fc4871e51e58e7f232dad10a96d97b0c2e2f0963cd24930847fb: Status 404 returned error can't find the container with id c22b7351fc53fc4871e51e58e7f232dad10a96d97b0c2e2f0963cd24930847fb
Jul 02 12:39:55.540735 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.540658    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f8c8cfdd6-t28hq"]
Jul 02 12:39:55.542795 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:39:55.542753    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1593a5a_d0b6_4d6d_8a47_9a653e2778db.slice/crio-da17c96116d8c09cecd042754efdcfc365d8fa48d7b63b38a73a101bd96c31d7 WatchSource:0}: Error finding container da17c96116d8c09cecd042754efdcfc365d8fa48d7b63b38a73a101bd96c31d7: Status 404 returned error can't find the container with id da17c96116d8c09cecd042754efdcfc365d8fa48d7b63b38a73a101bd96c31d7
Jul 02 12:39:55.561377 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.561359    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-lrwvv"]
Jul 02 12:39:55.563377 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:39:55.563355    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf185005f_3257_447b_9488_d15974db3138.slice/crio-d568eca329b2b50fc9e2ee4402627aad6aa3f6d7781c8d86b66c78b2177c90ac WatchSource:0}: Error finding container d568eca329b2b50fc9e2ee4402627aad6aa3f6d7781c8d86b66c78b2177c90ac: Status 404 returned error can't find the container with id d568eca329b2b50fc9e2ee4402627aad6aa3f6d7781c8d86b66c78b2177c90ac
Jul 02 12:39:55.580687 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:55.580667    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5499b6fb8f-2c2lq"]
Jul 02 12:39:55.583793 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:39:55.583771    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5afef10c_ce2e_4a9d_8e49_e385b6ed71a2.slice/crio-311f37d07fddce2dd29d6a2ef732be712fe8b71f4f25cc5aec09a8b9d6d6fafe WatchSource:0}: Error finding container 311f37d07fddce2dd29d6a2ef732be712fe8b71f4f25cc5aec09a8b9d6d6fafe: Status 404 returned error can't find the container with id 311f37d07fddce2dd29d6a2ef732be712fe8b71f4f25cc5aec09a8b9d6d6fafe
Jul 02 12:39:56.276497 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:56.276294    2576 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-network-diagnostics/network-check-source-56499946bc-9fqqb" secret="" err="failed to sync secret cache: timed out waiting for the condition"
Jul 02 12:39:56.276497 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:56.276476    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-56499946bc-9fqqb"
Jul 02 12:39:56.299899 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:56.299823    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5499b6fb8f-2c2lq" event={"ID":"5afef10c-ce2e-4a9d-8e49-e385b6ed71a2","Type":"ContainerStarted","Data":"311f37d07fddce2dd29d6a2ef732be712fe8b71f4f25cc5aec09a8b9d6d6fafe"}
Jul 02 12:39:56.301562 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:56.301495    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-nfq4q" event={"ID":"ffc27c9b-735f-444c-8b69-433ef52cf365","Type":"ContainerStarted","Data":"c22b7351fc53fc4871e51e58e7f232dad10a96d97b0c2e2f0963cd24930847fb"}
Jul 02 12:39:56.304071 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:56.304003    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f8c8cfdd6-t28hq" event={"ID":"e1593a5a-d0b6-4d6d-8a47-9a653e2778db","Type":"ContainerStarted","Data":"da17c96116d8c09cecd042754efdcfc365d8fa48d7b63b38a73a101bd96c31d7"}
Jul 02 12:39:56.307111 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:56.307043    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-lrwvv" event={"ID":"f185005f-3257-447b-9488-d15974db3138","Type":"ContainerStarted","Data":"d568eca329b2b50fc9e2ee4402627aad6aa3f6d7781c8d86b66c78b2177c90ac"}
Jul 02 12:39:56.310639 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:56.310608    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xn552" event={"ID":"880228b8-bf07-4d91-802b-f05146179c2a","Type":"ContainerStarted","Data":"9f126b0a1714f60dd30cfab0ed7133b340396ebdcd4ba1f4b5fb40848b4d7d1d"}
Jul 02 12:39:56.334156 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:56.333724    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-czd5z\""
Jul 02 12:39:56.449245 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:56.449159    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-56499946bc-9fqqb"]
Jul 02 12:39:56.707665 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:39:56.707602    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafc2b803_2e88_4ea8_ac6c_05ac215e33e9.slice/crio-79c1a6105762956aff23f53185ce89374a77ae502d723efa2034cfc648d5aacd WatchSource:0}: Error finding container 79c1a6105762956aff23f53185ce89374a77ae502d723efa2034cfc648d5aacd: Status 404 returned error can't find the container with id 79c1a6105762956aff23f53185ce89374a77ae502d723efa2034cfc648d5aacd
Jul 02 12:39:57.317003 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:57.316854    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-lrwvv" event={"ID":"f185005f-3257-447b-9488-d15974db3138","Type":"ContainerStarted","Data":"c3b28508be7cdfb7e3930606eeb61ebaae2d4d79f9dad0bdd9197ad1a9700159"}
Jul 02 12:39:57.317845 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:57.317790    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-lrwvv"
Jul 02 12:39:57.319743 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:57.319675    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-56499946bc-9fqqb" event={"ID":"afc2b803-2e88-4ea8-ac6c-05ac215e33e9","Type":"ContainerStarted","Data":"ddb9db39bcfd46922948ea9ca01826557c44648789c4e8ec761a455f3e625f0a"}
Jul 02 12:39:57.319743 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:57.319721    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-56499946bc-9fqqb" event={"ID":"afc2b803-2e88-4ea8-ac6c-05ac215e33e9","Type":"ContainerStarted","Data":"79c1a6105762956aff23f53185ce89374a77ae502d723efa2034cfc648d5aacd"}
Jul 02 12:39:57.322520 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:57.322479    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-nfq4q" event={"ID":"ffc27c9b-735f-444c-8b69-433ef52cf365","Type":"ContainerStarted","Data":"dd59ed8c4b69bc13517dacd72b217d619f9c9369085c409818ca4024bbbe02c7"}
Jul 02 12:39:57.324338 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:57.324306    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-lrwvv"
Jul 02 12:39:57.334202 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:57.334158    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-lrwvv" podStartSLOduration=0.823563211 podStartE2EDuration="2.334140355s" totalImagesPullingTime="1.510577144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:39:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:39:57.33334316 +0000 UTC m=+85.940806207" watchObservedRunningTime="2026-07-02 12:39:57.334140355 +0000 UTC m=+85.941603381"
Jul 02 12:39:57.350970 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:57.350587    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-56499946bc-9fqqb" podStartSLOduration=3.350568855 podStartE2EDuration="3.350568855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:39:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:39:57.349197043 +0000 UTC m=+85.956660068" watchObservedRunningTime="2026-07-02 12:39:57.350568855 +0000 UTC m=+85.958031879"
Jul 02 12:39:57.393197 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:57.392553    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-nfq4q" podStartSLOduration=1.836236069 podStartE2EDuration="3.392537486s" totalImagesPullingTime="1.556301417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:39:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:39:57.365945524 +0000 UTC m=+85.973408547" watchObservedRunningTime="2026-07-02 12:39:57.392537486 +0000 UTC m=+86.000000513"
Jul 02 12:39:57.615810 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:57.614712    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7b7d896497-vzhw4"]
Jul 02 12:39:57.619385 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:57.619361    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7b7d896497-vzhw4"
Jul 02 12:39:57.623208 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:57.622535    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 02 12:39:57.623208 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:57.622781    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 02 12:39:57.623208 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:57.623055    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 02 12:39:57.624729 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:57.624483    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-rlscn\""
Jul 02 12:39:57.634604 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:57.634541    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7b7d896497-vzhw4"]
Jul 02 12:39:57.766450 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:57.766403    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d253c0c2-9179-4e35-9522-1c512e869900-metrics-client-ca\") pod \"prometheus-operator-7b7d896497-vzhw4\" (UID: \"d253c0c2-9179-4e35-9522-1c512e869900\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-vzhw4"
Jul 02 12:39:57.766642 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:57.766526    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp2fz\" (UniqueName: \"kubernetes.io/projected/d253c0c2-9179-4e35-9522-1c512e869900-kube-api-access-wp2fz\") pod \"prometheus-operator-7b7d896497-vzhw4\" (UID: \"d253c0c2-9179-4e35-9522-1c512e869900\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-vzhw4"
Jul 02 12:39:57.771595 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:57.767090    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d253c0c2-9179-4e35-9522-1c512e869900-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7b7d896497-vzhw4\" (UID: \"d253c0c2-9179-4e35-9522-1c512e869900\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-vzhw4"
Jul 02 12:39:57.771595 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:57.767141    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d253c0c2-9179-4e35-9522-1c512e869900-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-vzhw4\" (UID: \"d253c0c2-9179-4e35-9522-1c512e869900\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-vzhw4"
Jul 02 12:39:57.868703 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:57.868617    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp2fz\" (UniqueName: \"kubernetes.io/projected/d253c0c2-9179-4e35-9522-1c512e869900-kube-api-access-wp2fz\") pod \"prometheus-operator-7b7d896497-vzhw4\" (UID: \"d253c0c2-9179-4e35-9522-1c512e869900\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-vzhw4"
Jul 02 12:39:57.868867 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:57.868733    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d253c0c2-9179-4e35-9522-1c512e869900-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7b7d896497-vzhw4\" (UID: \"d253c0c2-9179-4e35-9522-1c512e869900\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-vzhw4"
Jul 02 12:39:57.868867 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:57.868777    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d253c0c2-9179-4e35-9522-1c512e869900-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-vzhw4\" (UID: \"d253c0c2-9179-4e35-9522-1c512e869900\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-vzhw4"
Jul 02 12:39:57.868867 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:57.868840    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d253c0c2-9179-4e35-9522-1c512e869900-metrics-client-ca\") pod \"prometheus-operator-7b7d896497-vzhw4\" (UID: \"d253c0c2-9179-4e35-9522-1c512e869900\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-vzhw4"
Jul 02 12:39:57.869489 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:57.869244    2576 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 02 12:39:57.869489 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:39:57.869299    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d253c0c2-9179-4e35-9522-1c512e869900-prometheus-operator-tls podName:d253c0c2-9179-4e35-9522-1c512e869900 nodeName:}" failed. No retries permitted until 2026-07-02 12:39:58.369284454 +0000 UTC m=+86.976747456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/d253c0c2-9179-4e35-9522-1c512e869900-prometheus-operator-tls") pod "prometheus-operator-7b7d896497-vzhw4" (UID: "d253c0c2-9179-4e35-9522-1c512e869900") : secret "prometheus-operator-tls" not found
Jul 02 12:39:57.869670 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:57.869612    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d253c0c2-9179-4e35-9522-1c512e869900-metrics-client-ca\") pod \"prometheus-operator-7b7d896497-vzhw4\" (UID: \"d253c0c2-9179-4e35-9522-1c512e869900\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-vzhw4"
Jul 02 12:39:57.872107 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:57.872060    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d253c0c2-9179-4e35-9522-1c512e869900-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7b7d896497-vzhw4\" (UID: \"d253c0c2-9179-4e35-9522-1c512e869900\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-vzhw4"
Jul 02 12:39:57.881685 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:57.881612    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp2fz\" (UniqueName: \"kubernetes.io/projected/d253c0c2-9179-4e35-9522-1c512e869900-kube-api-access-wp2fz\") pod \"prometheus-operator-7b7d896497-vzhw4\" (UID: \"d253c0c2-9179-4e35-9522-1c512e869900\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-vzhw4"
Jul 02 12:39:58.374969 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:58.374888    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d253c0c2-9179-4e35-9522-1c512e869900-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-vzhw4\" (UID: \"d253c0c2-9179-4e35-9522-1c512e869900\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-vzhw4"
Jul 02 12:39:58.381952 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:58.381900    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d253c0c2-9179-4e35-9522-1c512e869900-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-vzhw4\" (UID: \"d253c0c2-9179-4e35-9522-1c512e869900\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-vzhw4"
Jul 02 12:39:58.534266 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:58.534028    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7b7d896497-vzhw4"
Jul 02 12:39:59.390802 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:59.390754    2576 patch_prober.go:28] interesting pod/image-registry-847db68bb4-fgghh container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 02 12:39:59.391276 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:39:59.390829    2576 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-847db68bb4-fgghh" podUID="69ae490b-bd41-4ab2-b05c-3393fea68351" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:40:00.823946 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:00.823848    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7b7d896497-vzhw4"]
Jul 02 12:40:00.828769 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:40:00.828737    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd253c0c2_9179_4e35_9522_1c512e869900.slice/crio-047d26b81460c988ff4103d77cf6af458733ffb736cf8a3beabcaf678c926323 WatchSource:0}: Error finding container 047d26b81460c988ff4103d77cf6af458733ffb736cf8a3beabcaf678c926323: Status 404 returned error can't find the container with id 047d26b81460c988ff4103d77cf6af458733ffb736cf8a3beabcaf678c926323
Jul 02 12:40:01.245593 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:01.245559    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-847db68bb4-fgghh"
Jul 02 12:40:01.342278 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:01.342241    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f8c8cfdd6-t28hq" event={"ID":"e1593a5a-d0b6-4d6d-8a47-9a653e2778db","Type":"ContainerStarted","Data":"ff10dd224ed42c25597ca3a1dcd9adeb74a59f10fb35e241f99becd368abd202"}
Jul 02 12:40:01.343743 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:01.343706    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7b7d896497-vzhw4" event={"ID":"d253c0c2-9179-4e35-9522-1c512e869900","Type":"ContainerStarted","Data":"047d26b81460c988ff4103d77cf6af458733ffb736cf8a3beabcaf678c926323"}
Jul 02 12:40:01.346178 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:01.346145    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xn552" event={"ID":"880228b8-bf07-4d91-802b-f05146179c2a","Type":"ContainerStarted","Data":"9b6723d6bf69e106374b08e01ecf787b31abbd99b854d2eb55e12460aa7b7ea0"}
Jul 02 12:40:01.388838 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:01.388796    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-f8c8cfdd6-t28hq" podStartSLOduration=1.247238063 podStartE2EDuration="6.388778754s" totalImagesPullingTime="5.141540691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:39:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:40:01.363568926 +0000 UTC m=+89.971031963" watchObservedRunningTime="2026-07-02 12:40:01.388778754 +0000 UTC m=+89.996241780"
Jul 02 12:40:01.389082 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:01.389048    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xn552" podStartSLOduration=34.02954515 podStartE2EDuration="43.389040058s" totalImagesPullingTime="9.359494908s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 12:39:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:40:01.387848042 +0000 UTC m=+89.995311088" watchObservedRunningTime="2026-07-02 12:40:01.389040058 +0000 UTC m=+89.996503082"
Jul 02 12:40:02.354474 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:02.354361    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7b7d896497-vzhw4" event={"ID":"d253c0c2-9179-4e35-9522-1c512e869900","Type":"ContainerStarted","Data":"2e879f179dacc2c122c28fa7026a7fa41120507825a141802f4aee33e4dd0c27"}
Jul 02 12:40:03.247579 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:03.247539    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-647779b8f4-t5vk7"]
Jul 02 12:40:03.250962 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:03.250937    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-647779b8f4-t5vk7"
Jul 02 12:40:03.260688 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:03.260664    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 02 12:40:03.270347 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:03.270324    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-647779b8f4-t5vk7"]
Jul 02 12:40:03.361256 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:03.361215    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7b7d896497-vzhw4" event={"ID":"d253c0c2-9179-4e35-9522-1c512e869900","Type":"ContainerStarted","Data":"715f7d945f7f38e1ab677438e4ac51d170e33ffa4e0168fc1c447fa8a91ccf86"}
Jul 02 12:40:03.380760 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:03.380706    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7b7d896497-vzhw4" podStartSLOduration=4.972994752 podStartE2EDuration="6.380690395s" totalImagesPullingTime="1.407695643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:39:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:40:03.379271397 +0000 UTC m=+91.986734444" watchObservedRunningTime="2026-07-02 12:40:03.380690395 +0000 UTC m=+91.988153455"
Jul 02 12:40:03.422706 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:03.422671    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/60d75a17-fa2d-403c-bc77-959ae8486a21-service-ca\") pod \"console-647779b8f4-t5vk7\" (UID: \"60d75a17-fa2d-403c-bc77-959ae8486a21\") " pod="openshift-console/console-647779b8f4-t5vk7"
Jul 02 12:40:03.422885 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:03.422735    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-848j6\" (UniqueName: \"kubernetes.io/projected/60d75a17-fa2d-403c-bc77-959ae8486a21-kube-api-access-848j6\") pod \"console-647779b8f4-t5vk7\" (UID: \"60d75a17-fa2d-403c-bc77-959ae8486a21\") " pod="openshift-console/console-647779b8f4-t5vk7"
Jul 02 12:40:03.422885 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:03.422821    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60d75a17-fa2d-403c-bc77-959ae8486a21-trusted-ca-bundle\") pod \"console-647779b8f4-t5vk7\" (UID: \"60d75a17-fa2d-403c-bc77-959ae8486a21\") " pod="openshift-console/console-647779b8f4-t5vk7"
Jul 02 12:40:03.423036 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:03.422903    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/60d75a17-fa2d-403c-bc77-959ae8486a21-console-oauth-config\") pod \"console-647779b8f4-t5vk7\" (UID: \"60d75a17-fa2d-403c-bc77-959ae8486a21\") " pod="openshift-console/console-647779b8f4-t5vk7"
Jul 02 12:40:03.423036 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:03.422954    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/60d75a17-fa2d-403c-bc77-959ae8486a21-console-config\") pod \"console-647779b8f4-t5vk7\" (UID: \"60d75a17-fa2d-403c-bc77-959ae8486a21\") " pod="openshift-console/console-647779b8f4-t5vk7"
Jul 02 12:40:03.423036 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:03.422985    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/60d75a17-fa2d-403c-bc77-959ae8486a21-oauth-serving-cert\") pod \"console-647779b8f4-t5vk7\" (UID: \"60d75a17-fa2d-403c-bc77-959ae8486a21\") " pod="openshift-console/console-647779b8f4-t5vk7"
Jul 02 12:40:03.423036 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:03.423015    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/60d75a17-fa2d-403c-bc77-959ae8486a21-console-serving-cert\") pod \"console-647779b8f4-t5vk7\" (UID: \"60d75a17-fa2d-403c-bc77-959ae8486a21\") " pod="openshift-console/console-647779b8f4-t5vk7"
Jul 02 12:40:03.523964 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:03.523860    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-848j6\" (UniqueName: \"kubernetes.io/projected/60d75a17-fa2d-403c-bc77-959ae8486a21-kube-api-access-848j6\") pod \"console-647779b8f4-t5vk7\" (UID: \"60d75a17-fa2d-403c-bc77-959ae8486a21\") " pod="openshift-console/console-647779b8f4-t5vk7"
Jul 02 12:40:03.524153 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:03.523983    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60d75a17-fa2d-403c-bc77-959ae8486a21-trusted-ca-bundle\") pod \"console-647779b8f4-t5vk7\" (UID: \"60d75a17-fa2d-403c-bc77-959ae8486a21\") " pod="openshift-console/console-647779b8f4-t5vk7"
Jul 02 12:40:03.524227 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:03.524199    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/60d75a17-fa2d-403c-bc77-959ae8486a21-console-oauth-config\") pod \"console-647779b8f4-t5vk7\" (UID: \"60d75a17-fa2d-403c-bc77-959ae8486a21\") " pod="openshift-console/console-647779b8f4-t5vk7"
Jul 02 12:40:03.524285 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:03.524251    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/60d75a17-fa2d-403c-bc77-959ae8486a21-console-config\") pod \"console-647779b8f4-t5vk7\" (UID: \"60d75a17-fa2d-403c-bc77-959ae8486a21\") " pod="openshift-console/console-647779b8f4-t5vk7"
Jul 02 12:40:03.524339 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:03.524291    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/60d75a17-fa2d-403c-bc77-959ae8486a21-oauth-serving-cert\") pod \"console-647779b8f4-t5vk7\" (UID: \"60d75a17-fa2d-403c-bc77-959ae8486a21\") " pod="openshift-console/console-647779b8f4-t5vk7"
Jul 02 12:40:03.524339 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:03.524331    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/60d75a17-fa2d-403c-bc77-959ae8486a21-console-serving-cert\") pod \"console-647779b8f4-t5vk7\" (UID: \"60d75a17-fa2d-403c-bc77-959ae8486a21\") " pod="openshift-console/console-647779b8f4-t5vk7"
Jul 02 12:40:03.524839 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:03.524532    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/60d75a17-fa2d-403c-bc77-959ae8486a21-service-ca\") pod \"console-647779b8f4-t5vk7\" (UID: \"60d75a17-fa2d-403c-bc77-959ae8486a21\") " pod="openshift-console/console-647779b8f4-t5vk7"
Jul 02 12:40:03.525203 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:03.525178    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/60d75a17-fa2d-403c-bc77-959ae8486a21-console-config\") pod \"console-647779b8f4-t5vk7\" (UID: \"60d75a17-fa2d-403c-bc77-959ae8486a21\") " pod="openshift-console/console-647779b8f4-t5vk7"
Jul 02 12:40:03.525203 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:03.525199    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/60d75a17-fa2d-403c-bc77-959ae8486a21-oauth-serving-cert\") pod \"console-647779b8f4-t5vk7\" (UID: \"60d75a17-fa2d-403c-bc77-959ae8486a21\") " pod="openshift-console/console-647779b8f4-t5vk7"
Jul 02 12:40:03.525331 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:03.525229    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/60d75a17-fa2d-403c-bc77-959ae8486a21-service-ca\") pod \"console-647779b8f4-t5vk7\" (UID: \"60d75a17-fa2d-403c-bc77-959ae8486a21\") " pod="openshift-console/console-647779b8f4-t5vk7"
Jul 02 12:40:03.525445 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:03.525426    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60d75a17-fa2d-403c-bc77-959ae8486a21-trusted-ca-bundle\") pod \"console-647779b8f4-t5vk7\" (UID: \"60d75a17-fa2d-403c-bc77-959ae8486a21\") " pod="openshift-console/console-647779b8f4-t5vk7"
Jul 02 12:40:03.527102 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:03.527084    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/60d75a17-fa2d-403c-bc77-959ae8486a21-console-oauth-config\") pod \"console-647779b8f4-t5vk7\" (UID: \"60d75a17-fa2d-403c-bc77-959ae8486a21\") " pod="openshift-console/console-647779b8f4-t5vk7"
Jul 02 12:40:03.527295 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:03.527277    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/60d75a17-fa2d-403c-bc77-959ae8486a21-console-serving-cert\") pod \"console-647779b8f4-t5vk7\" (UID: \"60d75a17-fa2d-403c-bc77-959ae8486a21\") " pod="openshift-console/console-647779b8f4-t5vk7"
Jul 02 12:40:03.535183 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:03.535159    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-848j6\" (UniqueName: \"kubernetes.io/projected/60d75a17-fa2d-403c-bc77-959ae8486a21-kube-api-access-848j6\") pod \"console-647779b8f4-t5vk7\" (UID: \"60d75a17-fa2d-403c-bc77-959ae8486a21\") " pod="openshift-console/console-647779b8f4-t5vk7"
Jul 02 12:40:03.561710 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:03.561676    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-647779b8f4-t5vk7"
Jul 02 12:40:03.714254 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:03.714222    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-647779b8f4-t5vk7"]
Jul 02 12:40:03.717997 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:40:03.717962    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60d75a17_fa2d_403c_bc77_959ae8486a21.slice/crio-6f5e860fec7234d8fefb30425be6da026356db244c7e3e18fb65bb2c239f4f1f WatchSource:0}: Error finding container 6f5e860fec7234d8fefb30425be6da026356db244c7e3e18fb65bb2c239f4f1f: Status 404 returned error can't find the container with id 6f5e860fec7234d8fefb30425be6da026356db244c7e3e18fb65bb2c239f4f1f
Jul 02 12:40:04.369209 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:04.369165    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-647779b8f4-t5vk7" event={"ID":"60d75a17-fa2d-403c-bc77-959ae8486a21","Type":"ContainerStarted","Data":"a62ab1ab3921afac6497aa94ad27cdbe134d173ea0e85e6cc8c15a039435836d"}
Jul 02 12:40:04.369666 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:04.369269    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-647779b8f4-t5vk7" event={"ID":"60d75a17-fa2d-403c-bc77-959ae8486a21","Type":"ContainerStarted","Data":"6f5e860fec7234d8fefb30425be6da026356db244c7e3e18fb65bb2c239f4f1f"}
Jul 02 12:40:04.393222 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:04.393163    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-647779b8f4-t5vk7" podStartSLOduration=1.393143756 podStartE2EDuration="1.393143756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:40:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:40:04.392509945 +0000 UTC m=+92.999972969" watchObservedRunningTime="2026-07-02 12:40:04.393143756 +0000 UTC m=+93.000606780"
Jul 02 12:40:05.282364 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.282329    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-5kfkq"]
Jul 02 12:40:05.285714 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.285697    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-845b954455-5kfkq"
Jul 02 12:40:05.288554 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.288536    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 02 12:40:05.288642 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.288627    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-gcsd2\""
Jul 02 12:40:05.288698 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.288678    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 02 12:40:05.289732 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.289717    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 02 12:40:05.306438 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.306409    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-5kfkq"]
Jul 02 12:40:05.379626 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.379597    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f8c8cfdd6-t28hq"
Jul 02 12:40:05.380064 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.379892    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f8c8cfdd6-t28hq"
Jul 02 12:40:05.384950 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.384906    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f8c8cfdd6-t28hq"
Jul 02 12:40:05.405906 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.405875    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-w5qjd"]
Jul 02 12:40:05.409325 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.409302    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w5qjd"
Jul 02 12:40:05.412516 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.412491    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 02 12:40:05.412728 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.412710    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 02 12:40:05.413189 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.413114    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fbqm2\""
Jul 02 12:40:05.413295 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.413236    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 02 12:40:05.443334 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.443296    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b9cd847c-daf5-40f5-9a23-2718fa67f25c-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-5kfkq\" (UID: \"b9cd847c-daf5-40f5-9a23-2718fa67f25c\") " pod="openshift-monitoring/kube-state-metrics-845b954455-5kfkq"
Jul 02 12:40:05.443334 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.443349    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9cd847c-daf5-40f5-9a23-2718fa67f25c-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-5kfkq\" (UID: \"b9cd847c-daf5-40f5-9a23-2718fa67f25c\") " pod="openshift-monitoring/kube-state-metrics-845b954455-5kfkq"
Jul 02 12:40:05.443746 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.443393    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9cd847c-daf5-40f5-9a23-2718fa67f25c-metrics-client-ca\") pod \"kube-state-metrics-845b954455-5kfkq\" (UID: \"b9cd847c-daf5-40f5-9a23-2718fa67f25c\") " pod="openshift-monitoring/kube-state-metrics-845b954455-5kfkq"
Jul 02 12:40:05.443746 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.443467    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qvp7\" (UniqueName: \"kubernetes.io/projected/b9cd847c-daf5-40f5-9a23-2718fa67f25c-kube-api-access-2qvp7\") pod \"kube-state-metrics-845b954455-5kfkq\" (UID: \"b9cd847c-daf5-40f5-9a23-2718fa67f25c\") " pod="openshift-monitoring/kube-state-metrics-845b954455-5kfkq"
Jul 02 12:40:05.443746 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.443499    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b9cd847c-daf5-40f5-9a23-2718fa67f25c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-5kfkq\" (UID: \"b9cd847c-daf5-40f5-9a23-2718fa67f25c\") " pod="openshift-monitoring/kube-state-metrics-845b954455-5kfkq"
Jul 02 12:40:05.443746 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.443528    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b9cd847c-daf5-40f5-9a23-2718fa67f25c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-5kfkq\" (UID: \"b9cd847c-daf5-40f5-9a23-2718fa67f25c\") " pod="openshift-monitoring/kube-state-metrics-845b954455-5kfkq"
Jul 02 12:40:05.544683 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.544589    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eedaddc4-a131-49bd-b077-4a61b9b8f04e-node-exporter-accelerators-collector-config\") pod \"node-exporter-w5qjd\" (UID: \"eedaddc4-a131-49bd-b077-4a61b9b8f04e\") " pod="openshift-monitoring/node-exporter-w5qjd"
Jul 02 12:40:05.544683 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.544645    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9cd847c-daf5-40f5-9a23-2718fa67f25c-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-5kfkq\" (UID: \"b9cd847c-daf5-40f5-9a23-2718fa67f25c\") " pod="openshift-monitoring/kube-state-metrics-845b954455-5kfkq"
Jul 02 12:40:05.544966 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.544705    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s8nt\" (UniqueName: \"kubernetes.io/projected/eedaddc4-a131-49bd-b077-4a61b9b8f04e-kube-api-access-8s8nt\") pod \"node-exporter-w5qjd\" (UID: \"eedaddc4-a131-49bd-b077-4a61b9b8f04e\") " pod="openshift-monitoring/node-exporter-w5qjd"
Jul 02 12:40:05.544966 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.544878    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eedaddc4-a131-49bd-b077-4a61b9b8f04e-node-exporter-wtmp\") pod \"node-exporter-w5qjd\" (UID: \"eedaddc4-a131-49bd-b077-4a61b9b8f04e\") " pod="openshift-monitoring/node-exporter-w5qjd"
Jul 02 12:40:05.544966 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.544937    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9cd847c-daf5-40f5-9a23-2718fa67f25c-metrics-client-ca\") pod \"kube-state-metrics-845b954455-5kfkq\" (UID: \"b9cd847c-daf5-40f5-9a23-2718fa67f25c\") " pod="openshift-monitoring/kube-state-metrics-845b954455-5kfkq"
Jul 02 12:40:05.545145 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.545043    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eedaddc4-a131-49bd-b077-4a61b9b8f04e-metrics-client-ca\") pod \"node-exporter-w5qjd\" (UID: \"eedaddc4-a131-49bd-b077-4a61b9b8f04e\") " pod="openshift-monitoring/node-exporter-w5qjd"
Jul 02 12:40:05.545145 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.545126    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eedaddc4-a131-49bd-b077-4a61b9b8f04e-node-exporter-textfile\") pod \"node-exporter-w5qjd\" (UID: \"eedaddc4-a131-49bd-b077-4a61b9b8f04e\") " pod="openshift-monitoring/node-exporter-w5qjd"
Jul 02 12:40:05.545254 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.545201    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eedaddc4-a131-49bd-b077-4a61b9b8f04e-sys\") pod \"node-exporter-w5qjd\" (UID: \"eedaddc4-a131-49bd-b077-4a61b9b8f04e\") " pod="openshift-monitoring/node-exporter-w5qjd"
Jul 02 12:40:05.545306 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.545280    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qvp7\" (UniqueName: \"kubernetes.io/projected/b9cd847c-daf5-40f5-9a23-2718fa67f25c-kube-api-access-2qvp7\") pod \"kube-state-metrics-845b954455-5kfkq\" (UID: \"b9cd847c-daf5-40f5-9a23-2718fa67f25c\") " pod="openshift-monitoring/kube-state-metrics-845b954455-5kfkq"
Jul 02 12:40:05.545362 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.545333    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b9cd847c-daf5-40f5-9a23-2718fa67f25c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-5kfkq\" (UID: \"b9cd847c-daf5-40f5-9a23-2718fa67f25c\") " pod="openshift-monitoring/kube-state-metrics-845b954455-5kfkq"
Jul 02 12:40:05.545415 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.545384    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b9cd847c-daf5-40f5-9a23-2718fa67f25c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-5kfkq\" (UID: \"b9cd847c-daf5-40f5-9a23-2718fa67f25c\") " pod="openshift-monitoring/kube-state-metrics-845b954455-5kfkq"
Jul 02 12:40:05.545460 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.545413    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eedaddc4-a131-49bd-b077-4a61b9b8f04e-node-exporter-tls\") pod \"node-exporter-w5qjd\" (UID: \"eedaddc4-a131-49bd-b077-4a61b9b8f04e\") " pod="openshift-monitoring/node-exporter-w5qjd"
Jul 02 12:40:05.545510 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.545483    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eedaddc4-a131-49bd-b077-4a61b9b8f04e-root\") pod \"node-exporter-w5qjd\" (UID: \"eedaddc4-a131-49bd-b077-4a61b9b8f04e\") " pod="openshift-monitoring/node-exporter-w5qjd"
Jul 02 12:40:05.545565 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.545508    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eedaddc4-a131-49bd-b077-4a61b9b8f04e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w5qjd\" (UID: \"eedaddc4-a131-49bd-b077-4a61b9b8f04e\") " pod="openshift-monitoring/node-exporter-w5qjd"
Jul 02 12:40:05.545820 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.545746    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b9cd847c-daf5-40f5-9a23-2718fa67f25c-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-5kfkq\" (UID: \"b9cd847c-daf5-40f5-9a23-2718fa67f25c\") " pod="openshift-monitoring/kube-state-metrics-845b954455-5kfkq"
Jul 02 12:40:05.545820 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.545802    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9cd847c-daf5-40f5-9a23-2718fa67f25c-metrics-client-ca\") pod \"kube-state-metrics-845b954455-5kfkq\" (UID: \"b9cd847c-daf5-40f5-9a23-2718fa67f25c\") " pod="openshift-monitoring/kube-state-metrics-845b954455-5kfkq"
Jul 02 12:40:05.546078 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.546021    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b9cd847c-daf5-40f5-9a23-2718fa67f25c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-5kfkq\" (UID: \"b9cd847c-daf5-40f5-9a23-2718fa67f25c\") " pod="openshift-monitoring/kube-state-metrics-845b954455-5kfkq"
Jul 02 12:40:05.546186 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.546113    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b9cd847c-daf5-40f5-9a23-2718fa67f25c-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-5kfkq\" (UID: \"b9cd847c-daf5-40f5-9a23-2718fa67f25c\") " pod="openshift-monitoring/kube-state-metrics-845b954455-5kfkq"
Jul 02 12:40:05.547893 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.547873    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b9cd847c-daf5-40f5-9a23-2718fa67f25c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-5kfkq\" (UID: \"b9cd847c-daf5-40f5-9a23-2718fa67f25c\") " pod="openshift-monitoring/kube-state-metrics-845b954455-5kfkq"
Jul 02 12:40:05.548001 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.547877    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9cd847c-daf5-40f5-9a23-2718fa67f25c-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-5kfkq\" (UID: \"b9cd847c-daf5-40f5-9a23-2718fa67f25c\") " pod="openshift-monitoring/kube-state-metrics-845b954455-5kfkq"
Jul 02 12:40:05.563632 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.563598    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qvp7\" (UniqueName: \"kubernetes.io/projected/b9cd847c-daf5-40f5-9a23-2718fa67f25c-kube-api-access-2qvp7\") pod \"kube-state-metrics-845b954455-5kfkq\" (UID: \"b9cd847c-daf5-40f5-9a23-2718fa67f25c\") " pod="openshift-monitoring/kube-state-metrics-845b954455-5kfkq"
Jul 02 12:40:05.596406 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.596368    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-845b954455-5kfkq"
Jul 02 12:40:05.646907 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.646861    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eedaddc4-a131-49bd-b077-4a61b9b8f04e-node-exporter-accelerators-collector-config\") pod \"node-exporter-w5qjd\" (UID: \"eedaddc4-a131-49bd-b077-4a61b9b8f04e\") " pod="openshift-monitoring/node-exporter-w5qjd"
Jul 02 12:40:05.646907 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.646925    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s8nt\" (UniqueName: \"kubernetes.io/projected/eedaddc4-a131-49bd-b077-4a61b9b8f04e-kube-api-access-8s8nt\") pod \"node-exporter-w5qjd\" (UID: \"eedaddc4-a131-49bd-b077-4a61b9b8f04e\") " pod="openshift-monitoring/node-exporter-w5qjd"
Jul 02 12:40:05.647160 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.646965    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eedaddc4-a131-49bd-b077-4a61b9b8f04e-node-exporter-wtmp\") pod \"node-exporter-w5qjd\" (UID: \"eedaddc4-a131-49bd-b077-4a61b9b8f04e\") " pod="openshift-monitoring/node-exporter-w5qjd"
Jul 02 12:40:05.647160 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.646996    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eedaddc4-a131-49bd-b077-4a61b9b8f04e-metrics-client-ca\") pod \"node-exporter-w5qjd\" (UID: \"eedaddc4-a131-49bd-b077-4a61b9b8f04e\") " pod="openshift-monitoring/node-exporter-w5qjd"
Jul 02 12:40:05.647160 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.647023    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eedaddc4-a131-49bd-b077-4a61b9b8f04e-node-exporter-textfile\") pod \"node-exporter-w5qjd\" (UID: \"eedaddc4-a131-49bd-b077-4a61b9b8f04e\") " pod="openshift-monitoring/node-exporter-w5qjd"
Jul 02 12:40:05.647160 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.647055    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eedaddc4-a131-49bd-b077-4a61b9b8f04e-sys\") pod \"node-exporter-w5qjd\" (UID: \"eedaddc4-a131-49bd-b077-4a61b9b8f04e\") " pod="openshift-monitoring/node-exporter-w5qjd"
Jul 02 12:40:05.647160 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.647091    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eedaddc4-a131-49bd-b077-4a61b9b8f04e-node-exporter-tls\") pod \"node-exporter-w5qjd\" (UID: \"eedaddc4-a131-49bd-b077-4a61b9b8f04e\") " pod="openshift-monitoring/node-exporter-w5qjd"
Jul 02 12:40:05.647160 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.647146    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eedaddc4-a131-49bd-b077-4a61b9b8f04e-root\") pod \"node-exporter-w5qjd\" (UID: \"eedaddc4-a131-49bd-b077-4a61b9b8f04e\") " pod="openshift-monitoring/node-exporter-w5qjd"
Jul 02 12:40:05.647395 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:40:05.647195    2576 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 02 12:40:05.647395 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:40:05.647260    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eedaddc4-a131-49bd-b077-4a61b9b8f04e-node-exporter-tls podName:eedaddc4-a131-49bd-b077-4a61b9b8f04e nodeName:}" failed. No retries permitted until 2026-07-02 12:40:06.147240316 +0000 UTC m=+94.754703338 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/eedaddc4-a131-49bd-b077-4a61b9b8f04e-node-exporter-tls") pod "node-exporter-w5qjd" (UID: "eedaddc4-a131-49bd-b077-4a61b9b8f04e") : secret "node-exporter-tls" not found
Jul 02 12:40:05.647395 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.647359    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eedaddc4-a131-49bd-b077-4a61b9b8f04e-sys\") pod \"node-exporter-w5qjd\" (UID: \"eedaddc4-a131-49bd-b077-4a61b9b8f04e\") " pod="openshift-monitoring/node-exporter-w5qjd"
Jul 02 12:40:05.647563 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.647539    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eedaddc4-a131-49bd-b077-4a61b9b8f04e-node-exporter-accelerators-collector-config\") pod \"node-exporter-w5qjd\" (UID: \"eedaddc4-a131-49bd-b077-4a61b9b8f04e\") " pod="openshift-monitoring/node-exporter-w5qjd"
Jul 02 12:40:05.647621 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.647193    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eedaddc4-a131-49bd-b077-4a61b9b8f04e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w5qjd\" (UID: \"eedaddc4-a131-49bd-b077-4a61b9b8f04e\") " pod="openshift-monitoring/node-exporter-w5qjd"
Jul 02 12:40:05.647621 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.647596    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eedaddc4-a131-49bd-b077-4a61b9b8f04e-node-exporter-wtmp\") pod \"node-exporter-w5qjd\" (UID: \"eedaddc4-a131-49bd-b077-4a61b9b8f04e\") " pod="openshift-monitoring/node-exporter-w5qjd"
Jul 02 12:40:05.647724 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.647653    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eedaddc4-a131-49bd-b077-4a61b9b8f04e-root\") pod \"node-exporter-w5qjd\" (UID: \"eedaddc4-a131-49bd-b077-4a61b9b8f04e\") " pod="openshift-monitoring/node-exporter-w5qjd"
Jul 02 12:40:05.647904 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.647881    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eedaddc4-a131-49bd-b077-4a61b9b8f04e-metrics-client-ca\") pod \"node-exporter-w5qjd\" (UID: \"eedaddc4-a131-49bd-b077-4a61b9b8f04e\") " pod="openshift-monitoring/node-exporter-w5qjd"
Jul 02 12:40:05.647993 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.647937    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eedaddc4-a131-49bd-b077-4a61b9b8f04e-node-exporter-textfile\") pod \"node-exporter-w5qjd\" (UID: \"eedaddc4-a131-49bd-b077-4a61b9b8f04e\") " pod="openshift-monitoring/node-exporter-w5qjd"
Jul 02 12:40:05.649866 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.649843    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eedaddc4-a131-49bd-b077-4a61b9b8f04e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w5qjd\" (UID: \"eedaddc4-a131-49bd-b077-4a61b9b8f04e\") " pod="openshift-monitoring/node-exporter-w5qjd"
Jul 02 12:40:05.668483 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.668454    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s8nt\" (UniqueName: \"kubernetes.io/projected/eedaddc4-a131-49bd-b077-4a61b9b8f04e-kube-api-access-8s8nt\") pod \"node-exporter-w5qjd\" (UID: \"eedaddc4-a131-49bd-b077-4a61b9b8f04e\") " pod="openshift-monitoring/node-exporter-w5qjd"
Jul 02 12:40:05.765880 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:05.765850    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-5kfkq"]
Jul 02 12:40:05.768313 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:40:05.768289    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9cd847c_daf5_40f5_9a23_2718fa67f25c.slice/crio-256c0b476165e63ecbe817a6f45cfc08c041c3559764878e15d00fc624573d71 WatchSource:0}: Error finding container 256c0b476165e63ecbe817a6f45cfc08c041c3559764878e15d00fc624573d71: Status 404 returned error can't find the container with id 256c0b476165e63ecbe817a6f45cfc08c041c3559764878e15d00fc624573d71
Jul 02 12:40:06.151702 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:06.151662    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eedaddc4-a131-49bd-b077-4a61b9b8f04e-node-exporter-tls\") pod \"node-exporter-w5qjd\" (UID: \"eedaddc4-a131-49bd-b077-4a61b9b8f04e\") " pod="openshift-monitoring/node-exporter-w5qjd"
Jul 02 12:40:06.154257 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:06.154222    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eedaddc4-a131-49bd-b077-4a61b9b8f04e-node-exporter-tls\") pod \"node-exporter-w5qjd\" (UID: \"eedaddc4-a131-49bd-b077-4a61b9b8f04e\") " pod="openshift-monitoring/node-exporter-w5qjd"
Jul 02 12:40:06.320305 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:06.320267    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w5qjd"
Jul 02 12:40:06.334067 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:40:06.334024    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeedaddc4_a131_49bd_b077_4a61b9b8f04e.slice/crio-c09685d5231b028930ca532a3218dfee4b47a4d54ad7eec5d9c8e56f69760889 WatchSource:0}: Error finding container c09685d5231b028930ca532a3218dfee4b47a4d54ad7eec5d9c8e56f69760889: Status 404 returned error can't find the container with id c09685d5231b028930ca532a3218dfee4b47a4d54ad7eec5d9c8e56f69760889
Jul 02 12:40:06.381594 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:06.381554    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w5qjd" event={"ID":"eedaddc4-a131-49bd-b077-4a61b9b8f04e","Type":"ContainerStarted","Data":"c09685d5231b028930ca532a3218dfee4b47a4d54ad7eec5d9c8e56f69760889"}
Jul 02 12:40:06.382964 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:06.382930    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-5kfkq" event={"ID":"b9cd847c-daf5-40f5-9a23-2718fa67f25c","Type":"ContainerStarted","Data":"256c0b476165e63ecbe817a6f45cfc08c041c3559764878e15d00fc624573d71"}
Jul 02 12:40:06.387722 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:06.387694    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f8c8cfdd6-t28hq"
Jul 02 12:40:08.241465 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:08.241425    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-68df4fdd69-qp4qs"]
Jul 02 12:40:08.245713 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:08.245690    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-68df4fdd69-qp4qs"
Jul 02 12:40:08.252055 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:08.251846    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 02 12:40:08.252055 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:08.251897    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 02 12:40:08.252055 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:08.251846    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 02 12:40:08.252055 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:08.251850    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 02 12:40:08.252356 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:08.252340    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 02 12:40:08.252422 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:08.252395    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-ebed2omfeussa\""
Jul 02 12:40:08.253394 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:08.253017    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-k2l5t\""
Jul 02 12:40:08.266866 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:08.266847    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-68df4fdd69-qp4qs"]
Jul 02 12:40:08.375215 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:08.375176    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3a8ddc64-6cb7-4593-9e82-07428d018488-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-68df4fdd69-qp4qs\" (UID: \"3a8ddc64-6cb7-4593-9e82-07428d018488\") " pod="openshift-monitoring/thanos-querier-68df4fdd69-qp4qs"
Jul 02 12:40:08.375396 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:08.375273    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a8ddc64-6cb7-4593-9e82-07428d018488-metrics-client-ca\") pod \"thanos-querier-68df4fdd69-qp4qs\" (UID: \"3a8ddc64-6cb7-4593-9e82-07428d018488\") " pod="openshift-monitoring/thanos-querier-68df4fdd69-qp4qs"
Jul 02 12:40:08.375396 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:08.375306    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3a8ddc64-6cb7-4593-9e82-07428d018488-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-68df4fdd69-qp4qs\" (UID: \"3a8ddc64-6cb7-4593-9e82-07428d018488\") " pod="openshift-monitoring/thanos-querier-68df4fdd69-qp4qs"
Jul 02 12:40:08.375516 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:08.375397    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hthwt\" (UniqueName: \"kubernetes.io/projected/3a8ddc64-6cb7-4593-9e82-07428d018488-kube-api-access-hthwt\") pod \"thanos-querier-68df4fdd69-qp4qs\" (UID: \"3a8ddc64-6cb7-4593-9e82-07428d018488\") " pod="openshift-monitoring/thanos-querier-68df4fdd69-qp4qs"
Jul 02 12:40:08.375516 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:08.375450    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3a8ddc64-6cb7-4593-9e82-07428d018488-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-68df4fdd69-qp4qs\" (UID: \"3a8ddc64-6cb7-4593-9e82-07428d018488\") " pod="openshift-monitoring/thanos-querier-68df4fdd69-qp4qs"
Jul 02 12:40:08.375516 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:08.375477    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3a8ddc64-6cb7-4593-9e82-07428d018488-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-68df4fdd69-qp4qs\" (UID: \"3a8ddc64-6cb7-4593-9e82-07428d018488\") " pod="openshift-monitoring/thanos-querier-68df4fdd69-qp4qs"
Jul 02 12:40:08.375668 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:08.375533    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3a8ddc64-6cb7-4593-9e82-07428d018488-secret-grpc-tls\") pod \"thanos-querier-68df4fdd69-qp4qs\" (UID: \"3a8ddc64-6cb7-4593-9e82-07428d018488\") " pod="openshift-monitoring/thanos-querier-68df4fdd69-qp4qs"
Jul 02 12:40:08.375668 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:08.375587    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3a8ddc64-6cb7-4593-9e82-07428d018488-secret-thanos-querier-tls\") pod \"thanos-querier-68df4fdd69-qp4qs\" (UID: \"3a8ddc64-6cb7-4593-9e82-07428d018488\") " pod="openshift-monitoring/thanos-querier-68df4fdd69-qp4qs"
Jul 02 12:40:08.476511 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:08.476474    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a8ddc64-6cb7-4593-9e82-07428d018488-metrics-client-ca\") pod \"thanos-querier-68df4fdd69-qp4qs\" (UID: \"3a8ddc64-6cb7-4593-9e82-07428d018488\") " pod="openshift-monitoring/thanos-querier-68df4fdd69-qp4qs"
Jul 02 12:40:08.476668 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:08.476519    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3a8ddc64-6cb7-4593-9e82-07428d018488-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-68df4fdd69-qp4qs\" (UID: \"3a8ddc64-6cb7-4593-9e82-07428d018488\") " pod="openshift-monitoring/thanos-querier-68df4fdd69-qp4qs"
Jul 02 12:40:08.476668 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:08.476583    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hthwt\" (UniqueName: \"kubernetes.io/projected/3a8ddc64-6cb7-4593-9e82-07428d018488-kube-api-access-hthwt\") pod \"thanos-querier-68df4fdd69-qp4qs\" (UID: \"3a8ddc64-6cb7-4593-9e82-07428d018488\") " pod="openshift-monitoring/thanos-querier-68df4fdd69-qp4qs"
Jul 02 12:40:08.476796 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:08.476768    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3a8ddc64-6cb7-4593-9e82-07428d018488-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-68df4fdd69-qp4qs\" (UID: \"3a8ddc64-6cb7-4593-9e82-07428d018488\") " pod="openshift-monitoring/thanos-querier-68df4fdd69-qp4qs"
Jul 02 12:40:08.476852 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:08.476823    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3a8ddc64-6cb7-4593-9e82-07428d018488-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-68df4fdd69-qp4qs\" (UID: \"3a8ddc64-6cb7-4593-9e82-07428d018488\") " pod="openshift-monitoring/thanos-querier-68df4fdd69-qp4qs"
Jul 02 12:40:08.476904 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:08.476853    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3a8ddc64-6cb7-4593-9e82-07428d018488-secret-grpc-tls\") pod \"thanos-querier-68df4fdd69-qp4qs\" (UID: \"3a8ddc64-6cb7-4593-9e82-07428d018488\") " pod="openshift-monitoring/thanos-querier-68df4fdd69-qp4qs"
Jul 02 12:40:08.476904 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:08.476877    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3a8ddc64-6cb7-4593-9e82-07428d018488-secret-thanos-querier-tls\") pod \"thanos-querier-68df4fdd69-qp4qs\" (UID: \"3a8ddc64-6cb7-4593-9e82-07428d018488\") " pod="openshift-monitoring/thanos-querier-68df4fdd69-qp4qs"
Jul 02 12:40:08.477036 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:08.476904    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3a8ddc64-6cb7-4593-9e82-07428d018488-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-68df4fdd69-qp4qs\" (UID: \"3a8ddc64-6cb7-4593-9e82-07428d018488\") " pod="openshift-monitoring/thanos-querier-68df4fdd69-qp4qs"
Jul 02 12:40:08.477336 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:08.477315    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a8ddc64-6cb7-4593-9e82-07428d018488-metrics-client-ca\") pod \"thanos-querier-68df4fdd69-qp4qs\" (UID: \"3a8ddc64-6cb7-4593-9e82-07428d018488\") " pod="openshift-monitoring/thanos-querier-68df4fdd69-qp4qs"
Jul 02 12:40:08.479796 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:08.479774    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3a8ddc64-6cb7-4593-9e82-07428d018488-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-68df4fdd69-qp4qs\" (UID: \"3a8ddc64-6cb7-4593-9e82-07428d018488\") " pod="openshift-monitoring/thanos-querier-68df4fdd69-qp4qs"
Jul 02 12:40:08.479887 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:08.479802    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3a8ddc64-6cb7-4593-9e82-07428d018488-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-68df4fdd69-qp4qs\" (UID: \"3a8ddc64-6cb7-4593-9e82-07428d018488\") " pod="openshift-monitoring/thanos-querier-68df4fdd69-qp4qs"
Jul 02 12:40:08.480095 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:08.480075    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3a8ddc64-6cb7-4593-9e82-07428d018488-secret-grpc-tls\") pod \"thanos-querier-68df4fdd69-qp4qs\" (UID: \"3a8ddc64-6cb7-4593-9e82-07428d018488\") " pod="openshift-monitoring/thanos-querier-68df4fdd69-qp4qs"
Jul 02 12:40:08.480180 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:08.480128    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3a8ddc64-6cb7-4593-9e82-07428d018488-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-68df4fdd69-qp4qs\" (UID: \"3a8ddc64-6cb7-4593-9e82-07428d018488\") " pod="openshift-monitoring/thanos-querier-68df4fdd69-qp4qs"
Jul 02 12:40:08.480388 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:08.480364    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3a8ddc64-6cb7-4593-9e82-07428d018488-secret-thanos-querier-tls\") pod \"thanos-querier-68df4fdd69-qp4qs\" (UID: \"3a8ddc64-6cb7-4593-9e82-07428d018488\") " pod="openshift-monitoring/thanos-querier-68df4fdd69-qp4qs"
Jul 02 12:40:08.480388 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:08.480364    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3a8ddc64-6cb7-4593-9e82-07428d018488-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-68df4fdd69-qp4qs\" (UID: \"3a8ddc64-6cb7-4593-9e82-07428d018488\") " pod="openshift-monitoring/thanos-querier-68df4fdd69-qp4qs"
Jul 02 12:40:08.487227 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:08.487203    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hthwt\" (UniqueName: \"kubernetes.io/projected/3a8ddc64-6cb7-4593-9e82-07428d018488-kube-api-access-hthwt\") pod \"thanos-querier-68df4fdd69-qp4qs\" (UID: \"3a8ddc64-6cb7-4593-9e82-07428d018488\") " pod="openshift-monitoring/thanos-querier-68df4fdd69-qp4qs"
Jul 02 12:40:08.556857 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:08.556795    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-68df4fdd69-qp4qs"
Jul 02 12:40:09.979064 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:09.979023    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5c86dcd54c-vfkvr"]
Jul 02 12:40:09.982826 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:09.982802    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5c86dcd54c-vfkvr"
Jul 02 12:40:10.008668 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:10.008644    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 02 12:40:10.009489 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:10.009462    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2j61mgd99b6k3\""
Jul 02 12:40:10.009489 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:10.009479    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 02 12:40:10.009699 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:10.009470    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 02 12:40:10.011581 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:10.011562    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-jtr7v\""
Jul 02 12:40:10.011581 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:10.011570    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 02 12:40:10.056661 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:10.056628    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5c86dcd54c-vfkvr"]
Jul 02 12:40:10.091926 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:10.091881    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/eacf016b-61ce-4c87-8963-a3c365506452-client-ca-bundle\") pod \"metrics-server-5c86dcd54c-vfkvr\" (UID: \"eacf016b-61ce-4c87-8963-a3c365506452\") " pod="openshift-monitoring/metrics-server-5c86dcd54c-vfkvr"
Jul 02 12:40:10.092086 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:10.091956    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/eacf016b-61ce-4c87-8963-a3c365506452-audit-log\") pod \"metrics-server-5c86dcd54c-vfkvr\" (UID: \"eacf016b-61ce-4c87-8963-a3c365506452\") " pod="openshift-monitoring/metrics-server-5c86dcd54c-vfkvr"
Jul 02 12:40:10.092086 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:10.092014    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/eacf016b-61ce-4c87-8963-a3c365506452-secret-metrics-server-client-certs\") pod \"metrics-server-5c86dcd54c-vfkvr\" (UID: \"eacf016b-61ce-4c87-8963-a3c365506452\") " pod="openshift-monitoring/metrics-server-5c86dcd54c-vfkvr"
Jul 02 12:40:10.092086 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:10.092046    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/eacf016b-61ce-4c87-8963-a3c365506452-metrics-server-audit-profiles\") pod \"metrics-server-5c86dcd54c-vfkvr\" (UID: \"eacf016b-61ce-4c87-8963-a3c365506452\") " pod="openshift-monitoring/metrics-server-5c86dcd54c-vfkvr"
Jul 02 12:40:10.092086 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:10.092078    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/eacf016b-61ce-4c87-8963-a3c365506452-secret-metrics-server-tls\") pod \"metrics-server-5c86dcd54c-vfkvr\" (UID: \"eacf016b-61ce-4c87-8963-a3c365506452\") " pod="openshift-monitoring/metrics-server-5c86dcd54c-vfkvr"
Jul 02 12:40:10.092241 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:10.092113    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eacf016b-61ce-4c87-8963-a3c365506452-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5c86dcd54c-vfkvr\" (UID: \"eacf016b-61ce-4c87-8963-a3c365506452\") " pod="openshift-monitoring/metrics-server-5c86dcd54c-vfkvr"
Jul 02 12:40:10.092241 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:10.092142    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hczlh\" (UniqueName: \"kubernetes.io/projected/eacf016b-61ce-4c87-8963-a3c365506452-kube-api-access-hczlh\") pod \"metrics-server-5c86dcd54c-vfkvr\" (UID: \"eacf016b-61ce-4c87-8963-a3c365506452\") " pod="openshift-monitoring/metrics-server-5c86dcd54c-vfkvr"
Jul 02 12:40:10.192759 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:10.192724    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/eacf016b-61ce-4c87-8963-a3c365506452-audit-log\") pod \"metrics-server-5c86dcd54c-vfkvr\" (UID: \"eacf016b-61ce-4c87-8963-a3c365506452\") " pod="openshift-monitoring/metrics-server-5c86dcd54c-vfkvr"
Jul 02 12:40:10.192939 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:10.192808    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/eacf016b-61ce-4c87-8963-a3c365506452-secret-metrics-server-client-certs\") pod \"metrics-server-5c86dcd54c-vfkvr\" (UID: \"eacf016b-61ce-4c87-8963-a3c365506452\") " pod="openshift-monitoring/metrics-server-5c86dcd54c-vfkvr"
Jul 02 12:40:10.192939 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:10.192846    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/eacf016b-61ce-4c87-8963-a3c365506452-metrics-server-audit-profiles\") pod \"metrics-server-5c86dcd54c-vfkvr\" (UID: \"eacf016b-61ce-4c87-8963-a3c365506452\") " pod="openshift-monitoring/metrics-server-5c86dcd54c-vfkvr"
Jul 02 12:40:10.192939 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:10.192892    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/eacf016b-61ce-4c87-8963-a3c365506452-secret-metrics-server-tls\") pod \"metrics-server-5c86dcd54c-vfkvr\" (UID: \"eacf016b-61ce-4c87-8963-a3c365506452\") " pod="openshift-monitoring/metrics-server-5c86dcd54c-vfkvr"
Jul 02 12:40:10.193091 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:10.192944    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eacf016b-61ce-4c87-8963-a3c365506452-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5c86dcd54c-vfkvr\" (UID: \"eacf016b-61ce-4c87-8963-a3c365506452\") " pod="openshift-monitoring/metrics-server-5c86dcd54c-vfkvr"
Jul 02 12:40:10.193091 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:10.192996    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hczlh\" (UniqueName: \"kubernetes.io/projected/eacf016b-61ce-4c87-8963-a3c365506452-kube-api-access-hczlh\") pod \"metrics-server-5c86dcd54c-vfkvr\" (UID: \"eacf016b-61ce-4c87-8963-a3c365506452\") " pod="openshift-monitoring/metrics-server-5c86dcd54c-vfkvr"
Jul 02 12:40:10.193091 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:10.193041    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/eacf016b-61ce-4c87-8963-a3c365506452-client-ca-bundle\") pod \"metrics-server-5c86dcd54c-vfkvr\" (UID: \"eacf016b-61ce-4c87-8963-a3c365506452\") " pod="openshift-monitoring/metrics-server-5c86dcd54c-vfkvr"
Jul 02 12:40:10.193665 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:10.193641    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/eacf016b-61ce-4c87-8963-a3c365506452-audit-log\") pod \"metrics-server-5c86dcd54c-vfkvr\" (UID: \"eacf016b-61ce-4c87-8963-a3c365506452\") " pod="openshift-monitoring/metrics-server-5c86dcd54c-vfkvr"
Jul 02 12:40:10.194157 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:10.194103    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eacf016b-61ce-4c87-8963-a3c365506452-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5c86dcd54c-vfkvr\" (UID: \"eacf016b-61ce-4c87-8963-a3c365506452\") " pod="openshift-monitoring/metrics-server-5c86dcd54c-vfkvr"
Jul 02 12:40:10.194430 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:10.194403    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/eacf016b-61ce-4c87-8963-a3c365506452-metrics-server-audit-profiles\") pod \"metrics-server-5c86dcd54c-vfkvr\" (UID: \"eacf016b-61ce-4c87-8963-a3c365506452\") " pod="openshift-monitoring/metrics-server-5c86dcd54c-vfkvr"
Jul 02 12:40:10.196534 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:10.196515    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/eacf016b-61ce-4c87-8963-a3c365506452-secret-metrics-server-client-certs\") pod \"metrics-server-5c86dcd54c-vfkvr\" (UID: \"eacf016b-61ce-4c87-8963-a3c365506452\") " pod="openshift-monitoring/metrics-server-5c86dcd54c-vfkvr"
Jul 02 12:40:10.196626 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:10.196575    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/eacf016b-61ce-4c87-8963-a3c365506452-client-ca-bundle\") pod \"metrics-server-5c86dcd54c-vfkvr\" (UID: \"eacf016b-61ce-4c87-8963-a3c365506452\") " pod="openshift-monitoring/metrics-server-5c86dcd54c-vfkvr"
Jul 02 12:40:10.196677 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:10.196637    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/eacf016b-61ce-4c87-8963-a3c365506452-secret-metrics-server-tls\") pod \"metrics-server-5c86dcd54c-vfkvr\" (UID: \"eacf016b-61ce-4c87-8963-a3c365506452\") " pod="openshift-monitoring/metrics-server-5c86dcd54c-vfkvr"
Jul 02 12:40:10.208351 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:10.208329    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hczlh\" (UniqueName: \"kubernetes.io/projected/eacf016b-61ce-4c87-8963-a3c365506452-kube-api-access-hczlh\") pod \"metrics-server-5c86dcd54c-vfkvr\" (UID: \"eacf016b-61ce-4c87-8963-a3c365506452\") " pod="openshift-monitoring/metrics-server-5c86dcd54c-vfkvr"
Jul 02 12:40:10.294138 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:10.294039    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5c86dcd54c-vfkvr"
Jul 02 12:40:12.249887 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:12.249853    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tczr2"
Jul 02 12:40:13.253182 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:13.253152    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-647779b8f4-t5vk7"]
Jul 02 12:40:13.299077 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:13.299043    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-87d995f48-mb6kg"]
Jul 02 12:40:13.475843 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:13.475805    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-87d995f48-mb6kg"]
Jul 02 12:40:13.476036 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:13.475964    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-87d995f48-mb6kg"
Jul 02 12:40:13.522649 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:13.522582    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01063732-7971-49d1-873a-bd89ecaf6064-oauth-serving-cert\") pod \"console-87d995f48-mb6kg\" (UID: \"01063732-7971-49d1-873a-bd89ecaf6064\") " pod="openshift-console/console-87d995f48-mb6kg"
Jul 02 12:40:13.522649 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:13.522630    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01063732-7971-49d1-873a-bd89ecaf6064-console-oauth-config\") pod \"console-87d995f48-mb6kg\" (UID: \"01063732-7971-49d1-873a-bd89ecaf6064\") " pod="openshift-console/console-87d995f48-mb6kg"
Jul 02 12:40:13.522855 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:13.522702    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01063732-7971-49d1-873a-bd89ecaf6064-service-ca\") pod \"console-87d995f48-mb6kg\" (UID: \"01063732-7971-49d1-873a-bd89ecaf6064\") " pod="openshift-console/console-87d995f48-mb6kg"
Jul 02 12:40:13.522855 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:13.522755    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4649\" (UniqueName: \"kubernetes.io/projected/01063732-7971-49d1-873a-bd89ecaf6064-kube-api-access-g4649\") pod \"console-87d995f48-mb6kg\" (UID: \"01063732-7971-49d1-873a-bd89ecaf6064\") " pod="openshift-console/console-87d995f48-mb6kg"
Jul 02 12:40:13.522995 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:13.522856    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01063732-7971-49d1-873a-bd89ecaf6064-trusted-ca-bundle\") pod \"console-87d995f48-mb6kg\" (UID: \"01063732-7971-49d1-873a-bd89ecaf6064\") " pod="openshift-console/console-87d995f48-mb6kg"
Jul 02 12:40:13.522995 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:13.522888    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01063732-7971-49d1-873a-bd89ecaf6064-console-serving-cert\") pod \"console-87d995f48-mb6kg\" (UID: \"01063732-7971-49d1-873a-bd89ecaf6064\") " pod="openshift-console/console-87d995f48-mb6kg"
Jul 02 12:40:13.522995 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:13.522936    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01063732-7971-49d1-873a-bd89ecaf6064-console-config\") pod \"console-87d995f48-mb6kg\" (UID: \"01063732-7971-49d1-873a-bd89ecaf6064\") " pod="openshift-console/console-87d995f48-mb6kg"
Jul 02 12:40:13.562756 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:13.562729    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-647779b8f4-t5vk7"
Jul 02 12:40:13.624098 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:13.624066    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01063732-7971-49d1-873a-bd89ecaf6064-service-ca\") pod \"console-87d995f48-mb6kg\" (UID: \"01063732-7971-49d1-873a-bd89ecaf6064\") " pod="openshift-console/console-87d995f48-mb6kg"
Jul 02 12:40:13.624262 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:13.624116    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4649\" (UniqueName: \"kubernetes.io/projected/01063732-7971-49d1-873a-bd89ecaf6064-kube-api-access-g4649\") pod \"console-87d995f48-mb6kg\" (UID: \"01063732-7971-49d1-873a-bd89ecaf6064\") " pod="openshift-console/console-87d995f48-mb6kg"
Jul 02 12:40:13.624344 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:13.624319    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01063732-7971-49d1-873a-bd89ecaf6064-trusted-ca-bundle\") pod \"console-87d995f48-mb6kg\" (UID: \"01063732-7971-49d1-873a-bd89ecaf6064\") " pod="openshift-console/console-87d995f48-mb6kg"
Jul 02 12:40:13.624401 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:13.624364    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01063732-7971-49d1-873a-bd89ecaf6064-console-serving-cert\") pod \"console-87d995f48-mb6kg\" (UID: \"01063732-7971-49d1-873a-bd89ecaf6064\") " pod="openshift-console/console-87d995f48-mb6kg"
Jul 02 12:40:13.624450 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:13.624397    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01063732-7971-49d1-873a-bd89ecaf6064-console-config\") pod \"console-87d995f48-mb6kg\" (UID: \"01063732-7971-49d1-873a-bd89ecaf6064\") " pod="openshift-console/console-87d995f48-mb6kg"
Jul 02 12:40:13.624450 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:13.624431    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01063732-7971-49d1-873a-bd89ecaf6064-oauth-serving-cert\") pod \"console-87d995f48-mb6kg\" (UID: \"01063732-7971-49d1-873a-bd89ecaf6064\") " pod="openshift-console/console-87d995f48-mb6kg"
Jul 02 12:40:13.624556 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:13.624473    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01063732-7971-49d1-873a-bd89ecaf6064-console-oauth-config\") pod \"console-87d995f48-mb6kg\" (UID: \"01063732-7971-49d1-873a-bd89ecaf6064\") " pod="openshift-console/console-87d995f48-mb6kg"
Jul 02 12:40:13.625279 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:13.624957    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01063732-7971-49d1-873a-bd89ecaf6064-service-ca\") pod \"console-87d995f48-mb6kg\" (UID: \"01063732-7971-49d1-873a-bd89ecaf6064\") " pod="openshift-console/console-87d995f48-mb6kg"
Jul 02 12:40:13.625279 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:13.625220    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01063732-7971-49d1-873a-bd89ecaf6064-console-config\") pod \"console-87d995f48-mb6kg\" (UID: \"01063732-7971-49d1-873a-bd89ecaf6064\") " pod="openshift-console/console-87d995f48-mb6kg"
Jul 02 12:40:13.625279 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:13.625251    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01063732-7971-49d1-873a-bd89ecaf6064-oauth-serving-cert\") pod \"console-87d995f48-mb6kg\" (UID: \"01063732-7971-49d1-873a-bd89ecaf6064\") " pod="openshift-console/console-87d995f48-mb6kg"
Jul 02 12:40:13.625695 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:13.625674    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01063732-7971-49d1-873a-bd89ecaf6064-trusted-ca-bundle\") pod \"console-87d995f48-mb6kg\" (UID: \"01063732-7971-49d1-873a-bd89ecaf6064\") " pod="openshift-console/console-87d995f48-mb6kg"
Jul 02 12:40:13.627123 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:13.627097    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01063732-7971-49d1-873a-bd89ecaf6064-console-oauth-config\") pod \"console-87d995f48-mb6kg\" (UID: \"01063732-7971-49d1-873a-bd89ecaf6064\") " pod="openshift-console/console-87d995f48-mb6kg"
Jul 02 12:40:13.627392 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:13.627374    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01063732-7971-49d1-873a-bd89ecaf6064-console-serving-cert\") pod \"console-87d995f48-mb6kg\" (UID: \"01063732-7971-49d1-873a-bd89ecaf6064\") " pod="openshift-console/console-87d995f48-mb6kg"
Jul 02 12:40:13.635837 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:13.635807    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4649\" (UniqueName: \"kubernetes.io/projected/01063732-7971-49d1-873a-bd89ecaf6064-kube-api-access-g4649\") pod \"console-87d995f48-mb6kg\" (UID: \"01063732-7971-49d1-873a-bd89ecaf6064\") " pod="openshift-console/console-87d995f48-mb6kg"
Jul 02 12:40:13.787186 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:13.787102    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-87d995f48-mb6kg"
Jul 02 12:40:14.930137 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:14.930071    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5c86dcd54c-vfkvr"]
Jul 02 12:40:14.941332 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:40:14.941303    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeacf016b_61ce_4c87_8963_a3c365506452.slice/crio-25a99efbd6ddb38c47c3aff1d7ef9e0a90fb7ad913ecfcd0f2b54f4e331e2319 WatchSource:0}: Error finding container 25a99efbd6ddb38c47c3aff1d7ef9e0a90fb7ad913ecfcd0f2b54f4e331e2319: Status 404 returned error can't find the container with id 25a99efbd6ddb38c47c3aff1d7ef9e0a90fb7ad913ecfcd0f2b54f4e331e2319
Jul 02 12:40:15.171757 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:15.171712    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-87d995f48-mb6kg"]
Jul 02 12:40:15.175652 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:15.175617    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-68df4fdd69-qp4qs"]
Jul 02 12:40:15.429081 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:15.429040    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5499b6fb8f-2c2lq" event={"ID":"5afef10c-ce2e-4a9d-8e49-e385b6ed71a2","Type":"ContainerStarted","Data":"072aff534a9e0ba359641ff82b1d0e084e95af6ce87a9192779e03aba8668638"}
Jul 02 12:40:15.429562 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:15.429527    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5499b6fb8f-2c2lq"
Jul 02 12:40:15.433345 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:15.433315    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-5kfkq" event={"ID":"b9cd847c-daf5-40f5-9a23-2718fa67f25c","Type":"ContainerStarted","Data":"d71a2908d9a87afc317bf608838bd4bca1ef81375122bb6edf00431146f0c712"}
Jul 02 12:40:15.433468 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:15.433371    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-5kfkq" event={"ID":"b9cd847c-daf5-40f5-9a23-2718fa67f25c","Type":"ContainerStarted","Data":"017db52a0d9370e3f7324bd466ef874aa6814b6edd11c84ad7ee074091391420"}
Jul 02 12:40:15.433468 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:15.433402    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-5kfkq" event={"ID":"b9cd847c-daf5-40f5-9a23-2718fa67f25c","Type":"ContainerStarted","Data":"58d5344ee854cc22bcf2ebb1c1752fac89245b6d2e615595ba174322b72588b2"}
Jul 02 12:40:15.434943 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:15.434872    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5c86dcd54c-vfkvr" event={"ID":"eacf016b-61ce-4c87-8963-a3c365506452","Type":"ContainerStarted","Data":"25a99efbd6ddb38c47c3aff1d7ef9e0a90fb7ad913ecfcd0f2b54f4e331e2319"}
Jul 02 12:40:15.440250 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:15.440204    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5499b6fb8f-2c2lq"
Jul 02 12:40:15.473815 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:40:15.473778    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01063732_7971_49d1_873a_bd89ecaf6064.slice/crio-c8f5327f3bb6651f1dc8c6fca0846fb73a22c4191306e5d8d103b8a635d26ac9 WatchSource:0}: Error finding container c8f5327f3bb6651f1dc8c6fca0846fb73a22c4191306e5d8d103b8a635d26ac9: Status 404 returned error can't find the container with id c8f5327f3bb6651f1dc8c6fca0846fb73a22c4191306e5d8d103b8a635d26ac9
Jul 02 12:40:15.474663 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:40:15.474630    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a8ddc64_6cb7_4593_9e82_07428d018488.slice/crio-c3f7819aa35b3a5ababf62df51d27f99d24fef4a052d6e9cfcfacd77cd5d4fc1 WatchSource:0}: Error finding container c3f7819aa35b3a5ababf62df51d27f99d24fef4a052d6e9cfcfacd77cd5d4fc1: Status 404 returned error can't find the container with id c3f7819aa35b3a5ababf62df51d27f99d24fef4a052d6e9cfcfacd77cd5d4fc1
Jul 02 12:40:15.477954 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:15.477879    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5499b6fb8f-2c2lq" podStartSLOduration=1.358538538 podStartE2EDuration="20.477858977s" totalImagesPullingTime="19.119320439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:39:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:40:15.476646462 +0000 UTC m=+104.084109517" watchObservedRunningTime="2026-07-02 12:40:15.477858977 +0000 UTC m=+104.085322001"
Jul 02 12:40:15.559605 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:15.559549    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-845b954455-5kfkq" podStartSLOduration=1.6230564429999998 podStartE2EDuration="10.559527399s" totalImagesPullingTime="8.936470956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:40:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:40:15.557890652 +0000 UTC m=+104.165353677" watchObservedRunningTime="2026-07-02 12:40:15.559527399 +0000 UTC m=+104.166990423"
Jul 02 12:40:16.446345 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:16.445270    2576 generic.go:356] "Generic (PLEG): container finished" podID="eedaddc4-a131-49bd-b077-4a61b9b8f04e" containerID="9fb2c1121a949831302dce6818c34d77208be5cf3274e31a89701fa407d99589" exitCode=0
Jul 02 12:40:16.446345 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:16.445387    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w5qjd" event={"ID":"eedaddc4-a131-49bd-b077-4a61b9b8f04e","Type":"ContainerDied","Data":"9fb2c1121a949831302dce6818c34d77208be5cf3274e31a89701fa407d99589"}
Jul 02 12:40:16.451083 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:16.450959    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68df4fdd69-qp4qs" event={"ID":"3a8ddc64-6cb7-4593-9e82-07428d018488","Type":"ContainerStarted","Data":"c3f7819aa35b3a5ababf62df51d27f99d24fef4a052d6e9cfcfacd77cd5d4fc1"}
Jul 02 12:40:16.454719 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:16.453807    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-87d995f48-mb6kg" event={"ID":"01063732-7971-49d1-873a-bd89ecaf6064","Type":"ContainerStarted","Data":"91ea11514afffa0931f3d1a4e4a38dd2bcb4df87d7a5cd3014500e183f56f1bf"}
Jul 02 12:40:16.454719 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:16.453859    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-87d995f48-mb6kg" event={"ID":"01063732-7971-49d1-873a-bd89ecaf6064","Type":"ContainerStarted","Data":"c8f5327f3bb6651f1dc8c6fca0846fb73a22c4191306e5d8d103b8a635d26ac9"}
Jul 02 12:40:16.550082 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:16.549620    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-87d995f48-mb6kg" podStartSLOduration=3.5495980080000002 podStartE2EDuration="3.549598008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:40:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:40:16.549066136 +0000 UTC m=+105.156529160" watchObservedRunningTime="2026-07-02 12:40:16.549598008 +0000 UTC m=+105.157061034"
Jul 02 12:40:17.460301 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:17.460274    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w5qjd" event={"ID":"eedaddc4-a131-49bd-b077-4a61b9b8f04e","Type":"ContainerStarted","Data":"928cbc153835027900c4ee872aee147fdfc5b81815629b548f6012e8644e0b77"}
Jul 02 12:40:18.467302 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:18.467181    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w5qjd" event={"ID":"eedaddc4-a131-49bd-b077-4a61b9b8f04e","Type":"ContainerStarted","Data":"df97ee578b10018394fc7e34608200811aef1fe0b327aeb16a878e26732bb9a0"}
Jul 02 12:40:18.504235 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:18.504183    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-w5qjd" podStartSLOduration=4.220269037 podStartE2EDuration="13.504168222s" totalImagesPullingTime="9.202861093s" totalInitContainerRuntime="81.038092ms" isStatefulPod=true podCreationTimestamp="2026-07-02 12:40:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:40:18.50152551 +0000 UTC m=+107.108988533" watchObservedRunningTime="2026-07-02 12:40:18.504168222 +0000 UTC m=+107.111631244"
Jul 02 12:40:19.066780 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:19.066742    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-847db68bb4-fgghh"]
Jul 02 12:40:19.474367 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:19.474325    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5c86dcd54c-vfkvr" event={"ID":"eacf016b-61ce-4c87-8963-a3c365506452","Type":"ContainerStarted","Data":"0bd4c5488b1582c6f19e3c7f80dc2a867eac811c070cdb7bb1a1eb1e2d78aaeb"}
Jul 02 12:40:19.477293 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:19.477262    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68df4fdd69-qp4qs" event={"ID":"3a8ddc64-6cb7-4593-9e82-07428d018488","Type":"ContainerStarted","Data":"049473fecafd073d9cf7cbdee981b0ec8af7fa50290f93e559f31d9dbfffc0b1"}
Jul 02 12:40:19.477435 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:19.477332    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68df4fdd69-qp4qs" event={"ID":"3a8ddc64-6cb7-4593-9e82-07428d018488","Type":"ContainerStarted","Data":"9172e8d4f171b7a8039b026305dd7ce7840190c38f501ca3abd3cc864e120f59"}
Jul 02 12:40:19.477435 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:19.477361    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68df4fdd69-qp4qs" event={"ID":"3a8ddc64-6cb7-4593-9e82-07428d018488","Type":"ContainerStarted","Data":"3b38c91905ffba9c7852c15e52e597ab094947cef265867dc2113b29ece0c1d9"}
Jul 02 12:40:19.496812 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:19.496754    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5c86dcd54c-vfkvr" podStartSLOduration=6.981712164 podStartE2EDuration="10.496734778s" totalImagesPullingTime="3.515022614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:40:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:40:19.495985543 +0000 UTC m=+108.103448566" watchObservedRunningTime="2026-07-02 12:40:19.496734778 +0000 UTC m=+108.104197800"
Jul 02 12:40:21.491993 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:21.491954    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68df4fdd69-qp4qs" event={"ID":"3a8ddc64-6cb7-4593-9e82-07428d018488","Type":"ContainerStarted","Data":"c4977316c151bed74ef1f1abc69b6ced87f2b0b92758038a1d99e229cc56b970"}
Jul 02 12:40:21.492455 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:21.492023    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68df4fdd69-qp4qs" event={"ID":"3a8ddc64-6cb7-4593-9e82-07428d018488","Type":"ContainerStarted","Data":"ffbd74a21634a5ea014f02c1c96a9964687e8a30c60f7764ed0301f2a863a119"}
Jul 02 12:40:21.492455 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:21.492058    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68df4fdd69-qp4qs" event={"ID":"3a8ddc64-6cb7-4593-9e82-07428d018488","Type":"ContainerStarted","Data":"f49ef96b6cb0eb1101a5b7292be829bbefac19f456c79f2428372a669eba429e"}
Jul 02 12:40:21.492455 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:21.492144    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-68df4fdd69-qp4qs"
Jul 02 12:40:21.524108 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:21.524045    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-68df4fdd69-qp4qs" podStartSLOduration=8.923706644 podStartE2EDuration="13.524024241s" totalImagesPullingTime="4.600317597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:40:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:40:21.523501288 +0000 UTC m=+110.130964324" watchObservedRunningTime="2026-07-02 12:40:21.524024241 +0000 UTC m=+110.131487266"
Jul 02 12:40:23.787810 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:23.787764    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-87d995f48-mb6kg"
Jul 02 12:40:23.788270 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:23.787846    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-87d995f48-mb6kg"
Jul 02 12:40:23.794162 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:23.794134    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-87d995f48-mb6kg"
Jul 02 12:40:24.509087 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:24.509061    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-87d995f48-mb6kg"
Jul 02 12:40:24.567270 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:24.567245    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f8c8cfdd6-t28hq"]
Jul 02 12:40:27.503354 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:27.503325    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-68df4fdd69-qp4qs"
Jul 02 12:40:30.294748 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:30.294712    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5c86dcd54c-vfkvr"
Jul 02 12:40:30.295246 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:30.294816    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5c86dcd54c-vfkvr"
Jul 02 12:40:38.276309 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:38.276234    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-647779b8f4-t5vk7" podUID="60d75a17-fa2d-403c-bc77-959ae8486a21" containerName="console" containerID="cri-o://a62ab1ab3921afac6497aa94ad27cdbe134d173ea0e85e6cc8c15a039435836d" gracePeriod=15
Jul 02 12:40:38.542587 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:38.542527    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-647779b8f4-t5vk7"
Jul 02 12:40:38.561606 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:38.561572    2576 generic.go:356] "Generic (PLEG): container finished" podID="60d75a17-fa2d-403c-bc77-959ae8486a21" containerID="a62ab1ab3921afac6497aa94ad27cdbe134d173ea0e85e6cc8c15a039435836d" exitCode=0
Jul 02 12:40:38.561739 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:38.561617    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-647779b8f4-t5vk7"
Jul 02 12:40:38.561739 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:38.561665    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-647779b8f4-t5vk7" event={"ID":"60d75a17-fa2d-403c-bc77-959ae8486a21","Type":"ContainerDied","Data":"a62ab1ab3921afac6497aa94ad27cdbe134d173ea0e85e6cc8c15a039435836d"}
Jul 02 12:40:38.561739 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:38.561719    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-647779b8f4-t5vk7" event={"ID":"60d75a17-fa2d-403c-bc77-959ae8486a21","Type":"ContainerDied","Data":"6f5e860fec7234d8fefb30425be6da026356db244c7e3e18fb65bb2c239f4f1f"}
Jul 02 12:40:38.561855 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:38.561746    2576 scope.go:117] "RemoveContainer" containerID="a62ab1ab3921afac6497aa94ad27cdbe134d173ea0e85e6cc8c15a039435836d"
Jul 02 12:40:38.571056 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:38.571038    2576 scope.go:117] "RemoveContainer" containerID="a62ab1ab3921afac6497aa94ad27cdbe134d173ea0e85e6cc8c15a039435836d"
Jul 02 12:40:38.571346 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:40:38.571323    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a62ab1ab3921afac6497aa94ad27cdbe134d173ea0e85e6cc8c15a039435836d\": container with ID starting with a62ab1ab3921afac6497aa94ad27cdbe134d173ea0e85e6cc8c15a039435836d not found: ID does not exist" containerID="a62ab1ab3921afac6497aa94ad27cdbe134d173ea0e85e6cc8c15a039435836d"
Jul 02 12:40:38.571397 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:38.571356    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a62ab1ab3921afac6497aa94ad27cdbe134d173ea0e85e6cc8c15a039435836d"} err="failed to get container status \"a62ab1ab3921afac6497aa94ad27cdbe134d173ea0e85e6cc8c15a039435836d\": rpc error: code = NotFound desc = could not find container \"a62ab1ab3921afac6497aa94ad27cdbe134d173ea0e85e6cc8c15a039435836d\": container with ID starting with a62ab1ab3921afac6497aa94ad27cdbe134d173ea0e85e6cc8c15a039435836d not found: ID does not exist"
Jul 02 12:40:38.684963 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:38.684931    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/60d75a17-fa2d-403c-bc77-959ae8486a21-service-ca\") pod \"60d75a17-fa2d-403c-bc77-959ae8486a21\" (UID: \"60d75a17-fa2d-403c-bc77-959ae8486a21\") "
Jul 02 12:40:38.685126 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:38.684976    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-848j6\" (UniqueName: \"kubernetes.io/projected/60d75a17-fa2d-403c-bc77-959ae8486a21-kube-api-access-848j6\") pod \"60d75a17-fa2d-403c-bc77-959ae8486a21\" (UID: \"60d75a17-fa2d-403c-bc77-959ae8486a21\") "
Jul 02 12:40:38.685126 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:38.684996    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60d75a17-fa2d-403c-bc77-959ae8486a21-trusted-ca-bundle\") pod \"60d75a17-fa2d-403c-bc77-959ae8486a21\" (UID: \"60d75a17-fa2d-403c-bc77-959ae8486a21\") "
Jul 02 12:40:38.685126 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:38.685014    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/60d75a17-fa2d-403c-bc77-959ae8486a21-console-serving-cert\") pod \"60d75a17-fa2d-403c-bc77-959ae8486a21\" (UID: \"60d75a17-fa2d-403c-bc77-959ae8486a21\") "
Jul 02 12:40:38.685126 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:38.685052    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/60d75a17-fa2d-403c-bc77-959ae8486a21-console-oauth-config\") pod \"60d75a17-fa2d-403c-bc77-959ae8486a21\" (UID: \"60d75a17-fa2d-403c-bc77-959ae8486a21\") "
Jul 02 12:40:38.685476 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:38.685144    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/60d75a17-fa2d-403c-bc77-959ae8486a21-console-config\") pod \"60d75a17-fa2d-403c-bc77-959ae8486a21\" (UID: \"60d75a17-fa2d-403c-bc77-959ae8486a21\") "
Jul 02 12:40:38.685476 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:38.685175    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/60d75a17-fa2d-403c-bc77-959ae8486a21-oauth-serving-cert\") pod \"60d75a17-fa2d-403c-bc77-959ae8486a21\" (UID: \"60d75a17-fa2d-403c-bc77-959ae8486a21\") "
Jul 02 12:40:38.685603 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:38.685469    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60d75a17-fa2d-403c-bc77-959ae8486a21-service-ca" (OuterVolumeSpecName: "service-ca") pod "60d75a17-fa2d-403c-bc77-959ae8486a21" (UID: "60d75a17-fa2d-403c-bc77-959ae8486a21"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:40:38.685603 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:38.685479    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60d75a17-fa2d-403c-bc77-959ae8486a21-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "60d75a17-fa2d-403c-bc77-959ae8486a21" (UID: "60d75a17-fa2d-403c-bc77-959ae8486a21"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:40:38.685603 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:38.685554    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60d75a17-fa2d-403c-bc77-959ae8486a21-console-config" (OuterVolumeSpecName: "console-config") pod "60d75a17-fa2d-403c-bc77-959ae8486a21" (UID: "60d75a17-fa2d-403c-bc77-959ae8486a21"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:40:38.685722 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:38.685676    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60d75a17-fa2d-403c-bc77-959ae8486a21-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "60d75a17-fa2d-403c-bc77-959ae8486a21" (UID: "60d75a17-fa2d-403c-bc77-959ae8486a21"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:40:38.687408 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:38.687382    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60d75a17-fa2d-403c-bc77-959ae8486a21-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "60d75a17-fa2d-403c-bc77-959ae8486a21" (UID: "60d75a17-fa2d-403c-bc77-959ae8486a21"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 12:40:38.687591 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:38.687570    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60d75a17-fa2d-403c-bc77-959ae8486a21-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "60d75a17-fa2d-403c-bc77-959ae8486a21" (UID: "60d75a17-fa2d-403c-bc77-959ae8486a21"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 12:40:38.687649 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:38.687589    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60d75a17-fa2d-403c-bc77-959ae8486a21-kube-api-access-848j6" (OuterVolumeSpecName: "kube-api-access-848j6") pod "60d75a17-fa2d-403c-bc77-959ae8486a21" (UID: "60d75a17-fa2d-403c-bc77-959ae8486a21"). InnerVolumeSpecName "kube-api-access-848j6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 12:40:38.786395 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:38.786366    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-848j6\" (UniqueName: \"kubernetes.io/projected/60d75a17-fa2d-403c-bc77-959ae8486a21-kube-api-access-848j6\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:40:38.786395 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:38.786391    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60d75a17-fa2d-403c-bc77-959ae8486a21-trusted-ca-bundle\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:40:38.786537 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:38.786400    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/60d75a17-fa2d-403c-bc77-959ae8486a21-console-serving-cert\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:40:38.786537 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:38.786410    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/60d75a17-fa2d-403c-bc77-959ae8486a21-console-oauth-config\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:40:38.786537 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:38.786419    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/60d75a17-fa2d-403c-bc77-959ae8486a21-console-config\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:40:38.786537 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:38.786427    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/60d75a17-fa2d-403c-bc77-959ae8486a21-oauth-serving-cert\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:40:38.786537 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:38.786434    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/60d75a17-fa2d-403c-bc77-959ae8486a21-service-ca\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:40:38.888750 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:38.888722    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-647779b8f4-t5vk7"]
Jul 02 12:40:38.898818 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:38.898693    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-647779b8f4-t5vk7"]
Jul 02 12:40:40.861832 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:40.861800    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60d75a17-fa2d-403c-bc77-959ae8486a21" path="/var/lib/kubelet/pods/60d75a17-fa2d-403c-bc77-959ae8486a21/volumes"
Jul 02 12:40:40.902310 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:40.902279    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7-metrics-certs\") pod \"network-metrics-daemon-sbnql\" (UID: \"86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7\") " pod="openshift-multus/network-metrics-daemon-sbnql"
Jul 02 12:40:40.904585 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:40.904564    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7-metrics-certs\") pod \"network-metrics-daemon-sbnql\" (UID: \"86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7\") " pod="openshift-multus/network-metrics-daemon-sbnql"
Jul 02 12:40:41.171835 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:41.171809    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bnkmz\""
Jul 02 12:40:41.179316 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:41.179301    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sbnql"
Jul 02 12:40:41.333567 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:41.333544    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sbnql"]
Jul 02 12:40:41.336120 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:40:41.336097    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86f0cfb8_8ff4_4bfe_9b61_42b97b9c76d7.slice/crio-dfa4c66265ca017447df2e448ec3fe05c82682f2fbcbcd7cdf85229c5369b3b4 WatchSource:0}: Error finding container dfa4c66265ca017447df2e448ec3fe05c82682f2fbcbcd7cdf85229c5369b3b4: Status 404 returned error can't find the container with id dfa4c66265ca017447df2e448ec3fe05c82682f2fbcbcd7cdf85229c5369b3b4
Jul 02 12:40:41.575371 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:41.575276    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sbnql" event={"ID":"86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7","Type":"ContainerStarted","Data":"dfa4c66265ca017447df2e448ec3fe05c82682f2fbcbcd7cdf85229c5369b3b4"}
Jul 02 12:40:44.103897 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:44.103808    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-847db68bb4-fgghh" podUID="69ae490b-bd41-4ab2-b05c-3393fea68351" containerName="registry" containerID="cri-o://c911f71aa4fdbca6c227b897a68ad926809fe144d43c6d16f013a9d0b7204cdb" gracePeriod=30
Jul 02 12:40:44.375985 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:44.375926    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-847db68bb4-fgghh"
Jul 02 12:40:44.536398 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:44.536367    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/69ae490b-bd41-4ab2-b05c-3393fea68351-registry-certificates\") pod \"69ae490b-bd41-4ab2-b05c-3393fea68351\" (UID: \"69ae490b-bd41-4ab2-b05c-3393fea68351\") "
Jul 02 12:40:44.536563 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:44.536414    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzpmp\" (UniqueName: \"kubernetes.io/projected/69ae490b-bd41-4ab2-b05c-3393fea68351-kube-api-access-wzpmp\") pod \"69ae490b-bd41-4ab2-b05c-3393fea68351\" (UID: \"69ae490b-bd41-4ab2-b05c-3393fea68351\") "
Jul 02 12:40:44.536563 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:44.536489    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/69ae490b-bd41-4ab2-b05c-3393fea68351-installation-pull-secrets\") pod \"69ae490b-bd41-4ab2-b05c-3393fea68351\" (UID: \"69ae490b-bd41-4ab2-b05c-3393fea68351\") "
Jul 02 12:40:44.536563 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:44.536542    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69ae490b-bd41-4ab2-b05c-3393fea68351-bound-sa-token\") pod \"69ae490b-bd41-4ab2-b05c-3393fea68351\" (UID: \"69ae490b-bd41-4ab2-b05c-3393fea68351\") "
Jul 02 12:40:44.536713 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:44.536575    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/69ae490b-bd41-4ab2-b05c-3393fea68351-ca-trust-extracted\") pod \"69ae490b-bd41-4ab2-b05c-3393fea68351\" (UID: \"69ae490b-bd41-4ab2-b05c-3393fea68351\") "
Jul 02 12:40:44.536713 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:44.536604    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/69ae490b-bd41-4ab2-b05c-3393fea68351-image-registry-private-configuration\") pod \"69ae490b-bd41-4ab2-b05c-3393fea68351\" (UID: \"69ae490b-bd41-4ab2-b05c-3393fea68351\") "
Jul 02 12:40:44.536713 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:44.536635    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69ae490b-bd41-4ab2-b05c-3393fea68351-registry-tls\") pod \"69ae490b-bd41-4ab2-b05c-3393fea68351\" (UID: \"69ae490b-bd41-4ab2-b05c-3393fea68351\") "
Jul 02 12:40:44.537023 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:44.536695    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/69ae490b-bd41-4ab2-b05c-3393fea68351-trusted-ca\") pod \"69ae490b-bd41-4ab2-b05c-3393fea68351\" (UID: \"69ae490b-bd41-4ab2-b05c-3393fea68351\") "
Jul 02 12:40:44.537435 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:44.536959    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69ae490b-bd41-4ab2-b05c-3393fea68351-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "69ae490b-bd41-4ab2-b05c-3393fea68351" (UID: "69ae490b-bd41-4ab2-b05c-3393fea68351"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:40:44.537435 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:44.537307    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69ae490b-bd41-4ab2-b05c-3393fea68351-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "69ae490b-bd41-4ab2-b05c-3393fea68351" (UID: "69ae490b-bd41-4ab2-b05c-3393fea68351"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:40:44.537592 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:44.537522    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/69ae490b-bd41-4ab2-b05c-3393fea68351-trusted-ca\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:40:44.537592 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:44.537540    2576 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/69ae490b-bd41-4ab2-b05c-3393fea68351-registry-certificates\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:40:44.539362 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:44.539334    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69ae490b-bd41-4ab2-b05c-3393fea68351-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "69ae490b-bd41-4ab2-b05c-3393fea68351" (UID: "69ae490b-bd41-4ab2-b05c-3393fea68351"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 12:40:44.539448 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:44.539370    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69ae490b-bd41-4ab2-b05c-3393fea68351-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "69ae490b-bd41-4ab2-b05c-3393fea68351" (UID: "69ae490b-bd41-4ab2-b05c-3393fea68351"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 12:40:44.539448 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:44.539396    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69ae490b-bd41-4ab2-b05c-3393fea68351-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "69ae490b-bd41-4ab2-b05c-3393fea68351" (UID: "69ae490b-bd41-4ab2-b05c-3393fea68351"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 12:40:44.539448 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:44.539431    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69ae490b-bd41-4ab2-b05c-3393fea68351-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "69ae490b-bd41-4ab2-b05c-3393fea68351" (UID: "69ae490b-bd41-4ab2-b05c-3393fea68351"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 12:40:44.539569 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:44.539455    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69ae490b-bd41-4ab2-b05c-3393fea68351-kube-api-access-wzpmp" (OuterVolumeSpecName: "kube-api-access-wzpmp") pod "69ae490b-bd41-4ab2-b05c-3393fea68351" (UID: "69ae490b-bd41-4ab2-b05c-3393fea68351"). InnerVolumeSpecName "kube-api-access-wzpmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 12:40:44.544907 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:44.544880    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69ae490b-bd41-4ab2-b05c-3393fea68351-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "69ae490b-bd41-4ab2-b05c-3393fea68351" (UID: "69ae490b-bd41-4ab2-b05c-3393fea68351"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 12:40:44.594043 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:44.593999    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sbnql" event={"ID":"86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7","Type":"ContainerStarted","Data":"6b832b36e7cae2e041783c600b36d71a63aeec0c27357c2c8161634b091b4908"}
Jul 02 12:40:44.594164 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:44.594107    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sbnql" event={"ID":"86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7","Type":"ContainerStarted","Data":"4e39aa60db477b26cb75011caaa55780591fa56d6fb4763212df5d109fc4053b"}
Jul 02 12:40:44.595214 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:44.595191    2576 generic.go:356] "Generic (PLEG): container finished" podID="69ae490b-bd41-4ab2-b05c-3393fea68351" containerID="c911f71aa4fdbca6c227b897a68ad926809fe144d43c6d16f013a9d0b7204cdb" exitCode=0
Jul 02 12:40:44.595294 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:44.595248    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-847db68bb4-fgghh" event={"ID":"69ae490b-bd41-4ab2-b05c-3393fea68351","Type":"ContainerDied","Data":"c911f71aa4fdbca6c227b897a68ad926809fe144d43c6d16f013a9d0b7204cdb"}
Jul 02 12:40:44.595294 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:44.595279    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-847db68bb4-fgghh"
Jul 02 12:40:44.595294 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:44.595291    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-847db68bb4-fgghh" event={"ID":"69ae490b-bd41-4ab2-b05c-3393fea68351","Type":"ContainerDied","Data":"3acee6408a31cb6ec7d6a712b470473d564b8524586f99d7b7e846ea92653741"}
Jul 02 12:40:44.595418 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:44.595319    2576 scope.go:117] "RemoveContainer" containerID="c911f71aa4fdbca6c227b897a68ad926809fe144d43c6d16f013a9d0b7204cdb"
Jul 02 12:40:44.603729 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:44.603710    2576 scope.go:117] "RemoveContainer" containerID="c911f71aa4fdbca6c227b897a68ad926809fe144d43c6d16f013a9d0b7204cdb"
Jul 02 12:40:44.603978 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:40:44.603957    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c911f71aa4fdbca6c227b897a68ad926809fe144d43c6d16f013a9d0b7204cdb\": container with ID starting with c911f71aa4fdbca6c227b897a68ad926809fe144d43c6d16f013a9d0b7204cdb not found: ID does not exist" containerID="c911f71aa4fdbca6c227b897a68ad926809fe144d43c6d16f013a9d0b7204cdb"
Jul 02 12:40:44.604043 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:44.603985    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c911f71aa4fdbca6c227b897a68ad926809fe144d43c6d16f013a9d0b7204cdb"} err="failed to get container status \"c911f71aa4fdbca6c227b897a68ad926809fe144d43c6d16f013a9d0b7204cdb\": rpc error: code = NotFound desc = could not find container \"c911f71aa4fdbca6c227b897a68ad926809fe144d43c6d16f013a9d0b7204cdb\": container with ID starting with c911f71aa4fdbca6c227b897a68ad926809fe144d43c6d16f013a9d0b7204cdb not found: ID does not exist"
Jul 02 12:40:44.612057 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:44.612019    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-sbnql" podStartSLOduration=129.429935624 podStartE2EDuration="2m11.612003609s" totalImagesPullingTime="2.182067985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:40:44.610059564 +0000 UTC m=+133.217522586" watchObservedRunningTime="2026-07-02 12:40:44.612003609 +0000 UTC m=+133.219466632"
Jul 02 12:40:44.628330 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:44.628275    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-847db68bb4-fgghh"]
Jul 02 12:40:44.634220 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:44.634203    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-847db68bb4-fgghh"]
Jul 02 12:40:44.638103 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:44.638082    2576 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/69ae490b-bd41-4ab2-b05c-3393fea68351-installation-pull-secrets\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:40:44.638154 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:44.638107    2576 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69ae490b-bd41-4ab2-b05c-3393fea68351-bound-sa-token\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:40:44.638154 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:44.638118    2576 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/69ae490b-bd41-4ab2-b05c-3393fea68351-ca-trust-extracted\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:40:44.638154 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:44.638126    2576 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/69ae490b-bd41-4ab2-b05c-3393fea68351-image-registry-private-configuration\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:40:44.638154 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:44.638134    2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69ae490b-bd41-4ab2-b05c-3393fea68351-registry-tls\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:40:44.638154 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:44.638142    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzpmp\" (UniqueName: \"kubernetes.io/projected/69ae490b-bd41-4ab2-b05c-3393fea68351-kube-api-access-wzpmp\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:40:44.862655 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:44.862622    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69ae490b-bd41-4ab2-b05c-3393fea68351" path="/var/lib/kubelet/pods/69ae490b-bd41-4ab2-b05c-3393fea68351/volumes"
Jul 02 12:40:48.611564 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:48.611533    2576 generic.go:356] "Generic (PLEG): container finished" podID="dabc8a5d-7710-42b1-93e2-bf1a0f5c94e5" containerID="79d2898796f8d9c8a5b8b8d61cc47d3ebe040adcd915389a3485d6dda6dcfb13" exitCode=0
Jul 02 12:40:48.611988 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:48.611629    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5f67b55555-r4pwp" event={"ID":"dabc8a5d-7710-42b1-93e2-bf1a0f5c94e5","Type":"ContainerDied","Data":"79d2898796f8d9c8a5b8b8d61cc47d3ebe040adcd915389a3485d6dda6dcfb13"}
Jul 02 12:40:48.612039 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:48.611998    2576 scope.go:117] "RemoveContainer" containerID="79d2898796f8d9c8a5b8b8d61cc47d3ebe040adcd915389a3485d6dda6dcfb13"
Jul 02 12:40:49.590527 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:49.590466    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-f8c8cfdd6-t28hq" podUID="e1593a5a-d0b6-4d6d-8a47-9a653e2778db" containerName="console" containerID="cri-o://ff10dd224ed42c25597ca3a1dcd9adeb74a59f10fb35e241f99becd368abd202" gracePeriod=15
Jul 02 12:40:49.617612 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:49.617584    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5f67b55555-r4pwp" event={"ID":"dabc8a5d-7710-42b1-93e2-bf1a0f5c94e5","Type":"ContainerStarted","Data":"642b777b3537c533f1012365fcba93dfae6befde30105a4f0305e72b07c30061"}
Jul 02 12:40:49.874662 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:49.874637    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f8c8cfdd6-t28hq"
Jul 02 12:40:49.885181 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:49.885159    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e1593a5a-d0b6-4d6d-8a47-9a653e2778db-oauth-serving-cert\") pod \"e1593a5a-d0b6-4d6d-8a47-9a653e2778db\" (UID: \"e1593a5a-d0b6-4d6d-8a47-9a653e2778db\") "
Jul 02 12:40:49.885276 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:49.885235    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e1593a5a-d0b6-4d6d-8a47-9a653e2778db-service-ca\") pod \"e1593a5a-d0b6-4d6d-8a47-9a653e2778db\" (UID: \"e1593a5a-d0b6-4d6d-8a47-9a653e2778db\") "
Jul 02 12:40:49.885276 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:49.885254    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e1593a5a-d0b6-4d6d-8a47-9a653e2778db-console-oauth-config\") pod \"e1593a5a-d0b6-4d6d-8a47-9a653e2778db\" (UID: \"e1593a5a-d0b6-4d6d-8a47-9a653e2778db\") "
Jul 02 12:40:49.885276 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:49.885273    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8j9xj\" (UniqueName: \"kubernetes.io/projected/e1593a5a-d0b6-4d6d-8a47-9a653e2778db-kube-api-access-8j9xj\") pod \"e1593a5a-d0b6-4d6d-8a47-9a653e2778db\" (UID: \"e1593a5a-d0b6-4d6d-8a47-9a653e2778db\") "
Jul 02 12:40:49.885392 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:49.885295    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e1593a5a-d0b6-4d6d-8a47-9a653e2778db-console-serving-cert\") pod \"e1593a5a-d0b6-4d6d-8a47-9a653e2778db\" (UID: \"e1593a5a-d0b6-4d6d-8a47-9a653e2778db\") "
Jul 02 12:40:49.885392 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:49.885312    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e1593a5a-d0b6-4d6d-8a47-9a653e2778db-console-config\") pod \"e1593a5a-d0b6-4d6d-8a47-9a653e2778db\" (UID: \"e1593a5a-d0b6-4d6d-8a47-9a653e2778db\") "
Jul 02 12:40:49.885676 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:49.885566    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1593a5a-d0b6-4d6d-8a47-9a653e2778db-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e1593a5a-d0b6-4d6d-8a47-9a653e2778db" (UID: "e1593a5a-d0b6-4d6d-8a47-9a653e2778db"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:40:49.885781 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:49.885676    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1593a5a-d0b6-4d6d-8a47-9a653e2778db-service-ca" (OuterVolumeSpecName: "service-ca") pod "e1593a5a-d0b6-4d6d-8a47-9a653e2778db" (UID: "e1593a5a-d0b6-4d6d-8a47-9a653e2778db"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:40:49.885781 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:49.885734    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1593a5a-d0b6-4d6d-8a47-9a653e2778db-console-config" (OuterVolumeSpecName: "console-config") pod "e1593a5a-d0b6-4d6d-8a47-9a653e2778db" (UID: "e1593a5a-d0b6-4d6d-8a47-9a653e2778db"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:40:49.887390 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:49.887359    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1593a5a-d0b6-4d6d-8a47-9a653e2778db-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e1593a5a-d0b6-4d6d-8a47-9a653e2778db" (UID: "e1593a5a-d0b6-4d6d-8a47-9a653e2778db"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 12:40:49.887838 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:49.887817    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1593a5a-d0b6-4d6d-8a47-9a653e2778db-kube-api-access-8j9xj" (OuterVolumeSpecName: "kube-api-access-8j9xj") pod "e1593a5a-d0b6-4d6d-8a47-9a653e2778db" (UID: "e1593a5a-d0b6-4d6d-8a47-9a653e2778db"). InnerVolumeSpecName "kube-api-access-8j9xj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 12:40:49.887838 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:49.887821    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1593a5a-d0b6-4d6d-8a47-9a653e2778db-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e1593a5a-d0b6-4d6d-8a47-9a653e2778db" (UID: "e1593a5a-d0b6-4d6d-8a47-9a653e2778db"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 12:40:49.986488 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:49.986454    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e1593a5a-d0b6-4d6d-8a47-9a653e2778db-service-ca\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:40:49.986488 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:49.986482    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e1593a5a-d0b6-4d6d-8a47-9a653e2778db-console-oauth-config\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:40:49.986685 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:49.986499    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j9xj\" (UniqueName: \"kubernetes.io/projected/e1593a5a-d0b6-4d6d-8a47-9a653e2778db-kube-api-access-8j9xj\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:40:49.986685 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:49.986512    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e1593a5a-d0b6-4d6d-8a47-9a653e2778db-console-serving-cert\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:40:49.986685 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:49.986523    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e1593a5a-d0b6-4d6d-8a47-9a653e2778db-console-config\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:40:49.986685 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:49.986534    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e1593a5a-d0b6-4d6d-8a47-9a653e2778db-oauth-serving-cert\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:40:50.299758 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:50.299727    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5c86dcd54c-vfkvr"
Jul 02 12:40:50.303401 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:50.303378    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5c86dcd54c-vfkvr"
Jul 02 12:40:50.623432 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:50.623360    2576 generic.go:356] "Generic (PLEG): container finished" podID="e1593a5a-d0b6-4d6d-8a47-9a653e2778db" containerID="ff10dd224ed42c25597ca3a1dcd9adeb74a59f10fb35e241f99becd368abd202" exitCode=0
Jul 02 12:40:50.623432 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:50.623407    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f8c8cfdd6-t28hq"
Jul 02 12:40:50.623841 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:50.623445    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f8c8cfdd6-t28hq" event={"ID":"e1593a5a-d0b6-4d6d-8a47-9a653e2778db","Type":"ContainerDied","Data":"ff10dd224ed42c25597ca3a1dcd9adeb74a59f10fb35e241f99becd368abd202"}
Jul 02 12:40:50.623841 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:50.623493    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f8c8cfdd6-t28hq" event={"ID":"e1593a5a-d0b6-4d6d-8a47-9a653e2778db","Type":"ContainerDied","Data":"da17c96116d8c09cecd042754efdcfc365d8fa48d7b63b38a73a101bd96c31d7"}
Jul 02 12:40:50.623841 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:50.623513    2576 scope.go:117] "RemoveContainer" containerID="ff10dd224ed42c25597ca3a1dcd9adeb74a59f10fb35e241f99becd368abd202"
Jul 02 12:40:50.632086 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:50.632069    2576 scope.go:117] "RemoveContainer" containerID="ff10dd224ed42c25597ca3a1dcd9adeb74a59f10fb35e241f99becd368abd202"
Jul 02 12:40:50.632340 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:40:50.632321    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff10dd224ed42c25597ca3a1dcd9adeb74a59f10fb35e241f99becd368abd202\": container with ID starting with ff10dd224ed42c25597ca3a1dcd9adeb74a59f10fb35e241f99becd368abd202 not found: ID does not exist" containerID="ff10dd224ed42c25597ca3a1dcd9adeb74a59f10fb35e241f99becd368abd202"
Jul 02 12:40:50.632392 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:50.632350    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff10dd224ed42c25597ca3a1dcd9adeb74a59f10fb35e241f99becd368abd202"} err="failed to get container status \"ff10dd224ed42c25597ca3a1dcd9adeb74a59f10fb35e241f99becd368abd202\": rpc error: code = NotFound desc = could not find container \"ff10dd224ed42c25597ca3a1dcd9adeb74a59f10fb35e241f99becd368abd202\": container with ID starting with ff10dd224ed42c25597ca3a1dcd9adeb74a59f10fb35e241f99becd368abd202 not found: ID does not exist"
Jul 02 12:40:50.645226 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:50.645204    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f8c8cfdd6-t28hq"]
Jul 02 12:40:50.650590 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:50.650570    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-f8c8cfdd6-t28hq"]
Jul 02 12:40:50.862388 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:40:50.862359    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1593a5a-d0b6-4d6d-8a47-9a653e2778db" path="/var/lib/kubelet/pods/e1593a5a-d0b6-4d6d-8a47-9a653e2778db/volumes"
Jul 02 12:41:27.210585 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.210509    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 02 12:41:27.210986 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.210800    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69ae490b-bd41-4ab2-b05c-3393fea68351" containerName="registry"
Jul 02 12:41:27.210986 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.210810    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="69ae490b-bd41-4ab2-b05c-3393fea68351" containerName="registry"
Jul 02 12:41:27.210986 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.210824    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60d75a17-fa2d-403c-bc77-959ae8486a21" containerName="console"
Jul 02 12:41:27.210986 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.210829    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="60d75a17-fa2d-403c-bc77-959ae8486a21" containerName="console"
Jul 02 12:41:27.210986 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.210838    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1593a5a-d0b6-4d6d-8a47-9a653e2778db" containerName="console"
Jul 02 12:41:27.210986 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.210844    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1593a5a-d0b6-4d6d-8a47-9a653e2778db" containerName="console"
Jul 02 12:41:27.210986 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.210893    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1593a5a-d0b6-4d6d-8a47-9a653e2778db" containerName="console"
Jul 02 12:41:27.210986 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.210902    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="69ae490b-bd41-4ab2-b05c-3393fea68351" containerName="registry"
Jul 02 12:41:27.210986 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.210925    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="60d75a17-fa2d-403c-bc77-959ae8486a21" containerName="console"
Jul 02 12:41:27.214853 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.214832    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:41:27.219230 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.219202    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 02 12:41:27.219368 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.219278    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 02 12:41:27.219368 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.219301    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 02 12:41:27.219368 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.219313    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 02 12:41:27.219368 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.219362    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 02 12:41:27.219616 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.219540    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-vxklc\""
Jul 02 12:41:27.219761 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.219744    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 02 12:41:27.219849 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.219773    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 02 12:41:27.219849 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.219798    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 02 12:41:27.224195 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.224176    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 02 12:41:27.237443 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.237423    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 02 12:41:27.295014 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.294977    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e61794ef-6594-4970-b7ff-3897895050fb-web-config\") pod \"alertmanager-main-0\" (UID: \"e61794ef-6594-4970-b7ff-3897895050fb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:41:27.295014 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.295016    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzqc4\" (UniqueName: \"kubernetes.io/projected/e61794ef-6594-4970-b7ff-3897895050fb-kube-api-access-xzqc4\") pod \"alertmanager-main-0\" (UID: \"e61794ef-6594-4970-b7ff-3897895050fb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:41:27.295229 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.295036    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e61794ef-6594-4970-b7ff-3897895050fb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e61794ef-6594-4970-b7ff-3897895050fb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:41:27.295229 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.295054    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e61794ef-6594-4970-b7ff-3897895050fb-config-out\") pod \"alertmanager-main-0\" (UID: \"e61794ef-6594-4970-b7ff-3897895050fb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:41:27.295229 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.295079    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e61794ef-6594-4970-b7ff-3897895050fb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e61794ef-6594-4970-b7ff-3897895050fb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:41:27.295229 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.295105    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e61794ef-6594-4970-b7ff-3897895050fb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e61794ef-6594-4970-b7ff-3897895050fb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:41:27.295229 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.295133    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e61794ef-6594-4970-b7ff-3897895050fb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e61794ef-6594-4970-b7ff-3897895050fb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:41:27.295229 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.295193    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e61794ef-6594-4970-b7ff-3897895050fb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e61794ef-6594-4970-b7ff-3897895050fb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:41:27.295229 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.295223    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e61794ef-6594-4970-b7ff-3897895050fb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e61794ef-6594-4970-b7ff-3897895050fb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:41:27.295472 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.295244    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e61794ef-6594-4970-b7ff-3897895050fb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e61794ef-6594-4970-b7ff-3897895050fb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:41:27.295472 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.295269    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e61794ef-6594-4970-b7ff-3897895050fb-config-volume\") pod \"alertmanager-main-0\" (UID: \"e61794ef-6594-4970-b7ff-3897895050fb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:41:27.295472 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.295296    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e61794ef-6594-4970-b7ff-3897895050fb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e61794ef-6594-4970-b7ff-3897895050fb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:41:27.295472 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.295321    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e61794ef-6594-4970-b7ff-3897895050fb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e61794ef-6594-4970-b7ff-3897895050fb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:41:27.396252 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.396217    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e61794ef-6594-4970-b7ff-3897895050fb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e61794ef-6594-4970-b7ff-3897895050fb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:41:27.396252 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.396254    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e61794ef-6594-4970-b7ff-3897895050fb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e61794ef-6594-4970-b7ff-3897895050fb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:41:27.396481 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.396274    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e61794ef-6594-4970-b7ff-3897895050fb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e61794ef-6594-4970-b7ff-3897895050fb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:41:27.396481 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.396293    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e61794ef-6594-4970-b7ff-3897895050fb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e61794ef-6594-4970-b7ff-3897895050fb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:41:27.396481 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.396320    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e61794ef-6594-4970-b7ff-3897895050fb-config-volume\") pod \"alertmanager-main-0\" (UID: \"e61794ef-6594-4970-b7ff-3897895050fb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:41:27.396481 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.396360    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e61794ef-6594-4970-b7ff-3897895050fb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e61794ef-6594-4970-b7ff-3897895050fb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:41:27.396481 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.396395    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e61794ef-6594-4970-b7ff-3897895050fb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e61794ef-6594-4970-b7ff-3897895050fb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:41:27.396481 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.396461    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e61794ef-6594-4970-b7ff-3897895050fb-web-config\") pod \"alertmanager-main-0\" (UID: \"e61794ef-6594-4970-b7ff-3897895050fb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:41:27.396798 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.396485    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzqc4\" (UniqueName: \"kubernetes.io/projected/e61794ef-6594-4970-b7ff-3897895050fb-kube-api-access-xzqc4\") pod \"alertmanager-main-0\" (UID: \"e61794ef-6594-4970-b7ff-3897895050fb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:41:27.396798 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.396532    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e61794ef-6594-4970-b7ff-3897895050fb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e61794ef-6594-4970-b7ff-3897895050fb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:41:27.396798 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.396561    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e61794ef-6594-4970-b7ff-3897895050fb-config-out\") pod \"alertmanager-main-0\" (UID: \"e61794ef-6594-4970-b7ff-3897895050fb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:41:27.396798 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.396584    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e61794ef-6594-4970-b7ff-3897895050fb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e61794ef-6594-4970-b7ff-3897895050fb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:41:27.396798 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.396620    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e61794ef-6594-4970-b7ff-3897895050fb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e61794ef-6594-4970-b7ff-3897895050fb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:41:27.397101 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.397068    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e61794ef-6594-4970-b7ff-3897895050fb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e61794ef-6594-4970-b7ff-3897895050fb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:41:27.397994 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.397394    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e61794ef-6594-4970-b7ff-3897895050fb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e61794ef-6594-4970-b7ff-3897895050fb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:41:27.397994 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.397678    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e61794ef-6594-4970-b7ff-3897895050fb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e61794ef-6594-4970-b7ff-3897895050fb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:41:27.399465 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.399436    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e61794ef-6594-4970-b7ff-3897895050fb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e61794ef-6594-4970-b7ff-3897895050fb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:41:27.399566 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.399447    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e61794ef-6594-4970-b7ff-3897895050fb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e61794ef-6594-4970-b7ff-3897895050fb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:41:27.399672 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.399645    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e61794ef-6594-4970-b7ff-3897895050fb-config-volume\") pod \"alertmanager-main-0\" (UID: \"e61794ef-6594-4970-b7ff-3897895050fb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:41:27.399672 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.399669    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e61794ef-6594-4970-b7ff-3897895050fb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e61794ef-6594-4970-b7ff-3897895050fb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:41:27.399840 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.399676    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e61794ef-6594-4970-b7ff-3897895050fb-config-out\") pod \"alertmanager-main-0\" (UID: \"e61794ef-6594-4970-b7ff-3897895050fb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:41:27.399840 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.399810    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e61794ef-6594-4970-b7ff-3897895050fb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e61794ef-6594-4970-b7ff-3897895050fb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:41:27.400086 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.400067    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e61794ef-6594-4970-b7ff-3897895050fb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e61794ef-6594-4970-b7ff-3897895050fb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:41:27.400580 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.400557    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e61794ef-6594-4970-b7ff-3897895050fb-web-config\") pod \"alertmanager-main-0\" (UID: \"e61794ef-6594-4970-b7ff-3897895050fb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:41:27.401465 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.401444    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e61794ef-6594-4970-b7ff-3897895050fb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e61794ef-6594-4970-b7ff-3897895050fb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:41:27.407815 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.407795    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzqc4\" (UniqueName: \"kubernetes.io/projected/e61794ef-6594-4970-b7ff-3897895050fb-kube-api-access-xzqc4\") pod \"alertmanager-main-0\" (UID: \"e61794ef-6594-4970-b7ff-3897895050fb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:41:27.526635 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.526553    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 02 12:41:27.676896 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.676852    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 02 12:41:27.679478 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:41:27.679449    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode61794ef_6594_4970_b7ff_3897895050fb.slice/crio-70e7ac26dc147fe42b1db3aed8b1a3a8c184daa12b2173e54b999e9e36ae3418 WatchSource:0}: Error finding container 70e7ac26dc147fe42b1db3aed8b1a3a8c184daa12b2173e54b999e9e36ae3418: Status 404 returned error can't find the container with id 70e7ac26dc147fe42b1db3aed8b1a3a8c184daa12b2173e54b999e9e36ae3418
Jul 02 12:41:27.776013 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:27.775983    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e61794ef-6594-4970-b7ff-3897895050fb","Type":"ContainerStarted","Data":"70e7ac26dc147fe42b1db3aed8b1a3a8c184daa12b2173e54b999e9e36ae3418"}
Jul 02 12:41:29.579534 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:29.579498    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-56bb4d87d4-wt4r7"]
Jul 02 12:41:29.583541 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:29.583516    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-56bb4d87d4-wt4r7"
Jul 02 12:41:29.587046 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:29.587021    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 02 12:41:29.588156 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:29.588130    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 02 12:41:29.588279 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:29.587243    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 02 12:41:29.588330 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:29.587347    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 02 12:41:29.588406 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:29.587438    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-f4hzs\""
Jul 02 12:41:29.588644 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:29.588608    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 02 12:41:29.592239 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:29.592214    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 02 12:41:29.600231 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:29.600208    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-56bb4d87d4-wt4r7"]
Jul 02 12:41:29.718006 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:29.717965    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0e9931d-45ab-4c8e-8175-3da0da60ba21-metrics-client-ca\") pod \"telemeter-client-56bb4d87d4-wt4r7\" (UID: \"f0e9931d-45ab-4c8e-8175-3da0da60ba21\") " pod="openshift-monitoring/telemeter-client-56bb4d87d4-wt4r7"
Jul 02 12:41:29.718006 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:29.718011    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f0e9931d-45ab-4c8e-8175-3da0da60ba21-telemeter-client-tls\") pod \"telemeter-client-56bb4d87d4-wt4r7\" (UID: \"f0e9931d-45ab-4c8e-8175-3da0da60ba21\") " pod="openshift-monitoring/telemeter-client-56bb4d87d4-wt4r7"
Jul 02 12:41:29.718271 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:29.718066    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0e9931d-45ab-4c8e-8175-3da0da60ba21-serving-certs-ca-bundle\") pod \"telemeter-client-56bb4d87d4-wt4r7\" (UID: \"f0e9931d-45ab-4c8e-8175-3da0da60ba21\") " pod="openshift-monitoring/telemeter-client-56bb4d87d4-wt4r7"
Jul 02 12:41:29.718271 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:29.718104    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9kp9\" (UniqueName: \"kubernetes.io/projected/f0e9931d-45ab-4c8e-8175-3da0da60ba21-kube-api-access-d9kp9\") pod \"telemeter-client-56bb4d87d4-wt4r7\" (UID: \"f0e9931d-45ab-4c8e-8175-3da0da60ba21\") " pod="openshift-monitoring/telemeter-client-56bb4d87d4-wt4r7"
Jul 02 12:41:29.718271 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:29.718128    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f0e9931d-45ab-4c8e-8175-3da0da60ba21-secret-telemeter-client\") pod \"telemeter-client-56bb4d87d4-wt4r7\" (UID: \"f0e9931d-45ab-4c8e-8175-3da0da60ba21\") " pod="openshift-monitoring/telemeter-client-56bb4d87d4-wt4r7"
Jul 02 12:41:29.718271 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:29.718145    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f0e9931d-45ab-4c8e-8175-3da0da60ba21-federate-client-tls\") pod \"telemeter-client-56bb4d87d4-wt4r7\" (UID: \"f0e9931d-45ab-4c8e-8175-3da0da60ba21\") " pod="openshift-monitoring/telemeter-client-56bb4d87d4-wt4r7"
Jul 02 12:41:29.718271 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:29.718243    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0e9931d-45ab-4c8e-8175-3da0da60ba21-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56bb4d87d4-wt4r7\" (UID: \"f0e9931d-45ab-4c8e-8175-3da0da60ba21\") " pod="openshift-monitoring/telemeter-client-56bb4d87d4-wt4r7"
Jul 02 12:41:29.718538 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:29.718285    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f0e9931d-45ab-4c8e-8175-3da0da60ba21-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56bb4d87d4-wt4r7\" (UID: \"f0e9931d-45ab-4c8e-8175-3da0da60ba21\") " pod="openshift-monitoring/telemeter-client-56bb4d87d4-wt4r7"
Jul 02 12:41:29.787093 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:29.787046    2576 generic.go:356] "Generic (PLEG): container finished" podID="e61794ef-6594-4970-b7ff-3897895050fb" containerID="c7b2ab166dfb0f3849d0b5467fb4556ed17b02f8fe2ca3f4d320c1b2f4545feb" exitCode=0
Jul 02 12:41:29.787252 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:29.787141    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e61794ef-6594-4970-b7ff-3897895050fb","Type":"ContainerDied","Data":"c7b2ab166dfb0f3849d0b5467fb4556ed17b02f8fe2ca3f4d320c1b2f4545feb"}
Jul 02 12:41:29.819223 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:29.819195    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0e9931d-45ab-4c8e-8175-3da0da60ba21-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56bb4d87d4-wt4r7\" (UID: \"f0e9931d-45ab-4c8e-8175-3da0da60ba21\") " pod="openshift-monitoring/telemeter-client-56bb4d87d4-wt4r7"
Jul 02 12:41:29.819376 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:29.819250    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f0e9931d-45ab-4c8e-8175-3da0da60ba21-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56bb4d87d4-wt4r7\" (UID: \"f0e9931d-45ab-4c8e-8175-3da0da60ba21\") " pod="openshift-monitoring/telemeter-client-56bb4d87d4-wt4r7"
Jul 02 12:41:29.819376 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:29.819281    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0e9931d-45ab-4c8e-8175-3da0da60ba21-metrics-client-ca\") pod \"telemeter-client-56bb4d87d4-wt4r7\" (UID: \"f0e9931d-45ab-4c8e-8175-3da0da60ba21\") " pod="openshift-monitoring/telemeter-client-56bb4d87d4-wt4r7"
Jul 02 12:41:29.819376 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:29.819308    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f0e9931d-45ab-4c8e-8175-3da0da60ba21-telemeter-client-tls\") pod \"telemeter-client-56bb4d87d4-wt4r7\" (UID: \"f0e9931d-45ab-4c8e-8175-3da0da60ba21\") " pod="openshift-monitoring/telemeter-client-56bb4d87d4-wt4r7"
Jul 02 12:41:29.819376 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:29.819366    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0e9931d-45ab-4c8e-8175-3da0da60ba21-serving-certs-ca-bundle\") pod \"telemeter-client-56bb4d87d4-wt4r7\" (UID: \"f0e9931d-45ab-4c8e-8175-3da0da60ba21\") " pod="openshift-monitoring/telemeter-client-56bb4d87d4-wt4r7"
Jul 02 12:41:29.819570 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:29.819398    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9kp9\" (UniqueName: \"kubernetes.io/projected/f0e9931d-45ab-4c8e-8175-3da0da60ba21-kube-api-access-d9kp9\") pod \"telemeter-client-56bb4d87d4-wt4r7\" (UID: \"f0e9931d-45ab-4c8e-8175-3da0da60ba21\") " pod="openshift-monitoring/telemeter-client-56bb4d87d4-wt4r7"
Jul 02 12:41:29.819570 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:29.819430    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f0e9931d-45ab-4c8e-8175-3da0da60ba21-secret-telemeter-client\") pod \"telemeter-client-56bb4d87d4-wt4r7\" (UID: \"f0e9931d-45ab-4c8e-8175-3da0da60ba21\") " pod="openshift-monitoring/telemeter-client-56bb4d87d4-wt4r7"
Jul 02 12:41:29.819570 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:29.819456    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f0e9931d-45ab-4c8e-8175-3da0da60ba21-federate-client-tls\") pod \"telemeter-client-56bb4d87d4-wt4r7\" (UID: \"f0e9931d-45ab-4c8e-8175-3da0da60ba21\") " pod="openshift-monitoring/telemeter-client-56bb4d87d4-wt4r7"
Jul 02 12:41:29.820161 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:29.820134    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0e9931d-45ab-4c8e-8175-3da0da60ba21-metrics-client-ca\") pod \"telemeter-client-56bb4d87d4-wt4r7\" (UID: \"f0e9931d-45ab-4c8e-8175-3da0da60ba21\") " pod="openshift-monitoring/telemeter-client-56bb4d87d4-wt4r7"
Jul 02 12:41:29.820297 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:29.820139    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0e9931d-45ab-4c8e-8175-3da0da60ba21-serving-certs-ca-bundle\") pod \"telemeter-client-56bb4d87d4-wt4r7\" (UID: \"f0e9931d-45ab-4c8e-8175-3da0da60ba21\") " pod="openshift-monitoring/telemeter-client-56bb4d87d4-wt4r7"
Jul 02 12:41:29.820433 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:29.820391    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0e9931d-45ab-4c8e-8175-3da0da60ba21-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56bb4d87d4-wt4r7\" (UID: \"f0e9931d-45ab-4c8e-8175-3da0da60ba21\") " pod="openshift-monitoring/telemeter-client-56bb4d87d4-wt4r7"
Jul 02 12:41:29.822276 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:29.822247    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f0e9931d-45ab-4c8e-8175-3da0da60ba21-secret-telemeter-client\") pod \"telemeter-client-56bb4d87d4-wt4r7\" (UID: \"f0e9931d-45ab-4c8e-8175-3da0da60ba21\") " pod="openshift-monitoring/telemeter-client-56bb4d87d4-wt4r7"
Jul 02 12:41:29.822371 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:29.822291    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f0e9931d-45ab-4c8e-8175-3da0da60ba21-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56bb4d87d4-wt4r7\" (UID: \"f0e9931d-45ab-4c8e-8175-3da0da60ba21\") " pod="openshift-monitoring/telemeter-client-56bb4d87d4-wt4r7"
Jul 02 12:41:29.822371 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:29.822257    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f0e9931d-45ab-4c8e-8175-3da0da60ba21-telemeter-client-tls\") pod \"telemeter-client-56bb4d87d4-wt4r7\" (UID: \"f0e9931d-45ab-4c8e-8175-3da0da60ba21\") " pod="openshift-monitoring/telemeter-client-56bb4d87d4-wt4r7"
Jul 02 12:41:29.822458 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:29.822442    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f0e9931d-45ab-4c8e-8175-3da0da60ba21-federate-client-tls\") pod \"telemeter-client-56bb4d87d4-wt4r7\" (UID: \"f0e9931d-45ab-4c8e-8175-3da0da60ba21\") " pod="openshift-monitoring/telemeter-client-56bb4d87d4-wt4r7"
Jul 02 12:41:29.826641 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:29.826614    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9kp9\" (UniqueName: \"kubernetes.io/projected/f0e9931d-45ab-4c8e-8175-3da0da60ba21-kube-api-access-d9kp9\") pod \"telemeter-client-56bb4d87d4-wt4r7\" (UID: \"f0e9931d-45ab-4c8e-8175-3da0da60ba21\") " pod="openshift-monitoring/telemeter-client-56bb4d87d4-wt4r7"
Jul 02 12:41:29.895532 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:29.895496    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-56bb4d87d4-wt4r7"
Jul 02 12:41:30.047685 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:30.047617    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-56bb4d87d4-wt4r7"]
Jul 02 12:41:30.792507 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:30.792473    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56bb4d87d4-wt4r7" event={"ID":"f0e9931d-45ab-4c8e-8175-3da0da60ba21","Type":"ContainerStarted","Data":"a5b9c7306acb007a84ba2e9afd6ffa5fac36e9246f4ad6bbddb79dd4ed813361"}
Jul 02 12:41:32.804857 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:32.804762    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e61794ef-6594-4970-b7ff-3897895050fb","Type":"ContainerStarted","Data":"77959c4717dfe74f80766f12d119bfc0d116b480e14eec8678adb8bdeaa66838"}
Jul 02 12:41:32.804857 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:32.804847    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e61794ef-6594-4970-b7ff-3897895050fb","Type":"ContainerStarted","Data":"56b163282e992519a286450bf6da201a412787501f1717cf5aee7b9609f1a088"}
Jul 02 12:41:32.805367 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:32.804897    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e61794ef-6594-4970-b7ff-3897895050fb","Type":"ContainerStarted","Data":"1add925edd1ec09725851673c0802b29e6ad67fbab3b5ff4ede35aca2b83af65"}
Jul 02 12:41:32.805367 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:32.804950    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e61794ef-6594-4970-b7ff-3897895050fb","Type":"ContainerStarted","Data":"ddcc0343187bcc76666736d6493f27891056e6c2bce910aab4fb776737e888df"}
Jul 02 12:41:32.805367 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:32.804993    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e61794ef-6594-4970-b7ff-3897895050fb","Type":"ContainerStarted","Data":"f85e2412d6de702ca8bf05c2bab7f832f89b12f72e05e9185d2fb729869c4eaf"}
Jul 02 12:41:32.805367 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:32.805038    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e61794ef-6594-4970-b7ff-3897895050fb","Type":"ContainerStarted","Data":"f8e9c4e2387630f6a75d8154d3b43484931911e684591181be6f0299e861ba64"}
Jul 02 12:41:32.806908 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:32.806883    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56bb4d87d4-wt4r7" event={"ID":"f0e9931d-45ab-4c8e-8175-3da0da60ba21","Type":"ContainerStarted","Data":"10994c1c74648b32e43243daa8e47850e13acd80448a7bb994b9dfc2e3c29498"}
Jul 02 12:41:32.807029 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:32.806957    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56bb4d87d4-wt4r7" event={"ID":"f0e9931d-45ab-4c8e-8175-3da0da60ba21","Type":"ContainerStarted","Data":"af1b01bffbe766b49d8503eabe27dc3011f9a0fea56865f2d3fc0592dffb74ff"}
Jul 02 12:41:32.807029 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:32.806978    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56bb4d87d4-wt4r7" event={"ID":"f0e9931d-45ab-4c8e-8175-3da0da60ba21","Type":"ContainerStarted","Data":"7dedc5569d93b4258ca088609cbe4a2839add36086da8d27bb0f68ea6bf249bf"}
Jul 02 12:41:32.834134 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:32.834085    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.4446737069999998 podStartE2EDuration="5.834070995s" totalImagesPullingTime="3.375634209s" totalInitContainerRuntime="13.763079ms" isStatefulPod=false podCreationTimestamp="2026-07-02 12:41:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:41:32.832173616 +0000 UTC m=+181.439636639" watchObservedRunningTime="2026-07-02 12:41:32.834070995 +0000 UTC m=+181.441534018"
Jul 02 12:41:32.854743 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:32.854694    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-56bb4d87d4-wt4r7" podStartSLOduration=1.77669448 podStartE2EDuration="3.8546779s" totalImagesPullingTime="2.07798342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:41:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:41:32.853460653 +0000 UTC m=+181.460923669" watchObservedRunningTime="2026-07-02 12:41:32.8546779 +0000 UTC m=+181.462140922"
Jul 02 12:41:33.566800 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:33.566750    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-76959fb9f4-hq6kg"]
Jul 02 12:41:33.570518 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:33.570493    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76959fb9f4-hq6kg"
Jul 02 12:41:33.583037 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:33.583008    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76959fb9f4-hq6kg"]
Jul 02 12:41:33.755483 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:33.755452    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4-console-config\") pod \"console-76959fb9f4-hq6kg\" (UID: \"264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4\") " pod="openshift-console/console-76959fb9f4-hq6kg"
Jul 02 12:41:33.755636 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:33.755498    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xzr6\" (UniqueName: \"kubernetes.io/projected/264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4-kube-api-access-6xzr6\") pod \"console-76959fb9f4-hq6kg\" (UID: \"264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4\") " pod="openshift-console/console-76959fb9f4-hq6kg"
Jul 02 12:41:33.755636 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:33.755579    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4-trusted-ca-bundle\") pod \"console-76959fb9f4-hq6kg\" (UID: \"264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4\") " pod="openshift-console/console-76959fb9f4-hq6kg"
Jul 02 12:41:33.755636 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:33.755621    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4-oauth-serving-cert\") pod \"console-76959fb9f4-hq6kg\" (UID: \"264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4\") " pod="openshift-console/console-76959fb9f4-hq6kg"
Jul 02 12:41:33.755754 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:33.755656    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4-service-ca\") pod \"console-76959fb9f4-hq6kg\" (UID: \"264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4\") " pod="openshift-console/console-76959fb9f4-hq6kg"
Jul 02 12:41:33.755790 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:33.755757    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4-console-serving-cert\") pod \"console-76959fb9f4-hq6kg\" (UID: \"264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4\") " pod="openshift-console/console-76959fb9f4-hq6kg"
Jul 02 12:41:33.755822 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:33.755801    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4-console-oauth-config\") pod \"console-76959fb9f4-hq6kg\" (UID: \"264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4\") " pod="openshift-console/console-76959fb9f4-hq6kg"
Jul 02 12:41:33.856745 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:33.856652    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4-console-config\") pod \"console-76959fb9f4-hq6kg\" (UID: \"264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4\") " pod="openshift-console/console-76959fb9f4-hq6kg"
Jul 02 12:41:33.857962 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:33.857931    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xzr6\" (UniqueName: \"kubernetes.io/projected/264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4-kube-api-access-6xzr6\") pod \"console-76959fb9f4-hq6kg\" (UID: \"264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4\") " pod="openshift-console/console-76959fb9f4-hq6kg"
Jul 02 12:41:33.858094 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:33.857970    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4-console-config\") pod \"console-76959fb9f4-hq6kg\" (UID: \"264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4\") " pod="openshift-console/console-76959fb9f4-hq6kg"
Jul 02 12:41:33.858094 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:33.857982    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4-trusted-ca-bundle\") pod \"console-76959fb9f4-hq6kg\" (UID: \"264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4\") " pod="openshift-console/console-76959fb9f4-hq6kg"
Jul 02 12:41:33.858094 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:33.858037    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4-oauth-serving-cert\") pod \"console-76959fb9f4-hq6kg\" (UID: \"264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4\") " pod="openshift-console/console-76959fb9f4-hq6kg"
Jul 02 12:41:33.858094 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:33.858083    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4-service-ca\") pod \"console-76959fb9f4-hq6kg\" (UID: \"264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4\") " pod="openshift-console/console-76959fb9f4-hq6kg"
Jul 02 12:41:33.858287 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:33.858155    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4-console-serving-cert\") pod \"console-76959fb9f4-hq6kg\" (UID: \"264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4\") " pod="openshift-console/console-76959fb9f4-hq6kg"
Jul 02 12:41:33.858287 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:33.858217    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4-console-oauth-config\") pod \"console-76959fb9f4-hq6kg\" (UID: \"264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4\") " pod="openshift-console/console-76959fb9f4-hq6kg"
Jul 02 12:41:33.858813 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:33.858784    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4-oauth-serving-cert\") pod \"console-76959fb9f4-hq6kg\" (UID: \"264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4\") " pod="openshift-console/console-76959fb9f4-hq6kg"
Jul 02 12:41:33.858813 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:33.858801    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4-service-ca\") pod \"console-76959fb9f4-hq6kg\" (UID: \"264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4\") " pod="openshift-console/console-76959fb9f4-hq6kg"
Jul 02 12:41:33.858992 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:33.858795    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4-trusted-ca-bundle\") pod \"console-76959fb9f4-hq6kg\" (UID: \"264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4\") " pod="openshift-console/console-76959fb9f4-hq6kg"
Jul 02 12:41:33.860758 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:33.860732    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4-console-oauth-config\") pod \"console-76959fb9f4-hq6kg\" (UID: \"264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4\") " pod="openshift-console/console-76959fb9f4-hq6kg"
Jul 02 12:41:33.861154 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:33.861136    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4-console-serving-cert\") pod \"console-76959fb9f4-hq6kg\" (UID: \"264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4\") " pod="openshift-console/console-76959fb9f4-hq6kg"
Jul 02 12:41:33.867619 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:33.867596    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xzr6\" (UniqueName: \"kubernetes.io/projected/264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4-kube-api-access-6xzr6\") pod \"console-76959fb9f4-hq6kg\" (UID: \"264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4\") " pod="openshift-console/console-76959fb9f4-hq6kg"
Jul 02 12:41:33.882497 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:33.882466    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76959fb9f4-hq6kg"
Jul 02 12:41:34.032219 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:34.032189    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76959fb9f4-hq6kg"]
Jul 02 12:41:34.034032 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:41:34.034007    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod264b0dbb_e9a9_4424_8ae4_2f66c9dc4ae4.slice/crio-a5b9e3cce7f44d29bc3a5f1665c85b4e4402e8620c8b41894ca2df8167934bd6 WatchSource:0}: Error finding container a5b9e3cce7f44d29bc3a5f1665c85b4e4402e8620c8b41894ca2df8167934bd6: Status 404 returned error can't find the container with id a5b9e3cce7f44d29bc3a5f1665c85b4e4402e8620c8b41894ca2df8167934bd6
Jul 02 12:41:34.820318 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:34.820281    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76959fb9f4-hq6kg" event={"ID":"264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4","Type":"ContainerStarted","Data":"0f8f795d22343aaff1d43fe0706633ee735a840b89f16fdcefc7ad2430de7725"}
Jul 02 12:41:34.820503 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:34.820336    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76959fb9f4-hq6kg" event={"ID":"264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4","Type":"ContainerStarted","Data":"a5b9e3cce7f44d29bc3a5f1665c85b4e4402e8620c8b41894ca2df8167934bd6"}
Jul 02 12:41:43.882906 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:43.882865    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76959fb9f4-hq6kg"
Jul 02 12:41:43.883404 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:43.882955    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76959fb9f4-hq6kg"
Jul 02 12:41:43.887786 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:43.887766    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-76959fb9f4-hq6kg"
Jul 02 12:41:43.908368 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:43.908325    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-76959fb9f4-hq6kg" podStartSLOduration=10.908310799 podStartE2EDuration="10.908310799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:41:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:41:34.851328299 +0000 UTC m=+183.458791322" watchObservedRunningTime="2026-07-02 12:41:43.908310799 +0000 UTC m=+192.515773822"
Jul 02 12:41:44.865485 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:44.865456    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76959fb9f4-hq6kg"
Jul 02 12:41:44.914284 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:41:44.914245    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-87d995f48-mb6kg"]
Jul 02 12:42:09.935753 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:09.935707    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-87d995f48-mb6kg" podUID="01063732-7971-49d1-873a-bd89ecaf6064" containerName="console" containerID="cri-o://91ea11514afffa0931f3d1a4e4a38dd2bcb4df87d7a5cd3014500e183f56f1bf" gracePeriod=15
Jul 02 12:42:10.183147 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:10.183121    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-87d995f48-mb6kg"
Jul 02 12:42:10.289525 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:10.289445    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01063732-7971-49d1-873a-bd89ecaf6064-console-serving-cert\") pod \"01063732-7971-49d1-873a-bd89ecaf6064\" (UID: \"01063732-7971-49d1-873a-bd89ecaf6064\") "
Jul 02 12:42:10.289525 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:10.289506    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01063732-7971-49d1-873a-bd89ecaf6064-console-config\") pod \"01063732-7971-49d1-873a-bd89ecaf6064\" (UID: \"01063732-7971-49d1-873a-bd89ecaf6064\") "
Jul 02 12:42:10.289742 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:10.289541    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01063732-7971-49d1-873a-bd89ecaf6064-trusted-ca-bundle\") pod \"01063732-7971-49d1-873a-bd89ecaf6064\" (UID: \"01063732-7971-49d1-873a-bd89ecaf6064\") "
Jul 02 12:42:10.289742 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:10.289570    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01063732-7971-49d1-873a-bd89ecaf6064-service-ca\") pod \"01063732-7971-49d1-873a-bd89ecaf6064\" (UID: \"01063732-7971-49d1-873a-bd89ecaf6064\") "
Jul 02 12:42:10.289742 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:10.289616    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01063732-7971-49d1-873a-bd89ecaf6064-oauth-serving-cert\") pod \"01063732-7971-49d1-873a-bd89ecaf6064\" (UID: \"01063732-7971-49d1-873a-bd89ecaf6064\") "
Jul 02 12:42:10.289742 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:10.289652    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01063732-7971-49d1-873a-bd89ecaf6064-console-oauth-config\") pod \"01063732-7971-49d1-873a-bd89ecaf6064\" (UID: \"01063732-7971-49d1-873a-bd89ecaf6064\") "
Jul 02 12:42:10.289742 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:10.289670    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4649\" (UniqueName: \"kubernetes.io/projected/01063732-7971-49d1-873a-bd89ecaf6064-kube-api-access-g4649\") pod \"01063732-7971-49d1-873a-bd89ecaf6064\" (UID: \"01063732-7971-49d1-873a-bd89ecaf6064\") "
Jul 02 12:42:10.290026 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:10.289961    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01063732-7971-49d1-873a-bd89ecaf6064-console-config" (OuterVolumeSpecName: "console-config") pod "01063732-7971-49d1-873a-bd89ecaf6064" (UID: "01063732-7971-49d1-873a-bd89ecaf6064"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:42:10.290116 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:10.290093    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01063732-7971-49d1-873a-bd89ecaf6064-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "01063732-7971-49d1-873a-bd89ecaf6064" (UID: "01063732-7971-49d1-873a-bd89ecaf6064"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:42:10.290187 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:10.290118    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01063732-7971-49d1-873a-bd89ecaf6064-service-ca" (OuterVolumeSpecName: "service-ca") pod "01063732-7971-49d1-873a-bd89ecaf6064" (UID: "01063732-7971-49d1-873a-bd89ecaf6064"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:42:10.290187 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:10.290122    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01063732-7971-49d1-873a-bd89ecaf6064-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "01063732-7971-49d1-873a-bd89ecaf6064" (UID: "01063732-7971-49d1-873a-bd89ecaf6064"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:42:10.291700 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:10.291664    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01063732-7971-49d1-873a-bd89ecaf6064-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "01063732-7971-49d1-873a-bd89ecaf6064" (UID: "01063732-7971-49d1-873a-bd89ecaf6064"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 12:42:10.291806 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:10.291762    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01063732-7971-49d1-873a-bd89ecaf6064-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "01063732-7971-49d1-873a-bd89ecaf6064" (UID: "01063732-7971-49d1-873a-bd89ecaf6064"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 12:42:10.291806 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:10.291779    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01063732-7971-49d1-873a-bd89ecaf6064-kube-api-access-g4649" (OuterVolumeSpecName: "kube-api-access-g4649") pod "01063732-7971-49d1-873a-bd89ecaf6064" (UID: "01063732-7971-49d1-873a-bd89ecaf6064"). InnerVolumeSpecName "kube-api-access-g4649". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 12:42:10.391313 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:10.391282    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01063732-7971-49d1-873a-bd89ecaf6064-trusted-ca-bundle\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:42:10.391313 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:10.391309    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01063732-7971-49d1-873a-bd89ecaf6064-service-ca\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:42:10.391313 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:10.391319    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01063732-7971-49d1-873a-bd89ecaf6064-oauth-serving-cert\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:42:10.391528 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:10.391327    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01063732-7971-49d1-873a-bd89ecaf6064-console-oauth-config\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:42:10.391528 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:10.391338    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4649\" (UniqueName: \"kubernetes.io/projected/01063732-7971-49d1-873a-bd89ecaf6064-kube-api-access-g4649\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:42:10.391528 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:10.391347    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01063732-7971-49d1-873a-bd89ecaf6064-console-serving-cert\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:42:10.391528 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:10.391355    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01063732-7971-49d1-873a-bd89ecaf6064-console-config\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:42:10.968249 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:10.968218    2576 generic.go:356] "Generic (PLEG): container finished" podID="01063732-7971-49d1-873a-bd89ecaf6064" containerID="91ea11514afffa0931f3d1a4e4a38dd2bcb4df87d7a5cd3014500e183f56f1bf" exitCode=0
Jul 02 12:42:10.968642 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:10.968267    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-87d995f48-mb6kg"
Jul 02 12:42:10.968642 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:10.968294    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-87d995f48-mb6kg" event={"ID":"01063732-7971-49d1-873a-bd89ecaf6064","Type":"ContainerDied","Data":"91ea11514afffa0931f3d1a4e4a38dd2bcb4df87d7a5cd3014500e183f56f1bf"}
Jul 02 12:42:10.968642 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:10.968333    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-87d995f48-mb6kg" event={"ID":"01063732-7971-49d1-873a-bd89ecaf6064","Type":"ContainerDied","Data":"c8f5327f3bb6651f1dc8c6fca0846fb73a22c4191306e5d8d103b8a635d26ac9"}
Jul 02 12:42:10.968642 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:10.968351    2576 scope.go:117] "RemoveContainer" containerID="91ea11514afffa0931f3d1a4e4a38dd2bcb4df87d7a5cd3014500e183f56f1bf"
Jul 02 12:42:10.978595 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:10.978575    2576 scope.go:117] "RemoveContainer" containerID="91ea11514afffa0931f3d1a4e4a38dd2bcb4df87d7a5cd3014500e183f56f1bf"
Jul 02 12:42:10.978871 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:42:10.978848    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91ea11514afffa0931f3d1a4e4a38dd2bcb4df87d7a5cd3014500e183f56f1bf\": container with ID starting with 91ea11514afffa0931f3d1a4e4a38dd2bcb4df87d7a5cd3014500e183f56f1bf not found: ID does not exist" containerID="91ea11514afffa0931f3d1a4e4a38dd2bcb4df87d7a5cd3014500e183f56f1bf"
Jul 02 12:42:10.978956 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:10.978880    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91ea11514afffa0931f3d1a4e4a38dd2bcb4df87d7a5cd3014500e183f56f1bf"} err="failed to get container status \"91ea11514afffa0931f3d1a4e4a38dd2bcb4df87d7a5cd3014500e183f56f1bf\": rpc error: code = NotFound desc = could not find container \"91ea11514afffa0931f3d1a4e4a38dd2bcb4df87d7a5cd3014500e183f56f1bf\": container with ID starting with 91ea11514afffa0931f3d1a4e4a38dd2bcb4df87d7a5cd3014500e183f56f1bf not found: ID does not exist"
Jul 02 12:42:10.987099 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:10.987073    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-87d995f48-mb6kg"]
Jul 02 12:42:10.993431 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:10.993408    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-87d995f48-mb6kg"]
Jul 02 12:42:12.862146 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:12.862105    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01063732-7971-49d1-873a-bd89ecaf6064" path="/var/lib/kubelet/pods/01063732-7971-49d1-873a-bd89ecaf6064/volumes"
Jul 02 12:42:41.629695 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:41.629664    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-c456986f6-4nswh"]
Jul 02 12:42:41.630103 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:41.629998    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01063732-7971-49d1-873a-bd89ecaf6064" containerName="console"
Jul 02 12:42:41.630103 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:41.630010    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="01063732-7971-49d1-873a-bd89ecaf6064" containerName="console"
Jul 02 12:42:41.630103 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:41.630082    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="01063732-7971-49d1-873a-bd89ecaf6064" containerName="console"
Jul 02 12:42:41.634182 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:41.634161    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c456986f6-4nswh"
Jul 02 12:42:41.645955 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:41.645928    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c456986f6-4nswh"]
Jul 02 12:42:41.728225 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:41.728195    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85db6f6b-7fe6-455f-b20b-587228f5042d-trusted-ca-bundle\") pod \"console-c456986f6-4nswh\" (UID: \"85db6f6b-7fe6-455f-b20b-587228f5042d\") " pod="openshift-console/console-c456986f6-4nswh"
Jul 02 12:42:41.728225 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:41.728229    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/85db6f6b-7fe6-455f-b20b-587228f5042d-console-oauth-config\") pod \"console-c456986f6-4nswh\" (UID: \"85db6f6b-7fe6-455f-b20b-587228f5042d\") " pod="openshift-console/console-c456986f6-4nswh"
Jul 02 12:42:41.728421 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:41.728258    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/85db6f6b-7fe6-455f-b20b-587228f5042d-console-config\") pod \"console-c456986f6-4nswh\" (UID: \"85db6f6b-7fe6-455f-b20b-587228f5042d\") " pod="openshift-console/console-c456986f6-4nswh"
Jul 02 12:42:41.728421 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:41.728329    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f79dw\" (UniqueName: \"kubernetes.io/projected/85db6f6b-7fe6-455f-b20b-587228f5042d-kube-api-access-f79dw\") pod \"console-c456986f6-4nswh\" (UID: \"85db6f6b-7fe6-455f-b20b-587228f5042d\") " pod="openshift-console/console-c456986f6-4nswh"
Jul 02 12:42:41.728421 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:41.728391    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/85db6f6b-7fe6-455f-b20b-587228f5042d-console-serving-cert\") pod \"console-c456986f6-4nswh\" (UID: \"85db6f6b-7fe6-455f-b20b-587228f5042d\") " pod="openshift-console/console-c456986f6-4nswh"
Jul 02 12:42:41.728421 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:41.728419    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/85db6f6b-7fe6-455f-b20b-587228f5042d-oauth-serving-cert\") pod \"console-c456986f6-4nswh\" (UID: \"85db6f6b-7fe6-455f-b20b-587228f5042d\") " pod="openshift-console/console-c456986f6-4nswh"
Jul 02 12:42:41.728544 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:41.728447    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/85db6f6b-7fe6-455f-b20b-587228f5042d-service-ca\") pod \"console-c456986f6-4nswh\" (UID: \"85db6f6b-7fe6-455f-b20b-587228f5042d\") " pod="openshift-console/console-c456986f6-4nswh"
Jul 02 12:42:41.829620 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:41.829579    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/85db6f6b-7fe6-455f-b20b-587228f5042d-console-serving-cert\") pod \"console-c456986f6-4nswh\" (UID: \"85db6f6b-7fe6-455f-b20b-587228f5042d\") " pod="openshift-console/console-c456986f6-4nswh"
Jul 02 12:42:41.829620 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:41.829623    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/85db6f6b-7fe6-455f-b20b-587228f5042d-oauth-serving-cert\") pod \"console-c456986f6-4nswh\" (UID: \"85db6f6b-7fe6-455f-b20b-587228f5042d\") " pod="openshift-console/console-c456986f6-4nswh"
Jul 02 12:42:41.829977 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:41.829661    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/85db6f6b-7fe6-455f-b20b-587228f5042d-service-ca\") pod \"console-c456986f6-4nswh\" (UID: \"85db6f6b-7fe6-455f-b20b-587228f5042d\") " pod="openshift-console/console-c456986f6-4nswh"
Jul 02 12:42:41.829977 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:41.829732    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85db6f6b-7fe6-455f-b20b-587228f5042d-trusted-ca-bundle\") pod \"console-c456986f6-4nswh\" (UID: \"85db6f6b-7fe6-455f-b20b-587228f5042d\") " pod="openshift-console/console-c456986f6-4nswh"
Jul 02 12:42:41.829977 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:41.829759    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/85db6f6b-7fe6-455f-b20b-587228f5042d-console-oauth-config\") pod \"console-c456986f6-4nswh\" (UID: \"85db6f6b-7fe6-455f-b20b-587228f5042d\") " pod="openshift-console/console-c456986f6-4nswh"
Jul 02 12:42:41.829977 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:41.829790    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/85db6f6b-7fe6-455f-b20b-587228f5042d-console-config\") pod \"console-c456986f6-4nswh\" (UID: \"85db6f6b-7fe6-455f-b20b-587228f5042d\") " pod="openshift-console/console-c456986f6-4nswh"
Jul 02 12:42:41.829977 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:41.829831    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f79dw\" (UniqueName: \"kubernetes.io/projected/85db6f6b-7fe6-455f-b20b-587228f5042d-kube-api-access-f79dw\") pod \"console-c456986f6-4nswh\" (UID: \"85db6f6b-7fe6-455f-b20b-587228f5042d\") " pod="openshift-console/console-c456986f6-4nswh"
Jul 02 12:42:41.830526 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:41.830500    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/85db6f6b-7fe6-455f-b20b-587228f5042d-oauth-serving-cert\") pod \"console-c456986f6-4nswh\" (UID: \"85db6f6b-7fe6-455f-b20b-587228f5042d\") " pod="openshift-console/console-c456986f6-4nswh"
Jul 02 12:42:41.830526 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:41.830516    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/85db6f6b-7fe6-455f-b20b-587228f5042d-service-ca\") pod \"console-c456986f6-4nswh\" (UID: \"85db6f6b-7fe6-455f-b20b-587228f5042d\") " pod="openshift-console/console-c456986f6-4nswh"
Jul 02 12:42:41.830658 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:41.830570    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/85db6f6b-7fe6-455f-b20b-587228f5042d-console-config\") pod \"console-c456986f6-4nswh\" (UID: \"85db6f6b-7fe6-455f-b20b-587228f5042d\") " pod="openshift-console/console-c456986f6-4nswh"
Jul 02 12:42:41.830658 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:41.830612    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85db6f6b-7fe6-455f-b20b-587228f5042d-trusted-ca-bundle\") pod \"console-c456986f6-4nswh\" (UID: \"85db6f6b-7fe6-455f-b20b-587228f5042d\") " pod="openshift-console/console-c456986f6-4nswh"
Jul 02 12:42:41.832141 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:41.832118    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/85db6f6b-7fe6-455f-b20b-587228f5042d-console-serving-cert\") pod \"console-c456986f6-4nswh\" (UID: \"85db6f6b-7fe6-455f-b20b-587228f5042d\") " pod="openshift-console/console-c456986f6-4nswh"
Jul 02 12:42:41.832221 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:41.832138    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/85db6f6b-7fe6-455f-b20b-587228f5042d-console-oauth-config\") pod \"console-c456986f6-4nswh\" (UID: \"85db6f6b-7fe6-455f-b20b-587228f5042d\") " pod="openshift-console/console-c456986f6-4nswh"
Jul 02 12:42:41.838545 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:41.838522    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f79dw\" (UniqueName: \"kubernetes.io/projected/85db6f6b-7fe6-455f-b20b-587228f5042d-kube-api-access-f79dw\") pod \"console-c456986f6-4nswh\" (UID: \"85db6f6b-7fe6-455f-b20b-587228f5042d\") " pod="openshift-console/console-c456986f6-4nswh"
Jul 02 12:42:41.943055 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:41.943022    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c456986f6-4nswh"
Jul 02 12:42:42.075061 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:42.075039    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c456986f6-4nswh"]
Jul 02 12:42:42.077840 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:42:42.077810    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85db6f6b_7fe6_455f_b20b_587228f5042d.slice/crio-41cfdee5753829a1c4a50b75ff25ebb07bf3b5f898cdadcefc125b3232887985 WatchSource:0}: Error finding container 41cfdee5753829a1c4a50b75ff25ebb07bf3b5f898cdadcefc125b3232887985: Status 404 returned error can't find the container with id 41cfdee5753829a1c4a50b75ff25ebb07bf3b5f898cdadcefc125b3232887985
Jul 02 12:42:42.095071 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:42.095046    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c456986f6-4nswh" event={"ID":"85db6f6b-7fe6-455f-b20b-587228f5042d","Type":"ContainerStarted","Data":"41cfdee5753829a1c4a50b75ff25ebb07bf3b5f898cdadcefc125b3232887985"}
Jul 02 12:42:43.100314 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:43.100273    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c456986f6-4nswh" event={"ID":"85db6f6b-7fe6-455f-b20b-587228f5042d","Type":"ContainerStarted","Data":"f797f3ce930a6b0ced975d1d272c3384f0c2a43ba592de1e77c67c6377d0afa7"}
Jul 02 12:42:43.119197 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:43.119151    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-c456986f6-4nswh" podStartSLOduration=2.119137617 podStartE2EDuration="2.119137617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:42:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:42:43.118174071 +0000 UTC m=+251.725637119" watchObservedRunningTime="2026-07-02 12:42:43.119137617 +0000 UTC m=+251.726600640"
Jul 02 12:42:51.944098 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:51.944061    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c456986f6-4nswh"
Jul 02 12:42:51.944495 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:51.944155    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c456986f6-4nswh"
Jul 02 12:42:51.948863 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:51.948842    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c456986f6-4nswh"
Jul 02 12:42:52.141122 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:52.141089    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c456986f6-4nswh"
Jul 02 12:42:52.190389 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:42:52.190343    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76959fb9f4-hq6kg"]
Jul 02 12:43:17.218973 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:43:17.218883    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-76959fb9f4-hq6kg" podUID="264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4" containerName="console" containerID="cri-o://0f8f795d22343aaff1d43fe0706633ee735a840b89f16fdcefc7ad2430de7725" gracePeriod=15
Jul 02 12:43:17.459814 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:43:17.459782    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76959fb9f4-hq6kg"
Jul 02 12:43:17.635249 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:43:17.635168    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4-console-config\") pod \"264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4\" (UID: \"264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4\") "
Jul 02 12:43:17.635249 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:43:17.635218    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4-console-oauth-config\") pod \"264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4\" (UID: \"264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4\") "
Jul 02 12:43:17.635418 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:43:17.635256    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4-console-serving-cert\") pod \"264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4\" (UID: \"264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4\") "
Jul 02 12:43:17.635418 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:43:17.635300    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4-service-ca\") pod \"264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4\" (UID: \"264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4\") "
Jul 02 12:43:17.635418 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:43:17.635333    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4-trusted-ca-bundle\") pod \"264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4\" (UID: \"264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4\") "
Jul 02 12:43:17.635418 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:43:17.635402    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xzr6\" (UniqueName: \"kubernetes.io/projected/264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4-kube-api-access-6xzr6\") pod \"264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4\" (UID: \"264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4\") "
Jul 02 12:43:17.635607 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:43:17.635429    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4-oauth-serving-cert\") pod \"264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4\" (UID: \"264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4\") "
Jul 02 12:43:17.635730 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:43:17.635697    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4-console-config" (OuterVolumeSpecName: "console-config") pod "264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4" (UID: "264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:43:17.635849 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:43:17.635735    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4-service-ca" (OuterVolumeSpecName: "service-ca") pod "264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4" (UID: "264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:43:17.636008 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:43:17.635981    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4" (UID: "264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:43:17.636071 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:43:17.636028    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4" (UID: "264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:43:17.637495 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:43:17.637471    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4-kube-api-access-6xzr6" (OuterVolumeSpecName: "kube-api-access-6xzr6") pod "264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4" (UID: "264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4"). InnerVolumeSpecName "kube-api-access-6xzr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 12:43:17.637865 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:43:17.637846    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4" (UID: "264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 12:43:17.637865 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:43:17.637859    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4" (UID: "264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 12:43:17.736428 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:43:17.736398    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4-console-config\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:43:17.736428 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:43:17.736422    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4-console-oauth-config\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:43:17.736428 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:43:17.736433    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4-console-serving-cert\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:43:17.736615 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:43:17.736443    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4-service-ca\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:43:17.736615 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:43:17.736451    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4-trusted-ca-bundle\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:43:17.736615 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:43:17.736461    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xzr6\" (UniqueName: \"kubernetes.io/projected/264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4-kube-api-access-6xzr6\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:43:17.736615 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:43:17.736469    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4-oauth-serving-cert\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:43:18.244631 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:43:18.244596    2576 generic.go:356] "Generic (PLEG): container finished" podID="264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4" containerID="0f8f795d22343aaff1d43fe0706633ee735a840b89f16fdcefc7ad2430de7725" exitCode=0
Jul 02 12:43:18.245089 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:43:18.244651    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76959fb9f4-hq6kg"
Jul 02 12:43:18.245089 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:43:18.244673    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76959fb9f4-hq6kg" event={"ID":"264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4","Type":"ContainerDied","Data":"0f8f795d22343aaff1d43fe0706633ee735a840b89f16fdcefc7ad2430de7725"}
Jul 02 12:43:18.245089 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:43:18.244723    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76959fb9f4-hq6kg" event={"ID":"264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4","Type":"ContainerDied","Data":"a5b9e3cce7f44d29bc3a5f1665c85b4e4402e8620c8b41894ca2df8167934bd6"}
Jul 02 12:43:18.245089 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:43:18.244743    2576 scope.go:117] "RemoveContainer" containerID="0f8f795d22343aaff1d43fe0706633ee735a840b89f16fdcefc7ad2430de7725"
Jul 02 12:43:18.253443 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:43:18.253427    2576 scope.go:117] "RemoveContainer" containerID="0f8f795d22343aaff1d43fe0706633ee735a840b89f16fdcefc7ad2430de7725"
Jul 02 12:43:18.253715 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:43:18.253693    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f8f795d22343aaff1d43fe0706633ee735a840b89f16fdcefc7ad2430de7725\": container with ID starting with 0f8f795d22343aaff1d43fe0706633ee735a840b89f16fdcefc7ad2430de7725 not found: ID does not exist" containerID="0f8f795d22343aaff1d43fe0706633ee735a840b89f16fdcefc7ad2430de7725"
Jul 02 12:43:18.253764 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:43:18.253725    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f8f795d22343aaff1d43fe0706633ee735a840b89f16fdcefc7ad2430de7725"} err="failed to get container status \"0f8f795d22343aaff1d43fe0706633ee735a840b89f16fdcefc7ad2430de7725\": rpc error: code = NotFound desc = could not find container \"0f8f795d22343aaff1d43fe0706633ee735a840b89f16fdcefc7ad2430de7725\": container with ID starting with 0f8f795d22343aaff1d43fe0706633ee735a840b89f16fdcefc7ad2430de7725 not found: ID does not exist"
Jul 02 12:43:18.266662 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:43:18.266639    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76959fb9f4-hq6kg"]
Jul 02 12:43:18.269736 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:43:18.269713    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-76959fb9f4-hq6kg"]
Jul 02 12:43:18.866304 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:43:18.866265    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4" path="/var/lib/kubelet/pods/264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4/volumes"
Jul 02 12:43:32.182322 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:43:32.182297    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2c7ks_116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a/ovn-acl-logging/0.log"
Jul 02 12:43:32.186736 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:43:32.186717    2576 kubelet.go:1648] "Image garbage collection succeeded"
Jul 02 12:43:32.807103 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:43:32.807078    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2c7ks_116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a/ovn-acl-logging/0.log"
Jul 02 12:44:35.789277 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:44:35.789202    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rppdp"]
Jul 02 12:44:35.789688 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:44:35.789525    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4" containerName="console"
Jul 02 12:44:35.789688 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:44:35.789536    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4" containerName="console"
Jul 02 12:44:35.789688 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:44:35.789594    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="264b0dbb-e9a9-4424-8ae4-2f66c9dc4ae4" containerName="console"
Jul 02 12:44:35.792764 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:44:35.792749    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rppdp"
Jul 02 12:44:35.795299 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:44:35.795275    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 02 12:44:35.795299 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:44:35.795304    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 02 12:44:35.796243 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:44:35.796226    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-cs8zk\""
Jul 02 12:44:35.801548 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:44:35.801524    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rppdp"]
Jul 02 12:44:35.957611 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:44:35.957569    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2ec3a57-2e43-42cc-9f58-a0d83d678a5a-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rppdp\" (UID: \"f2ec3a57-2e43-42cc-9f58-a0d83d678a5a\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rppdp"
Jul 02 12:44:35.957800 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:44:35.957631    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kggmz\" (UniqueName: \"kubernetes.io/projected/f2ec3a57-2e43-42cc-9f58-a0d83d678a5a-kube-api-access-kggmz\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rppdp\" (UID: \"f2ec3a57-2e43-42cc-9f58-a0d83d678a5a\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rppdp"
Jul 02 12:44:35.957800 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:44:35.957712    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2ec3a57-2e43-42cc-9f58-a0d83d678a5a-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rppdp\" (UID: \"f2ec3a57-2e43-42cc-9f58-a0d83d678a5a\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rppdp"
Jul 02 12:44:36.059210 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:44:36.059121    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2ec3a57-2e43-42cc-9f58-a0d83d678a5a-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rppdp\" (UID: \"f2ec3a57-2e43-42cc-9f58-a0d83d678a5a\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rppdp"
Jul 02 12:44:36.059210 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:44:36.059179    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kggmz\" (UniqueName: \"kubernetes.io/projected/f2ec3a57-2e43-42cc-9f58-a0d83d678a5a-kube-api-access-kggmz\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rppdp\" (UID: \"f2ec3a57-2e43-42cc-9f58-a0d83d678a5a\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rppdp"
Jul 02 12:44:36.059402 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:44:36.059217    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2ec3a57-2e43-42cc-9f58-a0d83d678a5a-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rppdp\" (UID: \"f2ec3a57-2e43-42cc-9f58-a0d83d678a5a\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rppdp"
Jul 02 12:44:36.059523 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:44:36.059495    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2ec3a57-2e43-42cc-9f58-a0d83d678a5a-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rppdp\" (UID: \"f2ec3a57-2e43-42cc-9f58-a0d83d678a5a\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rppdp"
Jul 02 12:44:36.059562 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:44:36.059524    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2ec3a57-2e43-42cc-9f58-a0d83d678a5a-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rppdp\" (UID: \"f2ec3a57-2e43-42cc-9f58-a0d83d678a5a\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rppdp"
Jul 02 12:44:36.067296 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:44:36.067272    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kggmz\" (UniqueName: \"kubernetes.io/projected/f2ec3a57-2e43-42cc-9f58-a0d83d678a5a-kube-api-access-kggmz\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rppdp\" (UID: \"f2ec3a57-2e43-42cc-9f58-a0d83d678a5a\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rppdp"
Jul 02 12:44:36.102664 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:44:36.102643    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rppdp"
Jul 02 12:44:36.230026 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:44:36.229991    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rppdp"]
Jul 02 12:44:36.232985 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:44:36.232954    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2ec3a57_2e43_42cc_9f58_a0d83d678a5a.slice/crio-5e61574ffcc5ef758a7c50163dfa8f7278d1f97c1b6043aeaf124f491dd10dfe WatchSource:0}: Error finding container 5e61574ffcc5ef758a7c50163dfa8f7278d1f97c1b6043aeaf124f491dd10dfe: Status 404 returned error can't find the container with id 5e61574ffcc5ef758a7c50163dfa8f7278d1f97c1b6043aeaf124f491dd10dfe
Jul 02 12:44:36.234969 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:44:36.234950    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 12:44:36.558477 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:44:36.558442    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rppdp" event={"ID":"f2ec3a57-2e43-42cc-9f58-a0d83d678a5a","Type":"ContainerStarted","Data":"5e61574ffcc5ef758a7c50163dfa8f7278d1f97c1b6043aeaf124f491dd10dfe"}
Jul 02 12:44:48.613668 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:44:48.613635    2576 generic.go:356] "Generic (PLEG): container finished" podID="f2ec3a57-2e43-42cc-9f58-a0d83d678a5a" containerID="607b7633418d3550be23217049894012f327d96c1114438d0e368e6f87c27b4f" exitCode=0
Jul 02 12:44:48.614100 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:44:48.613725    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rppdp" event={"ID":"f2ec3a57-2e43-42cc-9f58-a0d83d678a5a","Type":"ContainerDied","Data":"607b7633418d3550be23217049894012f327d96c1114438d0e368e6f87c27b4f"}
Jul 02 12:44:51.628454 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:44:51.628417    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rppdp" event={"ID":"f2ec3a57-2e43-42cc-9f58-a0d83d678a5a","Type":"ContainerStarted","Data":"a9cf5f1682da3b912e90b2fae187ddc99369c149dea74726b6a67304489ed573"}
Jul 02 12:44:52.634629 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:44:52.634594    2576 generic.go:356] "Generic (PLEG): container finished" podID="f2ec3a57-2e43-42cc-9f58-a0d83d678a5a" containerID="a9cf5f1682da3b912e90b2fae187ddc99369c149dea74726b6a67304489ed573" exitCode=0
Jul 02 12:44:52.634629 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:44:52.634633    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rppdp" event={"ID":"f2ec3a57-2e43-42cc-9f58-a0d83d678a5a","Type":"ContainerDied","Data":"a9cf5f1682da3b912e90b2fae187ddc99369c149dea74726b6a67304489ed573"}
Jul 02 12:44:59.669797 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:44:59.669760    2576 generic.go:356] "Generic (PLEG): container finished" podID="f2ec3a57-2e43-42cc-9f58-a0d83d678a5a" containerID="8e4285e62268075432a2482878436217a464302ea0091180815233017a36b322" exitCode=0
Jul 02 12:44:59.670194 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:44:59.669845    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rppdp" event={"ID":"f2ec3a57-2e43-42cc-9f58-a0d83d678a5a","Type":"ContainerDied","Data":"8e4285e62268075432a2482878436217a464302ea0091180815233017a36b322"}
Jul 02 12:45:00.806340 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:00.806319    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rppdp"
Jul 02 12:45:00.876756 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:00.876729    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2ec3a57-2e43-42cc-9f58-a0d83d678a5a-util\") pod \"f2ec3a57-2e43-42cc-9f58-a0d83d678a5a\" (UID: \"f2ec3a57-2e43-42cc-9f58-a0d83d678a5a\") "
Jul 02 12:45:00.876893 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:00.876783    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2ec3a57-2e43-42cc-9f58-a0d83d678a5a-bundle\") pod \"f2ec3a57-2e43-42cc-9f58-a0d83d678a5a\" (UID: \"f2ec3a57-2e43-42cc-9f58-a0d83d678a5a\") "
Jul 02 12:45:00.876996 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:00.876893    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kggmz\" (UniqueName: \"kubernetes.io/projected/f2ec3a57-2e43-42cc-9f58-a0d83d678a5a-kube-api-access-kggmz\") pod \"f2ec3a57-2e43-42cc-9f58-a0d83d678a5a\" (UID: \"f2ec3a57-2e43-42cc-9f58-a0d83d678a5a\") "
Jul 02 12:45:00.877500 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:00.877472    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ec3a57-2e43-42cc-9f58-a0d83d678a5a-bundle" (OuterVolumeSpecName: "bundle") pod "f2ec3a57-2e43-42cc-9f58-a0d83d678a5a" (UID: "f2ec3a57-2e43-42cc-9f58-a0d83d678a5a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 12:45:00.879139 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:00.879117    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2ec3a57-2e43-42cc-9f58-a0d83d678a5a-kube-api-access-kggmz" (OuterVolumeSpecName: "kube-api-access-kggmz") pod "f2ec3a57-2e43-42cc-9f58-a0d83d678a5a" (UID: "f2ec3a57-2e43-42cc-9f58-a0d83d678a5a"). InnerVolumeSpecName "kube-api-access-kggmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 12:45:00.881901 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:00.881881    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ec3a57-2e43-42cc-9f58-a0d83d678a5a-util" (OuterVolumeSpecName: "util") pod "f2ec3a57-2e43-42cc-9f58-a0d83d678a5a" (UID: "f2ec3a57-2e43-42cc-9f58-a0d83d678a5a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 12:45:00.978353 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:00.978275    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kggmz\" (UniqueName: \"kubernetes.io/projected/f2ec3a57-2e43-42cc-9f58-a0d83d678a5a-kube-api-access-kggmz\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:45:00.978353 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:00.978306    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2ec3a57-2e43-42cc-9f58-a0d83d678a5a-util\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:45:00.978353 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:00.978316    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2ec3a57-2e43-42cc-9f58-a0d83d678a5a-bundle\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:45:01.679723 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:01.679694    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rppdp"
Jul 02 12:45:01.679723 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:01.679717    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rppdp" event={"ID":"f2ec3a57-2e43-42cc-9f58-a0d83d678a5a","Type":"ContainerDied","Data":"5e61574ffcc5ef758a7c50163dfa8f7278d1f97c1b6043aeaf124f491dd10dfe"}
Jul 02 12:45:01.679987 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:01.679761    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e61574ffcc5ef758a7c50163dfa8f7278d1f97c1b6043aeaf124f491dd10dfe"
Jul 02 12:45:14.139831 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:14.139796    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-68bff59c-nf7ch"]
Jul 02 12:45:14.140199 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:14.140164    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2ec3a57-2e43-42cc-9f58-a0d83d678a5a" containerName="util"
Jul 02 12:45:14.140199 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:14.140176    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2ec3a57-2e43-42cc-9f58-a0d83d678a5a" containerName="util"
Jul 02 12:45:14.140199 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:14.140193    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2ec3a57-2e43-42cc-9f58-a0d83d678a5a" containerName="extract"
Jul 02 12:45:14.140199 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:14.140199    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2ec3a57-2e43-42cc-9f58-a0d83d678a5a" containerName="extract"
Jul 02 12:45:14.140328 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:14.140215    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2ec3a57-2e43-42cc-9f58-a0d83d678a5a" containerName="pull"
Jul 02 12:45:14.140328 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:14.140223    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2ec3a57-2e43-42cc-9f58-a0d83d678a5a" containerName="pull"
Jul 02 12:45:14.140328 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:14.140276    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2ec3a57-2e43-42cc-9f58-a0d83d678a5a" containerName="extract"
Jul 02 12:45:14.146077 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:14.146055    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-nf7ch"
Jul 02 12:45:14.149124 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:14.149103    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 02 12:45:14.149192 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:14.149122    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 02 12:45:14.150683 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:14.150667    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 02 12:45:14.150753 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:14.150708    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 02 12:45:14.152396 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:14.152379    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-kgtlp\""
Jul 02 12:45:14.154107 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:14.154083    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 02 12:45:14.165741 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:14.165721    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-nf7ch"]
Jul 02 12:45:14.299566 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:14.299534    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/45f538bc-2f6c-4b99-9608-dc1d59fd5ec2-certificates\") pod \"keda-operator-68bff59c-nf7ch\" (UID: \"45f538bc-2f6c-4b99-9608-dc1d59fd5ec2\") " pod="openshift-keda/keda-operator-68bff59c-nf7ch"
Jul 02 12:45:14.299566 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:14.299567    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/45f538bc-2f6c-4b99-9608-dc1d59fd5ec2-cabundle0\") pod \"keda-operator-68bff59c-nf7ch\" (UID: \"45f538bc-2f6c-4b99-9608-dc1d59fd5ec2\") " pod="openshift-keda/keda-operator-68bff59c-nf7ch"
Jul 02 12:45:14.299766 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:14.299662    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjts5\" (UniqueName: \"kubernetes.io/projected/45f538bc-2f6c-4b99-9608-dc1d59fd5ec2-kube-api-access-tjts5\") pod \"keda-operator-68bff59c-nf7ch\" (UID: \"45f538bc-2f6c-4b99-9608-dc1d59fd5ec2\") " pod="openshift-keda/keda-operator-68bff59c-nf7ch"
Jul 02 12:45:14.400988 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:14.400960    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/45f538bc-2f6c-4b99-9608-dc1d59fd5ec2-certificates\") pod \"keda-operator-68bff59c-nf7ch\" (UID: \"45f538bc-2f6c-4b99-9608-dc1d59fd5ec2\") " pod="openshift-keda/keda-operator-68bff59c-nf7ch"
Jul 02 12:45:14.400988 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:14.400993    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/45f538bc-2f6c-4b99-9608-dc1d59fd5ec2-cabundle0\") pod \"keda-operator-68bff59c-nf7ch\" (UID: \"45f538bc-2f6c-4b99-9608-dc1d59fd5ec2\") " pod="openshift-keda/keda-operator-68bff59c-nf7ch"
Jul 02 12:45:14.401181 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:14.401050    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjts5\" (UniqueName: \"kubernetes.io/projected/45f538bc-2f6c-4b99-9608-dc1d59fd5ec2-kube-api-access-tjts5\") pod \"keda-operator-68bff59c-nf7ch\" (UID: \"45f538bc-2f6c-4b99-9608-dc1d59fd5ec2\") " pod="openshift-keda/keda-operator-68bff59c-nf7ch"
Jul 02 12:45:14.401181 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:45:14.401128    2576 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Jul 02 12:45:14.401181 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:45:14.401151    2576 secret.go:281] references non-existent secret key: ca.crt
Jul 02 12:45:14.401181 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:45:14.401161    2576 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 02 12:45:14.401181 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:45:14.401175    2576 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-68bff59c-nf7ch: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 02 12:45:14.401330 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:45:14.401247    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45f538bc-2f6c-4b99-9608-dc1d59fd5ec2-certificates podName:45f538bc-2f6c-4b99-9608-dc1d59fd5ec2 nodeName:}" failed. No retries permitted until 2026-07-02 12:45:14.901227701 +0000 UTC m=+403.508690719 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/45f538bc-2f6c-4b99-9608-dc1d59fd5ec2-certificates") pod "keda-operator-68bff59c-nf7ch" (UID: "45f538bc-2f6c-4b99-9608-dc1d59fd5ec2") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 02 12:45:14.401688 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:14.401668    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/45f538bc-2f6c-4b99-9608-dc1d59fd5ec2-cabundle0\") pod \"keda-operator-68bff59c-nf7ch\" (UID: \"45f538bc-2f6c-4b99-9608-dc1d59fd5ec2\") " pod="openshift-keda/keda-operator-68bff59c-nf7ch"
Jul 02 12:45:14.411882 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:14.411861    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjts5\" (UniqueName: \"kubernetes.io/projected/45f538bc-2f6c-4b99-9608-dc1d59fd5ec2-kube-api-access-tjts5\") pod \"keda-operator-68bff59c-nf7ch\" (UID: \"45f538bc-2f6c-4b99-9608-dc1d59fd5ec2\") " pod="openshift-keda/keda-operator-68bff59c-nf7ch"
Jul 02 12:45:14.493350 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:14.493320    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-zk8hm"]
Jul 02 12:45:14.496722 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:14.496707    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-zk8hm"
Jul 02 12:45:14.499319 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:14.499297    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 02 12:45:14.507541 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:14.507519    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-zk8hm"]
Jul 02 12:45:14.602410 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:14.602384    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qt7m\" (UniqueName: \"kubernetes.io/projected/59f85457-f6ea-4a12-a403-beebd278f938-kube-api-access-8qt7m\") pod \"keda-metrics-apiserver-559f75f947-zk8hm\" (UID: \"59f85457-f6ea-4a12-a403-beebd278f938\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-zk8hm"
Jul 02 12:45:14.602580 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:14.602445    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/59f85457-f6ea-4a12-a403-beebd278f938-certificates\") pod \"keda-metrics-apiserver-559f75f947-zk8hm\" (UID: \"59f85457-f6ea-4a12-a403-beebd278f938\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-zk8hm"
Jul 02 12:45:14.602580 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:14.602492    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/59f85457-f6ea-4a12-a403-beebd278f938-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-zk8hm\" (UID: \"59f85457-f6ea-4a12-a403-beebd278f938\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-zk8hm"
Jul 02 12:45:14.703682 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:14.703593    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/59f85457-f6ea-4a12-a403-beebd278f938-certificates\") pod \"keda-metrics-apiserver-559f75f947-zk8hm\" (UID: \"59f85457-f6ea-4a12-a403-beebd278f938\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-zk8hm"
Jul 02 12:45:14.703682 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:14.703638    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/59f85457-f6ea-4a12-a403-beebd278f938-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-zk8hm\" (UID: \"59f85457-f6ea-4a12-a403-beebd278f938\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-zk8hm"
Jul 02 12:45:14.703927 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:14.703714    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qt7m\" (UniqueName: \"kubernetes.io/projected/59f85457-f6ea-4a12-a403-beebd278f938-kube-api-access-8qt7m\") pod \"keda-metrics-apiserver-559f75f947-zk8hm\" (UID: \"59f85457-f6ea-4a12-a403-beebd278f938\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-zk8hm"
Jul 02 12:45:14.703927 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:45:14.703733    2576 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Jul 02 12:45:14.703927 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:45:14.703756    2576 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-559f75f947-zk8hm: secret "keda-metrics-apiserver-certs" not found
Jul 02 12:45:14.703927 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:45:14.703813    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59f85457-f6ea-4a12-a403-beebd278f938-certificates podName:59f85457-f6ea-4a12-a403-beebd278f938 nodeName:}" failed. No retries permitted until 2026-07-02 12:45:15.203798646 +0000 UTC m=+403.811261647 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/59f85457-f6ea-4a12-a403-beebd278f938-certificates") pod "keda-metrics-apiserver-559f75f947-zk8hm" (UID: "59f85457-f6ea-4a12-a403-beebd278f938") : secret "keda-metrics-apiserver-certs" not found
Jul 02 12:45:14.704095 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:14.704037    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/59f85457-f6ea-4a12-a403-beebd278f938-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-zk8hm\" (UID: \"59f85457-f6ea-4a12-a403-beebd278f938\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-zk8hm"
Jul 02 12:45:14.713373 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:14.713347    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qt7m\" (UniqueName: \"kubernetes.io/projected/59f85457-f6ea-4a12-a403-beebd278f938-kube-api-access-8qt7m\") pod \"keda-metrics-apiserver-559f75f947-zk8hm\" (UID: \"59f85457-f6ea-4a12-a403-beebd278f938\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-zk8hm"
Jul 02 12:45:14.831950 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:14.831906    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-x462p"]
Jul 02 12:45:14.835349 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:14.835328    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-x462p"
Jul 02 12:45:14.838438 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:14.838416    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 02 12:45:14.844713 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:14.844683    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-x462p"]
Jul 02 12:45:14.905652 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:14.905615    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/45f538bc-2f6c-4b99-9608-dc1d59fd5ec2-certificates\") pod \"keda-operator-68bff59c-nf7ch\" (UID: \"45f538bc-2f6c-4b99-9608-dc1d59fd5ec2\") " pod="openshift-keda/keda-operator-68bff59c-nf7ch"
Jul 02 12:45:14.908539 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:14.908507    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/45f538bc-2f6c-4b99-9608-dc1d59fd5ec2-certificates\") pod \"keda-operator-68bff59c-nf7ch\" (UID: \"45f538bc-2f6c-4b99-9608-dc1d59fd5ec2\") " pod="openshift-keda/keda-operator-68bff59c-nf7ch"
Jul 02 12:45:15.007366 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:15.007282    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d790b920-d977-4010-80cc-6a984743deba-certificates\") pod \"keda-admission-5fbd5c4644-x462p\" (UID: \"d790b920-d977-4010-80cc-6a984743deba\") " pod="openshift-keda/keda-admission-5fbd5c4644-x462p"
Jul 02 12:45:15.007509 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:15.007444    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbhhm\" (UniqueName: \"kubernetes.io/projected/d790b920-d977-4010-80cc-6a984743deba-kube-api-access-qbhhm\") pod \"keda-admission-5fbd5c4644-x462p\" (UID: \"d790b920-d977-4010-80cc-6a984743deba\") " pod="openshift-keda/keda-admission-5fbd5c4644-x462p"
Jul 02 12:45:15.056174 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:15.056129    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-nf7ch"
Jul 02 12:45:15.109085 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:15.109054    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d790b920-d977-4010-80cc-6a984743deba-certificates\") pod \"keda-admission-5fbd5c4644-x462p\" (UID: \"d790b920-d977-4010-80cc-6a984743deba\") " pod="openshift-keda/keda-admission-5fbd5c4644-x462p"
Jul 02 12:45:15.109251 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:15.109174    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbhhm\" (UniqueName: \"kubernetes.io/projected/d790b920-d977-4010-80cc-6a984743deba-kube-api-access-qbhhm\") pod \"keda-admission-5fbd5c4644-x462p\" (UID: \"d790b920-d977-4010-80cc-6a984743deba\") " pod="openshift-keda/keda-admission-5fbd5c4644-x462p"
Jul 02 12:45:15.111579 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:15.111553    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d790b920-d977-4010-80cc-6a984743deba-certificates\") pod \"keda-admission-5fbd5c4644-x462p\" (UID: \"d790b920-d977-4010-80cc-6a984743deba\") " pod="openshift-keda/keda-admission-5fbd5c4644-x462p"
Jul 02 12:45:15.122237 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:15.122211    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbhhm\" (UniqueName: \"kubernetes.io/projected/d790b920-d977-4010-80cc-6a984743deba-kube-api-access-qbhhm\") pod \"keda-admission-5fbd5c4644-x462p\" (UID: \"d790b920-d977-4010-80cc-6a984743deba\") " pod="openshift-keda/keda-admission-5fbd5c4644-x462p"
Jul 02 12:45:15.146085 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:15.145824    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-x462p"
Jul 02 12:45:15.210535 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:15.210219    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/59f85457-f6ea-4a12-a403-beebd278f938-certificates\") pod \"keda-metrics-apiserver-559f75f947-zk8hm\" (UID: \"59f85457-f6ea-4a12-a403-beebd278f938\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-zk8hm"
Jul 02 12:45:15.213181 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:15.213021    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-nf7ch"]
Jul 02 12:45:15.213418 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:15.213373    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/59f85457-f6ea-4a12-a403-beebd278f938-certificates\") pod \"keda-metrics-apiserver-559f75f947-zk8hm\" (UID: \"59f85457-f6ea-4a12-a403-beebd278f938\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-zk8hm"
Jul 02 12:45:15.218641 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:45:15.218615    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45f538bc_2f6c_4b99_9608_dc1d59fd5ec2.slice/crio-d658aabe15caa4a9a1863fed898f9ef86a53e139c5ca1104b8307e0febd16c5e WatchSource:0}: Error finding container d658aabe15caa4a9a1863fed898f9ef86a53e139c5ca1104b8307e0febd16c5e: Status 404 returned error can't find the container with id d658aabe15caa4a9a1863fed898f9ef86a53e139c5ca1104b8307e0febd16c5e
Jul 02 12:45:15.305145 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:15.305012    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-x462p"]
Jul 02 12:45:15.308666 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:45:15.308641    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd790b920_d977_4010_80cc_6a984743deba.slice/crio-c4cbedefb16f2f27986e8a02c9ed98a552c056aae702b410d3303f2f086aa476 WatchSource:0}: Error finding container c4cbedefb16f2f27986e8a02c9ed98a552c056aae702b410d3303f2f086aa476: Status 404 returned error can't find the container with id c4cbedefb16f2f27986e8a02c9ed98a552c056aae702b410d3303f2f086aa476
Jul 02 12:45:15.408115 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:15.408086    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-zk8hm"
Jul 02 12:45:15.559320 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:15.559298    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-zk8hm"]
Jul 02 12:45:15.561766 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:45:15.561734    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59f85457_f6ea_4a12_a403_beebd278f938.slice/crio-c7e47a7a1024702d763bb59c3f81923e350360f3375d0c052177d2d667934336 WatchSource:0}: Error finding container c7e47a7a1024702d763bb59c3f81923e350360f3375d0c052177d2d667934336: Status 404 returned error can't find the container with id c7e47a7a1024702d763bb59c3f81923e350360f3375d0c052177d2d667934336
Jul 02 12:45:15.737116 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:15.737083    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-nf7ch" event={"ID":"45f538bc-2f6c-4b99-9608-dc1d59fd5ec2","Type":"ContainerStarted","Data":"d658aabe15caa4a9a1863fed898f9ef86a53e139c5ca1104b8307e0febd16c5e"}
Jul 02 12:45:15.738262 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:15.738235    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-zk8hm" event={"ID":"59f85457-f6ea-4a12-a403-beebd278f938","Type":"ContainerStarted","Data":"c7e47a7a1024702d763bb59c3f81923e350360f3375d0c052177d2d667934336"}
Jul 02 12:45:15.739327 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:15.739309    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-x462p" event={"ID":"d790b920-d977-4010-80cc-6a984743deba","Type":"ContainerStarted","Data":"c4cbedefb16f2f27986e8a02c9ed98a552c056aae702b410d3303f2f086aa476"}
Jul 02 12:45:17.753869 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:17.753635    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-x462p" event={"ID":"d790b920-d977-4010-80cc-6a984743deba","Type":"ContainerStarted","Data":"faca27e2cd72360cb6dcb59d3a064ac15d6d5e2e4d5712f7a2f0e4d4c42b2940"}
Jul 02 12:45:17.753869 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:17.753758    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-5fbd5c4644-x462p"
Jul 02 12:45:17.774227 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:17.774172    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-5fbd5c4644-x462p" podStartSLOduration=2.374562353 podStartE2EDuration="3.77415228s" totalImagesPullingTime="1.399589927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:45:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:45:17.771680912 +0000 UTC m=+406.379143937" watchObservedRunningTime="2026-07-02 12:45:17.77415228 +0000 UTC m=+406.381615305"
Jul 02 12:45:19.764637 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:19.764528    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-nf7ch" event={"ID":"45f538bc-2f6c-4b99-9608-dc1d59fd5ec2","Type":"ContainerStarted","Data":"e064db59ecf73c1f4f34b4936c6c304ace651ab4d0b622174684fbff099acf25"}
Jul 02 12:45:19.765130 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:19.764645    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-68bff59c-nf7ch"
Jul 02 12:45:19.766210 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:19.766182    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-zk8hm" event={"ID":"59f85457-f6ea-4a12-a403-beebd278f938","Type":"ContainerStarted","Data":"aea5273a99432451349f4bad6edf73e192ef791840822da406493eeb2eaf81b7"}
Jul 02 12:45:19.766351 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:19.766229    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-zk8hm"
Jul 02 12:45:19.782630 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:19.782586    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-68bff59c-nf7ch" podStartSLOduration=1.517710225 podStartE2EDuration="5.782570784s" totalImagesPullingTime="4.264860559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:45:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:45:19.781184595 +0000 UTC m=+408.388647618" watchObservedRunningTime="2026-07-02 12:45:19.782570784 +0000 UTC m=+408.390033807"
Jul 02 12:45:19.798331 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:19.798284    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-559f75f947-zk8hm" podStartSLOduration=1.876032428 podStartE2EDuration="5.798270721s" totalImagesPullingTime="3.922238293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:45:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:45:19.7970725 +0000 UTC m=+408.404535523" watchObservedRunningTime="2026-07-02 12:45:19.798270721 +0000 UTC m=+408.405733743"
Jul 02 12:45:30.775508 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:30.775476    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-zk8hm"
Jul 02 12:45:38.761100 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:38.761071    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-5fbd5c4644-x462p"
Jul 02 12:45:40.773291 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:45:40.773263    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-68bff59c-nf7ch"
Jul 02 12:46:40.138934 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:40.138880    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74684955bf-fx9k9"]
Jul 02 12:46:40.142287 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:40.142265    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74684955bf-fx9k9"
Jul 02 12:46:40.158258 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:40.158236    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74684955bf-fx9k9"]
Jul 02 12:46:40.243867 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:40.243834    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3049d03b-9514-4f31-b63d-8821c15d7998-service-ca\") pod \"console-74684955bf-fx9k9\" (UID: \"3049d03b-9514-4f31-b63d-8821c15d7998\") " pod="openshift-console/console-74684955bf-fx9k9"
Jul 02 12:46:40.243867 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:40.243870    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3049d03b-9514-4f31-b63d-8821c15d7998-trusted-ca-bundle\") pod \"console-74684955bf-fx9k9\" (UID: \"3049d03b-9514-4f31-b63d-8821c15d7998\") " pod="openshift-console/console-74684955bf-fx9k9"
Jul 02 12:46:40.244141 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:40.243995    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3049d03b-9514-4f31-b63d-8821c15d7998-console-oauth-config\") pod \"console-74684955bf-fx9k9\" (UID: \"3049d03b-9514-4f31-b63d-8821c15d7998\") " pod="openshift-console/console-74684955bf-fx9k9"
Jul 02 12:46:40.244141 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:40.244022    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp7s6\" (UniqueName: \"kubernetes.io/projected/3049d03b-9514-4f31-b63d-8821c15d7998-kube-api-access-qp7s6\") pod \"console-74684955bf-fx9k9\" (UID: \"3049d03b-9514-4f31-b63d-8821c15d7998\") " pod="openshift-console/console-74684955bf-fx9k9"
Jul 02 12:46:40.244141 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:40.244057    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3049d03b-9514-4f31-b63d-8821c15d7998-console-serving-cert\") pod \"console-74684955bf-fx9k9\" (UID: \"3049d03b-9514-4f31-b63d-8821c15d7998\") " pod="openshift-console/console-74684955bf-fx9k9"
Jul 02 12:46:40.244141 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:40.244091    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3049d03b-9514-4f31-b63d-8821c15d7998-console-config\") pod \"console-74684955bf-fx9k9\" (UID: \"3049d03b-9514-4f31-b63d-8821c15d7998\") " pod="openshift-console/console-74684955bf-fx9k9"
Jul 02 12:46:40.244141 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:40.244126    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3049d03b-9514-4f31-b63d-8821c15d7998-oauth-serving-cert\") pod \"console-74684955bf-fx9k9\" (UID: \"3049d03b-9514-4f31-b63d-8821c15d7998\") " pod="openshift-console/console-74684955bf-fx9k9"
Jul 02 12:46:40.344971 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:40.344943    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3049d03b-9514-4f31-b63d-8821c15d7998-console-config\") pod \"console-74684955bf-fx9k9\" (UID: \"3049d03b-9514-4f31-b63d-8821c15d7998\") " pod="openshift-console/console-74684955bf-fx9k9"
Jul 02 12:46:40.345140 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:40.344996    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3049d03b-9514-4f31-b63d-8821c15d7998-oauth-serving-cert\") pod \"console-74684955bf-fx9k9\" (UID: \"3049d03b-9514-4f31-b63d-8821c15d7998\") " pod="openshift-console/console-74684955bf-fx9k9"
Jul 02 12:46:40.345140 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:40.345043    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3049d03b-9514-4f31-b63d-8821c15d7998-service-ca\") pod \"console-74684955bf-fx9k9\" (UID: \"3049d03b-9514-4f31-b63d-8821c15d7998\") " pod="openshift-console/console-74684955bf-fx9k9"
Jul 02 12:46:40.345140 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:40.345058    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3049d03b-9514-4f31-b63d-8821c15d7998-trusted-ca-bundle\") pod \"console-74684955bf-fx9k9\" (UID: \"3049d03b-9514-4f31-b63d-8821c15d7998\") " pod="openshift-console/console-74684955bf-fx9k9"
Jul 02 12:46:40.345140 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:40.345102    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3049d03b-9514-4f31-b63d-8821c15d7998-console-oauth-config\") pod \"console-74684955bf-fx9k9\" (UID: \"3049d03b-9514-4f31-b63d-8821c15d7998\") " pod="openshift-console/console-74684955bf-fx9k9"
Jul 02 12:46:40.345140 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:40.345117    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp7s6\" (UniqueName: \"kubernetes.io/projected/3049d03b-9514-4f31-b63d-8821c15d7998-kube-api-access-qp7s6\") pod \"console-74684955bf-fx9k9\" (UID: \"3049d03b-9514-4f31-b63d-8821c15d7998\") " pod="openshift-console/console-74684955bf-fx9k9"
Jul 02 12:46:40.345368 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:40.345142    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3049d03b-9514-4f31-b63d-8821c15d7998-console-serving-cert\") pod \"console-74684955bf-fx9k9\" (UID: \"3049d03b-9514-4f31-b63d-8821c15d7998\") " pod="openshift-console/console-74684955bf-fx9k9"
Jul 02 12:46:40.345761 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:40.345735    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3049d03b-9514-4f31-b63d-8821c15d7998-oauth-serving-cert\") pod \"console-74684955bf-fx9k9\" (UID: \"3049d03b-9514-4f31-b63d-8821c15d7998\") " pod="openshift-console/console-74684955bf-fx9k9"
Jul 02 12:46:40.345964 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:40.345795    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3049d03b-9514-4f31-b63d-8821c15d7998-console-config\") pod \"console-74684955bf-fx9k9\" (UID: \"3049d03b-9514-4f31-b63d-8821c15d7998\") " pod="openshift-console/console-74684955bf-fx9k9"
Jul 02 12:46:40.345964 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:40.345953    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3049d03b-9514-4f31-b63d-8821c15d7998-trusted-ca-bundle\") pod \"console-74684955bf-fx9k9\" (UID: \"3049d03b-9514-4f31-b63d-8821c15d7998\") " pod="openshift-console/console-74684955bf-fx9k9"
Jul 02 12:46:40.345964 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:40.345795    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3049d03b-9514-4f31-b63d-8821c15d7998-service-ca\") pod \"console-74684955bf-fx9k9\" (UID: \"3049d03b-9514-4f31-b63d-8821c15d7998\") " pod="openshift-console/console-74684955bf-fx9k9"
Jul 02 12:46:40.347558 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:40.347537    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3049d03b-9514-4f31-b63d-8821c15d7998-console-oauth-config\") pod \"console-74684955bf-fx9k9\" (UID: \"3049d03b-9514-4f31-b63d-8821c15d7998\") " pod="openshift-console/console-74684955bf-fx9k9"
Jul 02 12:46:40.347608 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:40.347577    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3049d03b-9514-4f31-b63d-8821c15d7998-console-serving-cert\") pod \"console-74684955bf-fx9k9\" (UID: \"3049d03b-9514-4f31-b63d-8821c15d7998\") " pod="openshift-console/console-74684955bf-fx9k9"
Jul 02 12:46:40.354172 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:40.354147    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp7s6\" (UniqueName: \"kubernetes.io/projected/3049d03b-9514-4f31-b63d-8821c15d7998-kube-api-access-qp7s6\") pod \"console-74684955bf-fx9k9\" (UID: \"3049d03b-9514-4f31-b63d-8821c15d7998\") " pod="openshift-console/console-74684955bf-fx9k9"
Jul 02 12:46:40.451186 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:40.451157    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74684955bf-fx9k9"
Jul 02 12:46:40.588622 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:40.588601    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74684955bf-fx9k9"]
Jul 02 12:46:40.593051 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:46:40.593022    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3049d03b_9514_4f31_b63d_8821c15d7998.slice/crio-5d873e138fe91c043f8aadc9d3977380135a693ed25bde720b9131b3648eba37 WatchSource:0}: Error finding container 5d873e138fe91c043f8aadc9d3977380135a693ed25bde720b9131b3648eba37: Status 404 returned error can't find the container with id 5d873e138fe91c043f8aadc9d3977380135a693ed25bde720b9131b3648eba37
Jul 02 12:46:41.116976 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:41.116940    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74684955bf-fx9k9" event={"ID":"3049d03b-9514-4f31-b63d-8821c15d7998","Type":"ContainerStarted","Data":"50cf9c14abd0ae12522ea1499b989281acbbe58fa9682893333c5c697ac9fa7f"}
Jul 02 12:46:41.117147 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:41.116998    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74684955bf-fx9k9" event={"ID":"3049d03b-9514-4f31-b63d-8821c15d7998","Type":"ContainerStarted","Data":"5d873e138fe91c043f8aadc9d3977380135a693ed25bde720b9131b3648eba37"}
Jul 02 12:46:41.138211 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:41.138157    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74684955bf-fx9k9" podStartSLOduration=1.138141195 podStartE2EDuration="1.138141195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:46:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:46:41.137827956 +0000 UTC m=+489.745290980" watchObservedRunningTime="2026-07-02 12:46:41.138141195 +0000 UTC m=+489.745604218"
Jul 02 12:46:42.748182 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:42.748153    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-ctcr5"]
Jul 02 12:46:42.753511 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:42.753481    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-ctcr5"
Jul 02 12:46:42.757417 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:42.757394    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 02 12:46:42.758761 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:42.758747    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 02 12:46:42.758949 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:42.758929    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-q8r8j\""
Jul 02 12:46:42.760399 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:42.760382    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 02 12:46:42.761812 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:42.761792    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-ctcr5"]
Jul 02 12:46:42.869140 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:42.869104    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d50dab12-740d-4d20-a229-b42c7f5e2f8b-data\") pod \"seaweedfs-64568bcd49-ctcr5\" (UID: \"d50dab12-740d-4d20-a229-b42c7f5e2f8b\") " pod="kserve/seaweedfs-64568bcd49-ctcr5"
Jul 02 12:46:42.869314 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:42.869190    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9qlm\" (UniqueName: \"kubernetes.io/projected/d50dab12-740d-4d20-a229-b42c7f5e2f8b-kube-api-access-c9qlm\") pod \"seaweedfs-64568bcd49-ctcr5\" (UID: \"d50dab12-740d-4d20-a229-b42c7f5e2f8b\") " pod="kserve/seaweedfs-64568bcd49-ctcr5"
Jul 02 12:46:42.969649 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:42.969620    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9qlm\" (UniqueName: \"kubernetes.io/projected/d50dab12-740d-4d20-a229-b42c7f5e2f8b-kube-api-access-c9qlm\") pod \"seaweedfs-64568bcd49-ctcr5\" (UID: \"d50dab12-740d-4d20-a229-b42c7f5e2f8b\") " pod="kserve/seaweedfs-64568bcd49-ctcr5"
Jul 02 12:46:42.970026 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:42.970003    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d50dab12-740d-4d20-a229-b42c7f5e2f8b-data\") pod \"seaweedfs-64568bcd49-ctcr5\" (UID: \"d50dab12-740d-4d20-a229-b42c7f5e2f8b\") " pod="kserve/seaweedfs-64568bcd49-ctcr5"
Jul 02 12:46:42.970385 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:42.970361    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d50dab12-740d-4d20-a229-b42c7f5e2f8b-data\") pod \"seaweedfs-64568bcd49-ctcr5\" (UID: \"d50dab12-740d-4d20-a229-b42c7f5e2f8b\") " pod="kserve/seaweedfs-64568bcd49-ctcr5"
Jul 02 12:46:42.980722 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:42.980701    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9qlm\" (UniqueName: \"kubernetes.io/projected/d50dab12-740d-4d20-a229-b42c7f5e2f8b-kube-api-access-c9qlm\") pod \"seaweedfs-64568bcd49-ctcr5\" (UID: \"d50dab12-740d-4d20-a229-b42c7f5e2f8b\") " pod="kserve/seaweedfs-64568bcd49-ctcr5"
Jul 02 12:46:43.065633 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:43.065549    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-ctcr5"
Jul 02 12:46:43.191793 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:43.191769    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-ctcr5"]
Jul 02 12:46:43.193825 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:46:43.193798    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd50dab12_740d_4d20_a229_b42c7f5e2f8b.slice/crio-5de05a8ae4d7ba685e94942fc64cdc54befabee643529e55e51ea3571b3652b0 WatchSource:0}: Error finding container 5de05a8ae4d7ba685e94942fc64cdc54befabee643529e55e51ea3571b3652b0: Status 404 returned error can't find the container with id 5de05a8ae4d7ba685e94942fc64cdc54befabee643529e55e51ea3571b3652b0
Jul 02 12:46:44.133084 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:44.133044    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-ctcr5" event={"ID":"d50dab12-740d-4d20-a229-b42c7f5e2f8b","Type":"ContainerStarted","Data":"5de05a8ae4d7ba685e94942fc64cdc54befabee643529e55e51ea3571b3652b0"}
Jul 02 12:46:47.147880 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:47.147767    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-ctcr5" event={"ID":"d50dab12-740d-4d20-a229-b42c7f5e2f8b","Type":"ContainerStarted","Data":"2ab7862d6d3f21351bfe53bf5305b10832e01a8998c51500cab83b81b0450e27"}
Jul 02 12:46:47.147880 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:47.147836    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-ctcr5"
Jul 02 12:46:47.165398 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:47.165345    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-ctcr5" podStartSLOduration=1.4618198279999999 podStartE2EDuration="5.165328917s" totalImagesPullingTime="3.703509089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:46:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:46:47.164976892 +0000 UTC m=+495.772439908" watchObservedRunningTime="2026-07-02 12:46:47.165328917 +0000 UTC m=+495.772791932"
Jul 02 12:46:50.451865 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:50.451833    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74684955bf-fx9k9"
Jul 02 12:46:50.452277 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:50.452131    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74684955bf-fx9k9"
Jul 02 12:46:50.456740 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:50.456718    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74684955bf-fx9k9"
Jul 02 12:46:51.170530 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:51.170502    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74684955bf-fx9k9"
Jul 02 12:46:51.274879 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:51.274838    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c456986f6-4nswh"]
Jul 02 12:46:53.155337 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:46:53.155309    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-ctcr5"
Jul 02 12:47:16.295106 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:16.295067    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-c456986f6-4nswh" podUID="85db6f6b-7fe6-455f-b20b-587228f5042d" containerName="console" containerID="cri-o://f797f3ce930a6b0ced975d1d272c3384f0c2a43ba592de1e77c67c6377d0afa7" gracePeriod=15
Jul 02 12:47:16.532982 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:16.532960    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c456986f6-4nswh"
Jul 02 12:47:16.570123 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:16.570064    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/85db6f6b-7fe6-455f-b20b-587228f5042d-console-serving-cert\") pod \"85db6f6b-7fe6-455f-b20b-587228f5042d\" (UID: \"85db6f6b-7fe6-455f-b20b-587228f5042d\") "
Jul 02 12:47:16.570123 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:16.570099    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/85db6f6b-7fe6-455f-b20b-587228f5042d-service-ca\") pod \"85db6f6b-7fe6-455f-b20b-587228f5042d\" (UID: \"85db6f6b-7fe6-455f-b20b-587228f5042d\") "
Jul 02 12:47:16.570123 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:16.570119    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/85db6f6b-7fe6-455f-b20b-587228f5042d-console-oauth-config\") pod \"85db6f6b-7fe6-455f-b20b-587228f5042d\" (UID: \"85db6f6b-7fe6-455f-b20b-587228f5042d\") "
Jul 02 12:47:16.570359 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:16.570143    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/85db6f6b-7fe6-455f-b20b-587228f5042d-oauth-serving-cert\") pod \"85db6f6b-7fe6-455f-b20b-587228f5042d\" (UID: \"85db6f6b-7fe6-455f-b20b-587228f5042d\") "
Jul 02 12:47:16.570359 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:16.570269    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85db6f6b-7fe6-455f-b20b-587228f5042d-trusted-ca-bundle\") pod \"85db6f6b-7fe6-455f-b20b-587228f5042d\" (UID: \"85db6f6b-7fe6-455f-b20b-587228f5042d\") "
Jul 02 12:47:16.570359 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:16.570307    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/85db6f6b-7fe6-455f-b20b-587228f5042d-console-config\") pod \"85db6f6b-7fe6-455f-b20b-587228f5042d\" (UID: \"85db6f6b-7fe6-455f-b20b-587228f5042d\") "
Jul 02 12:47:16.570359 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:16.570345    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f79dw\" (UniqueName: \"kubernetes.io/projected/85db6f6b-7fe6-455f-b20b-587228f5042d-kube-api-access-f79dw\") pod \"85db6f6b-7fe6-455f-b20b-587228f5042d\" (UID: \"85db6f6b-7fe6-455f-b20b-587228f5042d\") "
Jul 02 12:47:16.570591 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:16.570569    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85db6f6b-7fe6-455f-b20b-587228f5042d-service-ca" (OuterVolumeSpecName: "service-ca") pod "85db6f6b-7fe6-455f-b20b-587228f5042d" (UID: "85db6f6b-7fe6-455f-b20b-587228f5042d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:47:16.570687 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:16.570647    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/85db6f6b-7fe6-455f-b20b-587228f5042d-service-ca\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:47:16.570687 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:16.570641    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85db6f6b-7fe6-455f-b20b-587228f5042d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "85db6f6b-7fe6-455f-b20b-587228f5042d" (UID: "85db6f6b-7fe6-455f-b20b-587228f5042d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:47:16.570687 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:16.570586    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85db6f6b-7fe6-455f-b20b-587228f5042d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "85db6f6b-7fe6-455f-b20b-587228f5042d" (UID: "85db6f6b-7fe6-455f-b20b-587228f5042d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:47:16.570819 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:16.570747    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85db6f6b-7fe6-455f-b20b-587228f5042d-console-config" (OuterVolumeSpecName: "console-config") pod "85db6f6b-7fe6-455f-b20b-587228f5042d" (UID: "85db6f6b-7fe6-455f-b20b-587228f5042d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:47:16.572380 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:16.572358    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85db6f6b-7fe6-455f-b20b-587228f5042d-kube-api-access-f79dw" (OuterVolumeSpecName: "kube-api-access-f79dw") pod "85db6f6b-7fe6-455f-b20b-587228f5042d" (UID: "85db6f6b-7fe6-455f-b20b-587228f5042d"). InnerVolumeSpecName "kube-api-access-f79dw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 12:47:16.572534 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:16.572518    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85db6f6b-7fe6-455f-b20b-587228f5042d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "85db6f6b-7fe6-455f-b20b-587228f5042d" (UID: "85db6f6b-7fe6-455f-b20b-587228f5042d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 12:47:16.572623 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:16.572608    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85db6f6b-7fe6-455f-b20b-587228f5042d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "85db6f6b-7fe6-455f-b20b-587228f5042d" (UID: "85db6f6b-7fe6-455f-b20b-587228f5042d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 12:47:16.671921 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:16.671867    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/85db6f6b-7fe6-455f-b20b-587228f5042d-console-serving-cert\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:47:16.671921 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:16.671902    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/85db6f6b-7fe6-455f-b20b-587228f5042d-console-oauth-config\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:47:16.671921 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:16.671932    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/85db6f6b-7fe6-455f-b20b-587228f5042d-oauth-serving-cert\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:47:16.672141 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:16.671942    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85db6f6b-7fe6-455f-b20b-587228f5042d-trusted-ca-bundle\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:47:16.672141 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:16.671950    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/85db6f6b-7fe6-455f-b20b-587228f5042d-console-config\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:47:16.672141 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:16.671959    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f79dw\" (UniqueName: \"kubernetes.io/projected/85db6f6b-7fe6-455f-b20b-587228f5042d-kube-api-access-f79dw\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:47:17.284960 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:17.284925    2576 generic.go:356] "Generic (PLEG): container finished" podID="85db6f6b-7fe6-455f-b20b-587228f5042d" containerID="f797f3ce930a6b0ced975d1d272c3384f0c2a43ba592de1e77c67c6377d0afa7" exitCode=0
Jul 02 12:47:17.285150 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:17.284974    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c456986f6-4nswh"
Jul 02 12:47:17.285150 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:17.284988    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c456986f6-4nswh" event={"ID":"85db6f6b-7fe6-455f-b20b-587228f5042d","Type":"ContainerDied","Data":"f797f3ce930a6b0ced975d1d272c3384f0c2a43ba592de1e77c67c6377d0afa7"}
Jul 02 12:47:17.285150 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:17.285036    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c456986f6-4nswh" event={"ID":"85db6f6b-7fe6-455f-b20b-587228f5042d","Type":"ContainerDied","Data":"41cfdee5753829a1c4a50b75ff25ebb07bf3b5f898cdadcefc125b3232887985"}
Jul 02 12:47:17.285150 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:17.285060    2576 scope.go:117] "RemoveContainer" containerID="f797f3ce930a6b0ced975d1d272c3384f0c2a43ba592de1e77c67c6377d0afa7"
Jul 02 12:47:17.293519 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:17.293496    2576 scope.go:117] "RemoveContainer" containerID="f797f3ce930a6b0ced975d1d272c3384f0c2a43ba592de1e77c67c6377d0afa7"
Jul 02 12:47:17.293781 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:47:17.293760    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f797f3ce930a6b0ced975d1d272c3384f0c2a43ba592de1e77c67c6377d0afa7\": container with ID starting with f797f3ce930a6b0ced975d1d272c3384f0c2a43ba592de1e77c67c6377d0afa7 not found: ID does not exist" containerID="f797f3ce930a6b0ced975d1d272c3384f0c2a43ba592de1e77c67c6377d0afa7"
Jul 02 12:47:17.293857 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:17.293791    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f797f3ce930a6b0ced975d1d272c3384f0c2a43ba592de1e77c67c6377d0afa7"} err="failed to get container status \"f797f3ce930a6b0ced975d1d272c3384f0c2a43ba592de1e77c67c6377d0afa7\": rpc error: code = NotFound desc = could not find container \"f797f3ce930a6b0ced975d1d272c3384f0c2a43ba592de1e77c67c6377d0afa7\": container with ID starting with f797f3ce930a6b0ced975d1d272c3384f0c2a43ba592de1e77c67c6377d0afa7 not found: ID does not exist"
Jul 02 12:47:17.304244 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:17.304220    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c456986f6-4nswh"]
Jul 02 12:47:17.308883 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:17.308859    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c456986f6-4nswh"]
Jul 02 12:47:18.863664 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:18.863630    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85db6f6b-7fe6-455f-b20b-587228f5042d" path="/var/lib/kubelet/pods/85db6f6b-7fe6-455f-b20b-587228f5042d/volumes"
Jul 02 12:47:52.565109 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:52.565077    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-sfx7w"]
Jul 02 12:47:52.565506 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:52.565450    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85db6f6b-7fe6-455f-b20b-587228f5042d" containerName="console"
Jul 02 12:47:52.565506 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:52.565461    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="85db6f6b-7fe6-455f-b20b-587228f5042d" containerName="console"
Jul 02 12:47:52.565578 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:52.565520    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="85db6f6b-7fe6-455f-b20b-587228f5042d" containerName="console"
Jul 02 12:47:52.568478 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:52.568462    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-sfx7w"
Jul 02 12:47:52.570904 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:52.570877    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-pkjkf\""
Jul 02 12:47:52.571539 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:52.571524    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 02 12:47:52.580211 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:52.580186    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-sfx7w"]
Jul 02 12:47:52.590343 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:52.590317    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-v6hf6"]
Jul 02 12:47:52.592964 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:52.592942    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjm47\" (UniqueName: \"kubernetes.io/projected/ae6d0696-29b3-4c83-985a-627e145834fb-kube-api-access-qjm47\") pod \"model-serving-api-fd65d7d6-sfx7w\" (UID: \"ae6d0696-29b3-4c83-985a-627e145834fb\") " pod="kserve/model-serving-api-fd65d7d6-sfx7w"
Jul 02 12:47:52.593056 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:52.593037    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae6d0696-29b3-4c83-985a-627e145834fb-tls-certs\") pod \"model-serving-api-fd65d7d6-sfx7w\" (UID: \"ae6d0696-29b3-4c83-985a-627e145834fb\") " pod="kserve/model-serving-api-fd65d7d6-sfx7w"
Jul 02 12:47:52.593670 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:52.593657    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-v6hf6"
Jul 02 12:47:52.597972 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:52.597953    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-7hntp\""
Jul 02 12:47:52.598394 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:52.598381    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 02 12:47:52.609575 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:52.609546    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-v6hf6"]
Jul 02 12:47:52.693760 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:52.693730    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae6d0696-29b3-4c83-985a-627e145834fb-tls-certs\") pod \"model-serving-api-fd65d7d6-sfx7w\" (UID: \"ae6d0696-29b3-4c83-985a-627e145834fb\") " pod="kserve/model-serving-api-fd65d7d6-sfx7w"
Jul 02 12:47:52.693954 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:52.693773    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjm47\" (UniqueName: \"kubernetes.io/projected/ae6d0696-29b3-4c83-985a-627e145834fb-kube-api-access-qjm47\") pod \"model-serving-api-fd65d7d6-sfx7w\" (UID: \"ae6d0696-29b3-4c83-985a-627e145834fb\") " pod="kserve/model-serving-api-fd65d7d6-sfx7w"
Jul 02 12:47:52.693954 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:52.693821    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a15ff7a3-b6fa-4198-bd40-9a88d1605187-cert\") pod \"odh-model-controller-84b54b7f97-v6hf6\" (UID: \"a15ff7a3-b6fa-4198-bd40-9a88d1605187\") " pod="kserve/odh-model-controller-84b54b7f97-v6hf6"
Jul 02 12:47:52.693954 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:52.693847    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbkh8\" (UniqueName: \"kubernetes.io/projected/a15ff7a3-b6fa-4198-bd40-9a88d1605187-kube-api-access-hbkh8\") pod \"odh-model-controller-84b54b7f97-v6hf6\" (UID: \"a15ff7a3-b6fa-4198-bd40-9a88d1605187\") " pod="kserve/odh-model-controller-84b54b7f97-v6hf6"
Jul 02 12:47:52.696253 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:52.696231    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae6d0696-29b3-4c83-985a-627e145834fb-tls-certs\") pod \"model-serving-api-fd65d7d6-sfx7w\" (UID: \"ae6d0696-29b3-4c83-985a-627e145834fb\") " pod="kserve/model-serving-api-fd65d7d6-sfx7w"
Jul 02 12:47:52.704922 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:52.704886    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjm47\" (UniqueName: \"kubernetes.io/projected/ae6d0696-29b3-4c83-985a-627e145834fb-kube-api-access-qjm47\") pod \"model-serving-api-fd65d7d6-sfx7w\" (UID: \"ae6d0696-29b3-4c83-985a-627e145834fb\") " pod="kserve/model-serving-api-fd65d7d6-sfx7w"
Jul 02 12:47:52.795076 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:52.795038    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a15ff7a3-b6fa-4198-bd40-9a88d1605187-cert\") pod \"odh-model-controller-84b54b7f97-v6hf6\" (UID: \"a15ff7a3-b6fa-4198-bd40-9a88d1605187\") " pod="kserve/odh-model-controller-84b54b7f97-v6hf6"
Jul 02 12:47:52.795241 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:52.795084    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbkh8\" (UniqueName: \"kubernetes.io/projected/a15ff7a3-b6fa-4198-bd40-9a88d1605187-kube-api-access-hbkh8\") pod \"odh-model-controller-84b54b7f97-v6hf6\" (UID: \"a15ff7a3-b6fa-4198-bd40-9a88d1605187\") " pod="kserve/odh-model-controller-84b54b7f97-v6hf6"
Jul 02 12:47:52.797426 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:52.797405    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a15ff7a3-b6fa-4198-bd40-9a88d1605187-cert\") pod \"odh-model-controller-84b54b7f97-v6hf6\" (UID: \"a15ff7a3-b6fa-4198-bd40-9a88d1605187\") " pod="kserve/odh-model-controller-84b54b7f97-v6hf6"
Jul 02 12:47:52.804328 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:52.804300    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbkh8\" (UniqueName: \"kubernetes.io/projected/a15ff7a3-b6fa-4198-bd40-9a88d1605187-kube-api-access-hbkh8\") pod \"odh-model-controller-84b54b7f97-v6hf6\" (UID: \"a15ff7a3-b6fa-4198-bd40-9a88d1605187\") " pod="kserve/odh-model-controller-84b54b7f97-v6hf6"
Jul 02 12:47:52.879389 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:52.879317    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-sfx7w"
Jul 02 12:47:52.903640 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:52.903613    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-v6hf6"
Jul 02 12:47:53.027340 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:53.027313    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-sfx7w"]
Jul 02 12:47:53.028833 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:47:53.028807    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae6d0696_29b3_4c83_985a_627e145834fb.slice/crio-5a64461f4c83b4df886e18938e559d74e04d3cbe7ffc7588b7aa71b1689cdd5e WatchSource:0}: Error finding container 5a64461f4c83b4df886e18938e559d74e04d3cbe7ffc7588b7aa71b1689cdd5e: Status 404 returned error can't find the container with id 5a64461f4c83b4df886e18938e559d74e04d3cbe7ffc7588b7aa71b1689cdd5e
Jul 02 12:47:53.055254 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:53.055233    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-v6hf6"]
Jul 02 12:47:53.057043 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:47:53.057022    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda15ff7a3_b6fa_4198_bd40_9a88d1605187.slice/crio-d9ae0f12f875e881cfaefdc5b27f138d29efda2a3b2453049e1a76c5529121d6 WatchSource:0}: Error finding container d9ae0f12f875e881cfaefdc5b27f138d29efda2a3b2453049e1a76c5529121d6: Status 404 returned error can't find the container with id d9ae0f12f875e881cfaefdc5b27f138d29efda2a3b2453049e1a76c5529121d6
Jul 02 12:47:53.454073 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:53.454040    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-sfx7w" event={"ID":"ae6d0696-29b3-4c83-985a-627e145834fb","Type":"ContainerStarted","Data":"5a64461f4c83b4df886e18938e559d74e04d3cbe7ffc7588b7aa71b1689cdd5e"}
Jul 02 12:47:53.455133 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:53.455110    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-v6hf6" event={"ID":"a15ff7a3-b6fa-4198-bd40-9a88d1605187","Type":"ContainerStarted","Data":"d9ae0f12f875e881cfaefdc5b27f138d29efda2a3b2453049e1a76c5529121d6"}
Jul 02 12:47:57.476628 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:57.476590    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-sfx7w" event={"ID":"ae6d0696-29b3-4c83-985a-627e145834fb","Type":"ContainerStarted","Data":"bf7fc66bab6e6c108e4554ae4a51252c877d06a79ddddc60b4c4d066a5c1349d"}
Jul 02 12:47:57.477127 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:57.476652    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-sfx7w"
Jul 02 12:47:57.478221 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:57.478200    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-v6hf6" event={"ID":"a15ff7a3-b6fa-4198-bd40-9a88d1605187","Type":"ContainerStarted","Data":"24d1709127c76ff834cef477b86d1c179904f822038f0a0196c9b769e9430e6e"}
Jul 02 12:47:57.478299 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:57.478265    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-v6hf6"
Jul 02 12:47:57.495960 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:57.495896    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-sfx7w" podStartSLOduration=1.71384212 podStartE2EDuration="5.495880922s" totalImagesPullingTime="3.782038802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:47:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:47:57.494603517 +0000 UTC m=+566.102066540" watchObservedRunningTime="2026-07-02 12:47:57.495880922 +0000 UTC m=+566.103343944"
Jul 02 12:47:57.514181 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:47:57.514136    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-v6hf6" podStartSLOduration=1.754775647 podStartE2EDuration="5.514120848s" totalImagesPullingTime="3.759345201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:47:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:47:57.513094252 +0000 UTC m=+566.120557276" watchObservedRunningTime="2026-07-02 12:47:57.514120848 +0000 UTC m=+566.121583870"
Jul 02 12:48:08.485429 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:48:08.485398    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-v6hf6"
Jul 02 12:48:08.487051 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:48:08.487030    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-sfx7w"
Jul 02 12:48:09.295452 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:48:09.295421    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-lvxwr"]
Jul 02 12:48:09.299095 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:48:09.299079    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-lvxwr"
Jul 02 12:48:09.306730 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:48:09.306703    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-lvxwr"]
Jul 02 12:48:09.352811 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:48:09.352785    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08faaa52-ffd6-433c-bddd-bbe21ea259ba-model-cache\") pod \"s3-init-lvxwr\" (UID: \"08faaa52-ffd6-433c-bddd-bbe21ea259ba\") " pod="kserve/s3-init-lvxwr"
Jul 02 12:48:09.352950 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:48:09.352830    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5dfb\" (UniqueName: \"kubernetes.io/projected/08faaa52-ffd6-433c-bddd-bbe21ea259ba-kube-api-access-s5dfb\") pod \"s3-init-lvxwr\" (UID: \"08faaa52-ffd6-433c-bddd-bbe21ea259ba\") " pod="kserve/s3-init-lvxwr"
Jul 02 12:48:09.453889 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:48:09.453864    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5dfb\" (UniqueName: \"kubernetes.io/projected/08faaa52-ffd6-433c-bddd-bbe21ea259ba-kube-api-access-s5dfb\") pod \"s3-init-lvxwr\" (UID: \"08faaa52-ffd6-433c-bddd-bbe21ea259ba\") " pod="kserve/s3-init-lvxwr"
Jul 02 12:48:09.454029 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:48:09.453953    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08faaa52-ffd6-433c-bddd-bbe21ea259ba-model-cache\") pod \"s3-init-lvxwr\" (UID: \"08faaa52-ffd6-433c-bddd-bbe21ea259ba\") " pod="kserve/s3-init-lvxwr"
Jul 02 12:48:09.454251 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:48:09.454235    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08faaa52-ffd6-433c-bddd-bbe21ea259ba-model-cache\") pod \"s3-init-lvxwr\" (UID: \"08faaa52-ffd6-433c-bddd-bbe21ea259ba\") " pod="kserve/s3-init-lvxwr"
Jul 02 12:48:09.461428 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:48:09.461410    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5dfb\" (UniqueName: \"kubernetes.io/projected/08faaa52-ffd6-433c-bddd-bbe21ea259ba-kube-api-access-s5dfb\") pod \"s3-init-lvxwr\" (UID: \"08faaa52-ffd6-433c-bddd-bbe21ea259ba\") " pod="kserve/s3-init-lvxwr"
Jul 02 12:48:09.624036 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:48:09.623959    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-lvxwr"
Jul 02 12:48:09.750045 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:48:09.750022    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-lvxwr"]
Jul 02 12:48:09.752200 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:48:09.752171    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08faaa52_ffd6_433c_bddd_bbe21ea259ba.slice/crio-b4450f7592c6e0056f41c830bb10f4fdf06f6082b9ad35b128d58d6a05bec73f WatchSource:0}: Error finding container b4450f7592c6e0056f41c830bb10f4fdf06f6082b9ad35b128d58d6a05bec73f: Status 404 returned error can't find the container with id b4450f7592c6e0056f41c830bb10f4fdf06f6082b9ad35b128d58d6a05bec73f
Jul 02 12:48:10.541882 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:48:10.541850    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-lvxwr" event={"ID":"08faaa52-ffd6-433c-bddd-bbe21ea259ba","Type":"ContainerStarted","Data":"b4450f7592c6e0056f41c830bb10f4fdf06f6082b9ad35b128d58d6a05bec73f"}
Jul 02 12:48:13.559445 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:48:13.559412    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-lvxwr" event={"ID":"08faaa52-ffd6-433c-bddd-bbe21ea259ba","Type":"ContainerStarted","Data":"7fd27063b6c4ee007e4ab2852669ce8af66255f952cbe75d005ae21e84140d85"}
Jul 02 12:48:15.572610 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:48:15.572576    2576 generic.go:356] "Generic (PLEG): container finished" podID="08faaa52-ffd6-433c-bddd-bbe21ea259ba" containerID="7fd27063b6c4ee007e4ab2852669ce8af66255f952cbe75d005ae21e84140d85" exitCode=0
Jul 02 12:48:15.572972 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:48:15.572646    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-lvxwr" event={"ID":"08faaa52-ffd6-433c-bddd-bbe21ea259ba","Type":"ContainerDied","Data":"7fd27063b6c4ee007e4ab2852669ce8af66255f952cbe75d005ae21e84140d85"}
Jul 02 12:48:20.604441 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:48:20.604412    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-lvxwr" event={"ID":"08faaa52-ffd6-433c-bddd-bbe21ea259ba","Type":"ContainerStarted","Data":"5566adbea1a827c45e71c10dc38c0cfa249dc818e10d2e8567a235642dce83ab"}
Jul 02 12:48:20.621292 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:48:20.621246    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-lvxwr" podStartSLOduration=1.817144504 podStartE2EDuration="11.621229801s" totalImagesPullingTime="8.546950069s" totalInitContainerRuntime="1.257135228s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:48:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:48:20.618926082 +0000 UTC m=+589.226389102" watchObservedRunningTime="2026-07-02 12:48:20.621229801 +0000 UTC m=+589.228692838"
Jul 02 12:48:32.213952 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:48:32.213903    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2c7ks_116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a/ovn-acl-logging/0.log"
Jul 02 12:48:32.835543 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:48:32.835515    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2c7ks_116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a/ovn-acl-logging/0.log"
Jul 02 12:49:25.914961 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:25.914904    2576 generic.go:356] "Generic (PLEG): container finished" podID="08faaa52-ffd6-433c-bddd-bbe21ea259ba" containerID="5566adbea1a827c45e71c10dc38c0cfa249dc818e10d2e8567a235642dce83ab" exitCode=0
Jul 02 12:49:25.915346 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:25.914978    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-lvxwr" event={"ID":"08faaa52-ffd6-433c-bddd-bbe21ea259ba","Type":"ContainerDied","Data":"5566adbea1a827c45e71c10dc38c0cfa249dc818e10d2e8567a235642dce83ab"}
Jul 02 12:49:27.051716 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:27.051691    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-lvxwr"
Jul 02 12:49:27.136401 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:27.136372    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08faaa52-ffd6-433c-bddd-bbe21ea259ba-model-cache\") pod \"08faaa52-ffd6-433c-bddd-bbe21ea259ba\" (UID: \"08faaa52-ffd6-433c-bddd-bbe21ea259ba\") "
Jul 02 12:49:27.136569 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:27.136451    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5dfb\" (UniqueName: \"kubernetes.io/projected/08faaa52-ffd6-433c-bddd-bbe21ea259ba-kube-api-access-s5dfb\") pod \"08faaa52-ffd6-433c-bddd-bbe21ea259ba\" (UID: \"08faaa52-ffd6-433c-bddd-bbe21ea259ba\") "
Jul 02 12:49:27.138586 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:27.138558    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08faaa52-ffd6-433c-bddd-bbe21ea259ba-kube-api-access-s5dfb" (OuterVolumeSpecName: "kube-api-access-s5dfb") pod "08faaa52-ffd6-433c-bddd-bbe21ea259ba" (UID: "08faaa52-ffd6-433c-bddd-bbe21ea259ba"). InnerVolumeSpecName "kube-api-access-s5dfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 12:49:27.155087 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:27.155057    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08faaa52-ffd6-433c-bddd-bbe21ea259ba-model-cache" (OuterVolumeSpecName: "model-cache") pod "08faaa52-ffd6-433c-bddd-bbe21ea259ba" (UID: "08faaa52-ffd6-433c-bddd-bbe21ea259ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 12:49:27.237758 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:27.237682    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5dfb\" (UniqueName: \"kubernetes.io/projected/08faaa52-ffd6-433c-bddd-bbe21ea259ba-kube-api-access-s5dfb\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:49:27.237758 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:27.237707    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08faaa52-ffd6-433c-bddd-bbe21ea259ba-model-cache\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:49:27.926551 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:27.926523    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-lvxwr"
Jul 02 12:49:27.926718 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:27.926525    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-lvxwr" event={"ID":"08faaa52-ffd6-433c-bddd-bbe21ea259ba","Type":"ContainerDied","Data":"b4450f7592c6e0056f41c830bb10f4fdf06f6082b9ad35b128d58d6a05bec73f"}
Jul 02 12:49:27.926718 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:27.926633    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4450f7592c6e0056f41c830bb10f4fdf06f6082b9ad35b128d58d6a05bec73f"
Jul 02 12:49:39.718021 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:39.717981    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz"]
Jul 02 12:49:39.718521 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:39.718505    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08faaa52-ffd6-433c-bddd-bbe21ea259ba" containerName="download-hf-model"
Jul 02 12:49:39.718563 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:39.718523    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="08faaa52-ffd6-433c-bddd-bbe21ea259ba" containerName="download-hf-model"
Jul 02 12:49:39.718563 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:39.718550    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08faaa52-ffd6-433c-bddd-bbe21ea259ba" containerName="s3-init"
Jul 02 12:49:39.718563 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:39.718558    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="08faaa52-ffd6-433c-bddd-bbe21ea259ba" containerName="s3-init"
Jul 02 12:49:39.718848 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:39.718648    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="08faaa52-ffd6-433c-bddd-bbe21ea259ba" containerName="s3-init"
Jul 02 12:49:39.722827 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:39.722810    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz"
Jul 02 12:49:39.725290 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:39.725265    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 12:49:39.725569 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:39.725547    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 12:49:39.725636 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:39.725567    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 12:49:39.725865 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:39.725840    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-2c740-kube-rbac-proxy-sar-config\""
Jul 02 12:49:39.725990 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:39.725901    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 12:49:39.726282 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:39.726264    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-2c740-predictor-serving-cert\""
Jul 02 12:49:39.726343 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:39.726268    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gnkzk\""
Jul 02 12:49:39.733601 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:39.733579    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz"]
Jul 02 12:49:39.840356 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:39.840327    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb23c80e-8cef-4219-9efd-ac6ed2cc8196-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz\" (UID: \"fb23c80e-8cef-4219-9efd-ac6ed2cc8196\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz"
Jul 02 12:49:39.840725 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:39.840403    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb23c80e-8cef-4219-9efd-ac6ed2cc8196-proxy-tls\") pod \"isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz\" (UID: \"fb23c80e-8cef-4219-9efd-ac6ed2cc8196\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz"
Jul 02 12:49:39.840725 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:39.840428    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb23c80e-8cef-4219-9efd-ac6ed2cc8196-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz\" (UID: \"fb23c80e-8cef-4219-9efd-ac6ed2cc8196\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz"
Jul 02 12:49:39.840725 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:39.840460    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb23c80e-8cef-4219-9efd-ac6ed2cc8196-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz\" (UID: \"fb23c80e-8cef-4219-9efd-ac6ed2cc8196\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz"
Jul 02 12:49:39.840725 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:39.840513    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9p5x\" (UniqueName: \"kubernetes.io/projected/fb23c80e-8cef-4219-9efd-ac6ed2cc8196-kube-api-access-p9p5x\") pod \"isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz\" (UID: \"fb23c80e-8cef-4219-9efd-ac6ed2cc8196\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz"
Jul 02 12:49:39.941219 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:39.941194    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb23c80e-8cef-4219-9efd-ac6ed2cc8196-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz\" (UID: \"fb23c80e-8cef-4219-9efd-ac6ed2cc8196\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz"
Jul 02 12:49:39.941367 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:39.941256    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb23c80e-8cef-4219-9efd-ac6ed2cc8196-proxy-tls\") pod \"isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz\" (UID: \"fb23c80e-8cef-4219-9efd-ac6ed2cc8196\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz"
Jul 02 12:49:39.941367 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:39.941290    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb23c80e-8cef-4219-9efd-ac6ed2cc8196-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz\" (UID: \"fb23c80e-8cef-4219-9efd-ac6ed2cc8196\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz"
Jul 02 12:49:39.941367 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:39.941325    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb23c80e-8cef-4219-9efd-ac6ed2cc8196-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz\" (UID: \"fb23c80e-8cef-4219-9efd-ac6ed2cc8196\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz"
Jul 02 12:49:39.941367 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:39.941340    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9p5x\" (UniqueName: \"kubernetes.io/projected/fb23c80e-8cef-4219-9efd-ac6ed2cc8196-kube-api-access-p9p5x\") pod \"isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz\" (UID: \"fb23c80e-8cef-4219-9efd-ac6ed2cc8196\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz"
Jul 02 12:49:39.941664 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:39.941634    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb23c80e-8cef-4219-9efd-ac6ed2cc8196-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz\" (UID: \"fb23c80e-8cef-4219-9efd-ac6ed2cc8196\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz"
Jul 02 12:49:39.941956 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:39.941938    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb23c80e-8cef-4219-9efd-ac6ed2cc8196-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz\" (UID: \"fb23c80e-8cef-4219-9efd-ac6ed2cc8196\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz"
Jul 02 12:49:39.942015 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:39.941994    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb23c80e-8cef-4219-9efd-ac6ed2cc8196-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz\" (UID: \"fb23c80e-8cef-4219-9efd-ac6ed2cc8196\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz"
Jul 02 12:49:39.943793 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:39.943774    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb23c80e-8cef-4219-9efd-ac6ed2cc8196-proxy-tls\") pod \"isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz\" (UID: \"fb23c80e-8cef-4219-9efd-ac6ed2cc8196\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz"
Jul 02 12:49:39.949585 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:39.949560    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9p5x\" (UniqueName: \"kubernetes.io/projected/fb23c80e-8cef-4219-9efd-ac6ed2cc8196-kube-api-access-p9p5x\") pod \"isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz\" (UID: \"fb23c80e-8cef-4219-9efd-ac6ed2cc8196\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz"
Jul 02 12:49:40.036031 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:40.035952    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz"
Jul 02 12:49:40.172878 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:40.172854    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz"]
Jul 02 12:49:40.175075 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:49:40.175047    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb23c80e_8cef_4219_9efd_ac6ed2cc8196.slice/crio-f008dd37bc2323159ab94afb597aecbfcaea08d553476f0a0ad1cca31d1f44ba WatchSource:0}: Error finding container f008dd37bc2323159ab94afb597aecbfcaea08d553476f0a0ad1cca31d1f44ba: Status 404 returned error can't find the container with id f008dd37bc2323159ab94afb597aecbfcaea08d553476f0a0ad1cca31d1f44ba
Jul 02 12:49:40.988659 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:40.988626    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" event={"ID":"fb23c80e-8cef-4219-9efd-ac6ed2cc8196","Type":"ContainerStarted","Data":"2d5488c5aef943a129c189743072e17d4c23ab0f8c38541c9535d857e4df82b3"}
Jul 02 12:49:40.989138 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:40.988692    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" event={"ID":"fb23c80e-8cef-4219-9efd-ac6ed2cc8196","Type":"ContainerStarted","Data":"f008dd37bc2323159ab94afb597aecbfcaea08d553476f0a0ad1cca31d1f44ba"}
Jul 02 12:49:45.009122 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:45.009085    2576 generic.go:356] "Generic (PLEG): container finished" podID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerID="2d5488c5aef943a129c189743072e17d4c23ab0f8c38541c9535d857e4df82b3" exitCode=0
Jul 02 12:49:45.009506 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:45.009163    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" event={"ID":"fb23c80e-8cef-4219-9efd-ac6ed2cc8196","Type":"ContainerDied","Data":"2d5488c5aef943a129c189743072e17d4c23ab0f8c38541c9535d857e4df82b3"}
Jul 02 12:49:45.010543 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:45.010527    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 12:49:59.100925 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:49:59.100861    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" event={"ID":"fb23c80e-8cef-4219-9efd-ac6ed2cc8196","Type":"ContainerStarted","Data":"faaa6ade763b185b7e13400f6c21241dcd5413bd5744f185070ac0840444767a"}
Jul 02 12:50:02.123088 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:50:02.123051    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" event={"ID":"fb23c80e-8cef-4219-9efd-ac6ed2cc8196","Type":"ContainerStarted","Data":"dc390b6756ffa9e26e663a001b51ff9d426c502027c8bad8fa25cdcb38cfeed6"}
Jul 02 12:50:05.141291 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:50:05.141252    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" event={"ID":"fb23c80e-8cef-4219-9efd-ac6ed2cc8196","Type":"ContainerStarted","Data":"6f1ad80f59a24348ce671e30db5a8cafe1938628e807b18bcaa3818119d82eab"}
Jul 02 12:50:05.141662 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:50:05.141453    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz"
Jul 02 12:50:05.141662 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:50:05.141603    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz"
Jul 02 12:50:05.143052 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:50:05.143026    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 02 12:50:05.165627 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:50:05.165579    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" podStartSLOduration=2.87903809 podStartE2EDuration="26.165562309s" totalImagesPullingTime="19.398945174s" totalInitContainerRuntime="3.887579045s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:49:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:50:05.164522023 +0000 UTC m=+693.771985049" watchObservedRunningTime="2026-07-02 12:50:05.165562309 +0000 UTC m=+693.773025331"
Jul 02 12:50:06.145937 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:50:06.145873    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz"
Jul 02 12:50:06.146411 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:50:06.146015    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 02 12:50:06.147138 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:50:06.147108    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:50:06.149958 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:50:06.149940    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz"
Jul 02 12:50:07.151815 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:50:07.151774    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 02 12:50:07.152252 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:50:07.152020    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:50:08.156898 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:50:08.156857    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 02 12:50:08.157304 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:50:08.157169    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:50:18.157415 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:50:18.157368    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 02 12:50:18.157960 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:50:18.157746    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:50:28.157258 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:50:28.157162    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 02 12:50:28.157720 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:50:28.157692    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:50:38.157667 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:50:38.157621    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 02 12:50:38.158099 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:50:38.158069    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:50:48.157115 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:50:48.157077    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 02 12:50:48.157604 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:50:48.157582    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:50:58.157481 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:50:58.157442    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 02 12:50:58.157963 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:50:58.157895    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:51:08.156952 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:08.156877    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 02 12:51:08.157467 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:08.157437    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:51:09.858080 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:09.858034    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 02 12:51:09.858473 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:09.858451    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:51:19.858566 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:19.858532    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz"
Jul 02 12:51:19.858971 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:19.858627    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz"
Jul 02 12:51:34.812991 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:34.812958    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz"]
Jul 02 12:51:34.813470 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:34.813230    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="kserve-container" containerID="cri-o://faaa6ade763b185b7e13400f6c21241dcd5413bd5744f185070ac0840444767a" gracePeriod=30
Jul 02 12:51:34.813470 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:34.813250    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="kube-rbac-proxy" containerID="cri-o://dc390b6756ffa9e26e663a001b51ff9d426c502027c8bad8fa25cdcb38cfeed6" gracePeriod=30
Jul 02 12:51:34.813470 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:34.813257    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="agent" containerID="cri-o://6f1ad80f59a24348ce671e30db5a8cafe1938628e807b18bcaa3818119d82eab" gracePeriod=30
Jul 02 12:51:34.832491 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:34.832445    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns"]
Jul 02 12:51:34.837331 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:34.837304    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns"
Jul 02 12:51:34.839886 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:34.839854    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-e1206-predictor-serving-cert\""
Jul 02 12:51:34.840066 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:34.840037    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-e1206-kube-rbac-proxy-sar-config\""
Jul 02 12:51:34.848942 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:34.848815    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns"]
Jul 02 12:51:34.910154 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:34.910121    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz"]
Jul 02 12:51:34.913994 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:34.913967    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz"
Jul 02 12:51:34.916276 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:34.916253    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-e1206-predictor-serving-cert\""
Jul 02 12:51:34.916373 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:34.916351    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-e1206-kube-rbac-proxy-sar-config\""
Jul 02 12:51:34.924674 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:34.924651    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz"]
Jul 02 12:51:34.972054 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:34.972027    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44bac380-51a6-475a-a4ce-a339dc65a434-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns\" (UID: \"44bac380-51a6-475a-a4ce-a339dc65a434\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns"
Jul 02 12:51:34.972189 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:34.972081    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z65m7\" (UniqueName: \"kubernetes.io/projected/44bac380-51a6-475a-a4ce-a339dc65a434-kube-api-access-z65m7\") pod \"isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns\" (UID: \"44bac380-51a6-475a-a4ce-a339dc65a434\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns"
Jul 02 12:51:34.972236 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:34.972190    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44bac380-51a6-475a-a4ce-a339dc65a434-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns\" (UID: \"44bac380-51a6-475a-a4ce-a339dc65a434\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns"
Jul 02 12:51:34.972273 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:34.972257    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44bac380-51a6-475a-a4ce-a339dc65a434-cabundle-cert\") pod \"isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns\" (UID: \"44bac380-51a6-475a-a4ce-a339dc65a434\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns"
Jul 02 12:51:34.972335 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:34.972319    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44bac380-51a6-475a-a4ce-a339dc65a434-proxy-tls\") pod \"isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns\" (UID: \"44bac380-51a6-475a-a4ce-a339dc65a434\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns"
Jul 02 12:51:35.072960 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:35.072855    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3450b92-228e-4217-8a8c-0f25d12e1938-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz\" (UID: \"c3450b92-228e-4217-8a8c-0f25d12e1938\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz"
Jul 02 12:51:35.072960 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:35.072902    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44bac380-51a6-475a-a4ce-a339dc65a434-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns\" (UID: \"44bac380-51a6-475a-a4ce-a339dc65a434\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns"
Jul 02 12:51:35.073180 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:35.072985    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44bac380-51a6-475a-a4ce-a339dc65a434-cabundle-cert\") pod \"isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns\" (UID: \"44bac380-51a6-475a-a4ce-a339dc65a434\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns"
Jul 02 12:51:35.073180 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:35.073025    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44bac380-51a6-475a-a4ce-a339dc65a434-proxy-tls\") pod \"isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns\" (UID: \"44bac380-51a6-475a-a4ce-a339dc65a434\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns"
Jul 02 12:51:35.073180 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:35.073058    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44bac380-51a6-475a-a4ce-a339dc65a434-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns\" (UID: \"44bac380-51a6-475a-a4ce-a339dc65a434\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns"
Jul 02 12:51:35.073180 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:35.073111    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9hlc\" (UniqueName: \"kubernetes.io/projected/c3450b92-228e-4217-8a8c-0f25d12e1938-kube-api-access-t9hlc\") pod \"isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz\" (UID: \"c3450b92-228e-4217-8a8c-0f25d12e1938\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz"
Jul 02 12:51:35.073180 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:35.073151    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3450b92-228e-4217-8a8c-0f25d12e1938-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz\" (UID: \"c3450b92-228e-4217-8a8c-0f25d12e1938\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz"
Jul 02 12:51:35.073398 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:35.073224    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z65m7\" (UniqueName: \"kubernetes.io/projected/44bac380-51a6-475a-a4ce-a339dc65a434-kube-api-access-z65m7\") pod \"isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns\" (UID: \"44bac380-51a6-475a-a4ce-a339dc65a434\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns"
Jul 02 12:51:35.073398 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:35.073260    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3450b92-228e-4217-8a8c-0f25d12e1938-proxy-tls\") pod \"isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz\" (UID: \"c3450b92-228e-4217-8a8c-0f25d12e1938\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz"
Jul 02 12:51:35.073398 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:35.073284    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3450b92-228e-4217-8a8c-0f25d12e1938-cabundle-cert\") pod \"isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz\" (UID: \"c3450b92-228e-4217-8a8c-0f25d12e1938\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz"
Jul 02 12:51:35.073513 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:35.073455    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44bac380-51a6-475a-a4ce-a339dc65a434-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns\" (UID: \"44bac380-51a6-475a-a4ce-a339dc65a434\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns"
Jul 02 12:51:35.073680 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:35.073663    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44bac380-51a6-475a-a4ce-a339dc65a434-cabundle-cert\") pod \"isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns\" (UID: \"44bac380-51a6-475a-a4ce-a339dc65a434\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns"
Jul 02 12:51:35.073738 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:35.073677    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44bac380-51a6-475a-a4ce-a339dc65a434-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns\" (UID: \"44bac380-51a6-475a-a4ce-a339dc65a434\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns"
Jul 02 12:51:35.075852 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:35.075825    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44bac380-51a6-475a-a4ce-a339dc65a434-proxy-tls\") pod \"isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns\" (UID: \"44bac380-51a6-475a-a4ce-a339dc65a434\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns"
Jul 02 12:51:35.081623 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:35.081596    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z65m7\" (UniqueName: \"kubernetes.io/projected/44bac380-51a6-475a-a4ce-a339dc65a434-kube-api-access-z65m7\") pod \"isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns\" (UID: \"44bac380-51a6-475a-a4ce-a339dc65a434\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns"
Jul 02 12:51:35.153382 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:35.153356    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns"
Jul 02 12:51:35.174185 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:35.174155    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9hlc\" (UniqueName: \"kubernetes.io/projected/c3450b92-228e-4217-8a8c-0f25d12e1938-kube-api-access-t9hlc\") pod \"isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz\" (UID: \"c3450b92-228e-4217-8a8c-0f25d12e1938\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz"
Jul 02 12:51:35.174293 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:35.174197    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3450b92-228e-4217-8a8c-0f25d12e1938-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz\" (UID: \"c3450b92-228e-4217-8a8c-0f25d12e1938\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz"
Jul 02 12:51:35.174293 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:35.174233    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3450b92-228e-4217-8a8c-0f25d12e1938-proxy-tls\") pod \"isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz\" (UID: \"c3450b92-228e-4217-8a8c-0f25d12e1938\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz"
Jul 02 12:51:35.174293 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:35.174254    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3450b92-228e-4217-8a8c-0f25d12e1938-cabundle-cert\") pod \"isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz\" (UID: \"c3450b92-228e-4217-8a8c-0f25d12e1938\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz"
Jul 02 12:51:35.174460 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:35.174292    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3450b92-228e-4217-8a8c-0f25d12e1938-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz\" (UID: \"c3450b92-228e-4217-8a8c-0f25d12e1938\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz"
Jul 02 12:51:35.174656 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:35.174632    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3450b92-228e-4217-8a8c-0f25d12e1938-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz\" (UID: \"c3450b92-228e-4217-8a8c-0f25d12e1938\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz"
Jul 02 12:51:35.174994 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:35.174973    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3450b92-228e-4217-8a8c-0f25d12e1938-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz\" (UID: \"c3450b92-228e-4217-8a8c-0f25d12e1938\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz"
Jul 02 12:51:35.175155 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:35.175132    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3450b92-228e-4217-8a8c-0f25d12e1938-cabundle-cert\") pod \"isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz\" (UID: \"c3450b92-228e-4217-8a8c-0f25d12e1938\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz"
Jul 02 12:51:35.176885 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:35.176849    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3450b92-228e-4217-8a8c-0f25d12e1938-proxy-tls\") pod \"isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz\" (UID: \"c3450b92-228e-4217-8a8c-0f25d12e1938\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz"
Jul 02 12:51:35.181778 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:35.181759    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9hlc\" (UniqueName: \"kubernetes.io/projected/c3450b92-228e-4217-8a8c-0f25d12e1938-kube-api-access-t9hlc\") pod \"isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz\" (UID: \"c3450b92-228e-4217-8a8c-0f25d12e1938\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz"
Jul 02 12:51:35.227118 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:35.227093    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz"
Jul 02 12:51:35.291606 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:35.291576    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns"]
Jul 02 12:51:35.294308 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:51:35.294070    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44bac380_51a6_475a_a4ce_a339dc65a434.slice/crio-83e0403aca8f363ed98c8c63eb94fcb5b1080dca815d00d871468da9a74b1435 WatchSource:0}: Error finding container 83e0403aca8f363ed98c8c63eb94fcb5b1080dca815d00d871468da9a74b1435: Status 404 returned error can't find the container with id 83e0403aca8f363ed98c8c63eb94fcb5b1080dca815d00d871468da9a74b1435
Jul 02 12:51:35.399461 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:35.399429    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz"]
Jul 02 12:51:35.403326 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:51:35.403292    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3450b92_228e_4217_8a8c_0f25d12e1938.slice/crio-41e218c30cf4dee0ecaa1031115642b9fb97d0dff8efe45f3e3a960a4daca538 WatchSource:0}: Error finding container 41e218c30cf4dee0ecaa1031115642b9fb97d0dff8efe45f3e3a960a4daca538: Status 404 returned error can't find the container with id 41e218c30cf4dee0ecaa1031115642b9fb97d0dff8efe45f3e3a960a4daca538
Jul 02 12:51:35.581578 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:35.581547    2576 generic.go:356] "Generic (PLEG): container finished" podID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerID="dc390b6756ffa9e26e663a001b51ff9d426c502027c8bad8fa25cdcb38cfeed6" exitCode=2
Jul 02 12:51:35.581779 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:35.581629    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" event={"ID":"fb23c80e-8cef-4219-9efd-ac6ed2cc8196","Type":"ContainerDied","Data":"dc390b6756ffa9e26e663a001b51ff9d426c502027c8bad8fa25cdcb38cfeed6"}
Jul 02 12:51:35.583437 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:35.583412    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns" event={"ID":"44bac380-51a6-475a-a4ce-a339dc65a434","Type":"ContainerStarted","Data":"c65ef4f92a4a54f26c13b3038f566cf710f4f5f06b8171672083e7f840006ac5"}
Jul 02 12:51:35.583580 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:35.583468    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns" event={"ID":"44bac380-51a6-475a-a4ce-a339dc65a434","Type":"ContainerStarted","Data":"83e0403aca8f363ed98c8c63eb94fcb5b1080dca815d00d871468da9a74b1435"}
Jul 02 12:51:35.585152 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:35.585125    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz" event={"ID":"c3450b92-228e-4217-8a8c-0f25d12e1938","Type":"ContainerStarted","Data":"ab6519185ff50a23b1784c7bc6a44c2ab1ca81846a3de373fce5e4533c9a6914"}
Jul 02 12:51:35.585251 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:35.585192    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz" event={"ID":"c3450b92-228e-4217-8a8c-0f25d12e1938","Type":"ContainerStarted","Data":"41e218c30cf4dee0ecaa1031115642b9fb97d0dff8efe45f3e3a960a4daca538"}
Jul 02 12:51:36.146552 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:36.146513    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Jul 02 12:51:39.609026 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:39.608936    2576 generic.go:356] "Generic (PLEG): container finished" podID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerID="faaa6ade763b185b7e13400f6c21241dcd5413bd5744f185070ac0840444767a" exitCode=0
Jul 02 12:51:39.609436 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:39.609014    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" event={"ID":"fb23c80e-8cef-4219-9efd-ac6ed2cc8196","Type":"ContainerDied","Data":"faaa6ade763b185b7e13400f6c21241dcd5413bd5744f185070ac0840444767a"}
Jul 02 12:51:39.610295 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:39.610277    2576 generic.go:356] "Generic (PLEG): container finished" podID="44bac380-51a6-475a-a4ce-a339dc65a434" containerID="c65ef4f92a4a54f26c13b3038f566cf710f4f5f06b8171672083e7f840006ac5" exitCode=0
Jul 02 12:51:39.610387 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:39.610341    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns" event={"ID":"44bac380-51a6-475a-a4ce-a339dc65a434","Type":"ContainerDied","Data":"c65ef4f92a4a54f26c13b3038f566cf710f4f5f06b8171672083e7f840006ac5"}
Jul 02 12:51:39.611768 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:39.611749    2576 generic.go:356] "Generic (PLEG): container finished" podID="c3450b92-228e-4217-8a8c-0f25d12e1938" containerID="ab6519185ff50a23b1784c7bc6a44c2ab1ca81846a3de373fce5e4533c9a6914" exitCode=0
Jul 02 12:51:39.611861 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:39.611774    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz" event={"ID":"c3450b92-228e-4217-8a8c-0f25d12e1938","Type":"ContainerDied","Data":"ab6519185ff50a23b1784c7bc6a44c2ab1ca81846a3de373fce5e4533c9a6914"}
Jul 02 12:51:39.858421 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:39.858375    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 02 12:51:39.858768 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:39.858737    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:51:40.621459 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:40.621215    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns" event={"ID":"44bac380-51a6-475a-a4ce-a339dc65a434","Type":"ContainerStarted","Data":"86a584fb8f8e18ba9179f5aba8770bed00b8cddc444e55450e7b9799bb0f2050"}
Jul 02 12:51:40.621459 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:40.621304    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns" event={"ID":"44bac380-51a6-475a-a4ce-a339dc65a434","Type":"ContainerStarted","Data":"3f7c1d27c709df18b858438451fb636cf8789b3b3ff250ddfab055b1f8c85a12"}
Jul 02 12:51:40.622080 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:40.621543    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns"
Jul 02 12:51:40.642400 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:40.642345    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns" podStartSLOduration=2.688802365 podStartE2EDuration="6.642324557s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.953522192s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:51:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:51:40.639816179 +0000 UTC m=+789.247279215" watchObservedRunningTime="2026-07-02 12:51:40.642324557 +0000 UTC m=+789.249787582"
Jul 02 12:51:41.146253 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:41.146208    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Jul 02 12:51:41.628538 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:41.628460    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns"
Jul 02 12:51:41.630332 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:41.630300    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns" podUID="44bac380-51a6-475a-a4ce-a339dc65a434" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 02 12:51:42.632919 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:42.632857    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns" podUID="44bac380-51a6-475a-a4ce-a339dc65a434" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 02 12:51:46.146365 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:46.146303    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Jul 02 12:51:46.146820 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:46.146650    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz"
Jul 02 12:51:47.643669 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:47.643637    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns"
Jul 02 12:51:47.644325 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:47.644294    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns" podUID="44bac380-51a6-475a-a4ce-a339dc65a434" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 02 12:51:49.858350 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:49.858293    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 02 12:51:49.858815 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:49.858779    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:51:51.146890 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:51.146838    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Jul 02 12:51:56.146892 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:56.146838    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Jul 02 12:51:57.645256 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:57.645214    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns" podUID="44bac380-51a6-475a-a4ce-a339dc65a434" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 02 12:51:59.858131 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:59.858093    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 02 12:51:59.858629 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:59.858242    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz"
Jul 02 12:51:59.858702 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:59.858632    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:51:59.858767 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:51:59.858754    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz"
Jul 02 12:52:00.739686 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:00.739646    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz" event={"ID":"c3450b92-228e-4217-8a8c-0f25d12e1938","Type":"ContainerStarted","Data":"4919045eaef12d8c729f8b4229f4de4198bfc714972932fd938f6894ae582348"}
Jul 02 12:52:00.739901 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:00.739738    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz" event={"ID":"c3450b92-228e-4217-8a8c-0f25d12e1938","Type":"ContainerStarted","Data":"c299e5a10c321c4fff62ffda42fdf110291775ae5a252410e739f7d6c7d7846d"}
Jul 02 12:52:00.740052 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:00.740032    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz"
Jul 02 12:52:00.740101 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:00.740090    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz"
Jul 02 12:52:00.741587 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:00.741557    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz" podUID="c3450b92-228e-4217-8a8c-0f25d12e1938" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 02 12:52:00.760288 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:00.760243    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz" podStartSLOduration=2.493072889 podStartE2EDuration="26.760226912s" totalImagesPullingTime="20.69722529s" totalInitContainerRuntime="3.569928733s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:51:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:52:00.759349499 +0000 UTC m=+809.366812525" watchObservedRunningTime="2026-07-02 12:52:00.760226912 +0000 UTC m=+809.367689935"
Jul 02 12:52:01.146433 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:01.146397    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Jul 02 12:52:01.745042 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:01.745007    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz" podUID="c3450b92-228e-4217-8a8c-0f25d12e1938" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 02 12:52:05.500355 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:05.500328    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz"
Jul 02 12:52:05.570415 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:05.570386    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb23c80e-8cef-4219-9efd-ac6ed2cc8196-cabundle-cert\") pod \"fb23c80e-8cef-4219-9efd-ac6ed2cc8196\" (UID: \"fb23c80e-8cef-4219-9efd-ac6ed2cc8196\") "
Jul 02 12:52:05.570584 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:05.570423    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb23c80e-8cef-4219-9efd-ac6ed2cc8196-kube-rbac-proxy-sar-config\") pod \"fb23c80e-8cef-4219-9efd-ac6ed2cc8196\" (UID: \"fb23c80e-8cef-4219-9efd-ac6ed2cc8196\") "
Jul 02 12:52:05.570584 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:05.570458    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb23c80e-8cef-4219-9efd-ac6ed2cc8196-kserve-provision-location\") pod \"fb23c80e-8cef-4219-9efd-ac6ed2cc8196\" (UID: \"fb23c80e-8cef-4219-9efd-ac6ed2cc8196\") "
Jul 02 12:52:05.570584 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:05.570476    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb23c80e-8cef-4219-9efd-ac6ed2cc8196-proxy-tls\") pod \"fb23c80e-8cef-4219-9efd-ac6ed2cc8196\" (UID: \"fb23c80e-8cef-4219-9efd-ac6ed2cc8196\") "
Jul 02 12:52:05.570778 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:05.570598    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9p5x\" (UniqueName: \"kubernetes.io/projected/fb23c80e-8cef-4219-9efd-ac6ed2cc8196-kube-api-access-p9p5x\") pod \"fb23c80e-8cef-4219-9efd-ac6ed2cc8196\" (UID: \"fb23c80e-8cef-4219-9efd-ac6ed2cc8196\") "
Jul 02 12:52:05.570778 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:05.570666    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb23c80e-8cef-4219-9efd-ac6ed2cc8196-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fb23c80e-8cef-4219-9efd-ac6ed2cc8196" (UID: "fb23c80e-8cef-4219-9efd-ac6ed2cc8196"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 12:52:05.570874 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:05.570771    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb23c80e-8cef-4219-9efd-ac6ed2cc8196-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fb23c80e-8cef-4219-9efd-ac6ed2cc8196" (UID: "fb23c80e-8cef-4219-9efd-ac6ed2cc8196"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:52:05.570874 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:05.570797    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb23c80e-8cef-4219-9efd-ac6ed2cc8196-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fb23c80e-8cef-4219-9efd-ac6ed2cc8196" (UID: "fb23c80e-8cef-4219-9efd-ac6ed2cc8196"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:52:05.570990 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:05.570878    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb23c80e-8cef-4219-9efd-ac6ed2cc8196-cabundle-cert\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:52:05.570990 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:05.570892    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb23c80e-8cef-4219-9efd-ac6ed2cc8196-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:52:05.570990 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:05.570902    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb23c80e-8cef-4219-9efd-ac6ed2cc8196-kserve-provision-location\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:52:05.572527 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:05.572507    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb23c80e-8cef-4219-9efd-ac6ed2cc8196-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fb23c80e-8cef-4219-9efd-ac6ed2cc8196" (UID: "fb23c80e-8cef-4219-9efd-ac6ed2cc8196"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 12:52:05.572695 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:05.572676    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb23c80e-8cef-4219-9efd-ac6ed2cc8196-kube-api-access-p9p5x" (OuterVolumeSpecName: "kube-api-access-p9p5x") pod "fb23c80e-8cef-4219-9efd-ac6ed2cc8196" (UID: "fb23c80e-8cef-4219-9efd-ac6ed2cc8196"). InnerVolumeSpecName "kube-api-access-p9p5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 12:52:05.671542 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:05.671512    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb23c80e-8cef-4219-9efd-ac6ed2cc8196-proxy-tls\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:52:05.671542 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:05.671539    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9p5x\" (UniqueName: \"kubernetes.io/projected/fb23c80e-8cef-4219-9efd-ac6ed2cc8196-kube-api-access-p9p5x\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:52:05.765261 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:05.765229    2576 generic.go:356] "Generic (PLEG): container finished" podID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerID="6f1ad80f59a24348ce671e30db5a8cafe1938628e807b18bcaa3818119d82eab" exitCode=0
Jul 02 12:52:05.765400 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:05.765289    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz"
Jul 02 12:52:05.765400 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:05.765290    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" event={"ID":"fb23c80e-8cef-4219-9efd-ac6ed2cc8196","Type":"ContainerDied","Data":"6f1ad80f59a24348ce671e30db5a8cafe1938628e807b18bcaa3818119d82eab"}
Jul 02 12:52:05.765474 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:05.765437    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz" event={"ID":"fb23c80e-8cef-4219-9efd-ac6ed2cc8196","Type":"ContainerDied","Data":"f008dd37bc2323159ab94afb597aecbfcaea08d553476f0a0ad1cca31d1f44ba"}
Jul 02 12:52:05.765474 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:05.765470    2576 scope.go:117] "RemoveContainer" containerID="6f1ad80f59a24348ce671e30db5a8cafe1938628e807b18bcaa3818119d82eab"
Jul 02 12:52:05.774819 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:05.774800    2576 scope.go:117] "RemoveContainer" containerID="dc390b6756ffa9e26e663a001b51ff9d426c502027c8bad8fa25cdcb38cfeed6"
Jul 02 12:52:05.782217 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:05.782199    2576 scope.go:117] "RemoveContainer" containerID="faaa6ade763b185b7e13400f6c21241dcd5413bd5744f185070ac0840444767a"
Jul 02 12:52:05.790034 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:05.790011    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz"]
Jul 02 12:52:05.791299 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:05.791277    2576 scope.go:117] "RemoveContainer" containerID="2d5488c5aef943a129c189743072e17d4c23ab0f8c38541c9535d857e4df82b3"
Jul 02 12:52:05.793394 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:05.793373    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2c740-predictor-65574885fd-twccz"]
Jul 02 12:52:05.799377 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:05.799360    2576 scope.go:117] "RemoveContainer" containerID="6f1ad80f59a24348ce671e30db5a8cafe1938628e807b18bcaa3818119d82eab"
Jul 02 12:52:05.799639 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:52:05.799619    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f1ad80f59a24348ce671e30db5a8cafe1938628e807b18bcaa3818119d82eab\": container with ID starting with 6f1ad80f59a24348ce671e30db5a8cafe1938628e807b18bcaa3818119d82eab not found: ID does not exist" containerID="6f1ad80f59a24348ce671e30db5a8cafe1938628e807b18bcaa3818119d82eab"
Jul 02 12:52:05.799695 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:05.799652    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f1ad80f59a24348ce671e30db5a8cafe1938628e807b18bcaa3818119d82eab"} err="failed to get container status \"6f1ad80f59a24348ce671e30db5a8cafe1938628e807b18bcaa3818119d82eab\": rpc error: code = NotFound desc = could not find container \"6f1ad80f59a24348ce671e30db5a8cafe1938628e807b18bcaa3818119d82eab\": container with ID starting with 6f1ad80f59a24348ce671e30db5a8cafe1938628e807b18bcaa3818119d82eab not found: ID does not exist"
Jul 02 12:52:05.799695 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:05.799677    2576 scope.go:117] "RemoveContainer" containerID="dc390b6756ffa9e26e663a001b51ff9d426c502027c8bad8fa25cdcb38cfeed6"
Jul 02 12:52:05.799928 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:52:05.799889    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc390b6756ffa9e26e663a001b51ff9d426c502027c8bad8fa25cdcb38cfeed6\": container with ID starting with dc390b6756ffa9e26e663a001b51ff9d426c502027c8bad8fa25cdcb38cfeed6 not found: ID does not exist" containerID="dc390b6756ffa9e26e663a001b51ff9d426c502027c8bad8fa25cdcb38cfeed6"
Jul 02 12:52:05.800024 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:05.800006    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc390b6756ffa9e26e663a001b51ff9d426c502027c8bad8fa25cdcb38cfeed6"} err="failed to get container status \"dc390b6756ffa9e26e663a001b51ff9d426c502027c8bad8fa25cdcb38cfeed6\": rpc error: code = NotFound desc = could not find container \"dc390b6756ffa9e26e663a001b51ff9d426c502027c8bad8fa25cdcb38cfeed6\": container with ID starting with dc390b6756ffa9e26e663a001b51ff9d426c502027c8bad8fa25cdcb38cfeed6 not found: ID does not exist"
Jul 02 12:52:05.800065 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:05.800025    2576 scope.go:117] "RemoveContainer" containerID="faaa6ade763b185b7e13400f6c21241dcd5413bd5744f185070ac0840444767a"
Jul 02 12:52:05.800265 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:52:05.800247    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faaa6ade763b185b7e13400f6c21241dcd5413bd5744f185070ac0840444767a\": container with ID starting with faaa6ade763b185b7e13400f6c21241dcd5413bd5744f185070ac0840444767a not found: ID does not exist" containerID="faaa6ade763b185b7e13400f6c21241dcd5413bd5744f185070ac0840444767a"
Jul 02 12:52:05.800338 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:05.800274    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faaa6ade763b185b7e13400f6c21241dcd5413bd5744f185070ac0840444767a"} err="failed to get container status \"faaa6ade763b185b7e13400f6c21241dcd5413bd5744f185070ac0840444767a\": rpc error: code = NotFound desc = could not find container \"faaa6ade763b185b7e13400f6c21241dcd5413bd5744f185070ac0840444767a\": container with ID starting with faaa6ade763b185b7e13400f6c21241dcd5413bd5744f185070ac0840444767a not found: ID does not exist"
Jul 02 12:52:05.800338 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:05.800296    2576 scope.go:117] "RemoveContainer" containerID="2d5488c5aef943a129c189743072e17d4c23ab0f8c38541c9535d857e4df82b3"
Jul 02 12:52:05.800568 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:52:05.800550    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d5488c5aef943a129c189743072e17d4c23ab0f8c38541c9535d857e4df82b3\": container with ID starting with 2d5488c5aef943a129c189743072e17d4c23ab0f8c38541c9535d857e4df82b3 not found: ID does not exist" containerID="2d5488c5aef943a129c189743072e17d4c23ab0f8c38541c9535d857e4df82b3"
Jul 02 12:52:05.800607 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:05.800572    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d5488c5aef943a129c189743072e17d4c23ab0f8c38541c9535d857e4df82b3"} err="failed to get container status \"2d5488c5aef943a129c189743072e17d4c23ab0f8c38541c9535d857e4df82b3\": rpc error: code = NotFound desc = could not find container \"2d5488c5aef943a129c189743072e17d4c23ab0f8c38541c9535d857e4df82b3\": container with ID starting with 2d5488c5aef943a129c189743072e17d4c23ab0f8c38541c9535d857e4df82b3 not found: ID does not exist"
Jul 02 12:52:06.749624 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:06.749595    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz"
Jul 02 12:52:06.750147 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:06.750120    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz" podUID="c3450b92-228e-4217-8a8c-0f25d12e1938" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 02 12:52:06.862988 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:06.862958    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" path="/var/lib/kubelet/pods/fb23c80e-8cef-4219-9efd-ac6ed2cc8196/volumes"
Jul 02 12:52:07.645150 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:07.645110    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns" podUID="44bac380-51a6-475a-a4ce-a339dc65a434" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 02 12:52:16.750182 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:16.750144    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz" podUID="c3450b92-228e-4217-8a8c-0f25d12e1938" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 02 12:52:17.644840 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:17.644804    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns" podUID="44bac380-51a6-475a-a4ce-a339dc65a434" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 02 12:52:26.750656 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:26.750615    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz" podUID="c3450b92-228e-4217-8a8c-0f25d12e1938" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 02 12:52:27.644370 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:27.644333    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns" podUID="44bac380-51a6-475a-a4ce-a339dc65a434" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 02 12:52:36.750358 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:36.750317    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz" podUID="c3450b92-228e-4217-8a8c-0f25d12e1938" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 02 12:52:37.644966 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:37.644904    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns" podUID="44bac380-51a6-475a-a4ce-a339dc65a434" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 02 12:52:46.750736 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:46.750699    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz" podUID="c3450b92-228e-4217-8a8c-0f25d12e1938" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 02 12:52:47.644361 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:47.644324    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns" podUID="44bac380-51a6-475a-a4ce-a339dc65a434" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 02 12:52:56.750168 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:56.750127    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz" podUID="c3450b92-228e-4217-8a8c-0f25d12e1938" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 02 12:52:57.645050 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:52:57.645014    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns"
Jul 02 12:53:06.751487 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:06.751459    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz"
Jul 02 12:53:32.244499 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:32.244473    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2c7ks_116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a/ovn-acl-logging/0.log"
Jul 02 12:53:32.869382 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:32.869354    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2c7ks_116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a/ovn-acl-logging/0.log"
Jul 02 12:53:35.245495 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:35.245463    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx"]
Jul 02 12:53:35.246012 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:35.245991    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="kserve-container"
Jul 02 12:53:35.246012 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:35.246010    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="kserve-container"
Jul 02 12:53:35.246116 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:35.246024    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="kube-rbac-proxy"
Jul 02 12:53:35.246116 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:35.246030    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="kube-rbac-proxy"
Jul 02 12:53:35.246116 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:35.246046    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="storage-initializer"
Jul 02 12:53:35.246116 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:35.246053    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="storage-initializer"
Jul 02 12:53:35.246116 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:35.246066    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="agent"
Jul 02 12:53:35.246116 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:35.246073    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="agent"
Jul 02 12:53:35.246405 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:35.246138    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="kube-rbac-proxy"
Jul 02 12:53:35.246405 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:35.246155    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="kserve-container"
Jul 02 12:53:35.246405 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:35.246164    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb23c80e-8cef-4219-9efd-ac6ed2cc8196" containerName="agent"
Jul 02 12:53:35.249975 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:35.249956    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx"
Jul 02 12:53:35.252750 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:35.252727    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-9587b-kube-rbac-proxy-sar-config\""
Jul 02 12:53:35.252880 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:35.252752    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-9587b-predictor-serving-cert\""
Jul 02 12:53:35.258872 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:35.258849    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns"]
Jul 02 12:53:35.259332 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:35.259135    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns" podUID="44bac380-51a6-475a-a4ce-a339dc65a434" containerName="kserve-container" containerID="cri-o://3f7c1d27c709df18b858438451fb636cf8789b3b3ff250ddfab055b1f8c85a12" gracePeriod=30
Jul 02 12:53:35.259332 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:35.259308    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns" podUID="44bac380-51a6-475a-a4ce-a339dc65a434" containerName="kube-rbac-proxy" containerID="cri-o://86a584fb8f8e18ba9179f5aba8770bed00b8cddc444e55450e7b9799bb0f2050" gracePeriod=30
Jul 02 12:53:35.267502 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:35.267476    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx"]
Jul 02 12:53:35.384733 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:35.384692    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsbhr\" (UniqueName: \"kubernetes.io/projected/7866608b-328f-49f4-b8c0-1cfb595e820d-kube-api-access-vsbhr\") pod \"isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx\" (UID: \"7866608b-328f-49f4-b8c0-1cfb595e820d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx"
Jul 02 12:53:35.384946 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:35.384779    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7866608b-328f-49f4-b8c0-1cfb595e820d-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx\" (UID: \"7866608b-328f-49f4-b8c0-1cfb595e820d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx"
Jul 02 12:53:35.384946 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:35.384900    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7866608b-328f-49f4-b8c0-1cfb595e820d-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx\" (UID: \"7866608b-328f-49f4-b8c0-1cfb595e820d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx"
Jul 02 12:53:35.385117 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:35.384964    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7866608b-328f-49f4-b8c0-1cfb595e820d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx\" (UID: \"7866608b-328f-49f4-b8c0-1cfb595e820d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx"
Jul 02 12:53:35.385117 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:35.385012    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7866608b-328f-49f4-b8c0-1cfb595e820d-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx\" (UID: \"7866608b-328f-49f4-b8c0-1cfb595e820d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx"
Jul 02 12:53:35.421339 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:35.421311    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz"]
Jul 02 12:53:35.421564 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:35.421534    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz" podUID="c3450b92-228e-4217-8a8c-0f25d12e1938" containerName="kserve-container" containerID="cri-o://c299e5a10c321c4fff62ffda42fdf110291775ae5a252410e739f7d6c7d7846d" gracePeriod=30
Jul 02 12:53:35.421712 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:35.421564    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz" podUID="c3450b92-228e-4217-8a8c-0f25d12e1938" containerName="kube-rbac-proxy" containerID="cri-o://4919045eaef12d8c729f8b4229f4de4198bfc714972932fd938f6894ae582348" gracePeriod=30
Jul 02 12:53:35.486269 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:35.486241    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7866608b-328f-49f4-b8c0-1cfb595e820d-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx\" (UID: \"7866608b-328f-49f4-b8c0-1cfb595e820d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx"
Jul 02 12:53:35.486397 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:35.486272    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7866608b-328f-49f4-b8c0-1cfb595e820d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx\" (UID: \"7866608b-328f-49f4-b8c0-1cfb595e820d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx"
Jul 02 12:53:35.486397 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:35.486301    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7866608b-328f-49f4-b8c0-1cfb595e820d-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx\" (UID: \"7866608b-328f-49f4-b8c0-1cfb595e820d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx"
Jul 02 12:53:35.486397 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:35.486343    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsbhr\" (UniqueName: \"kubernetes.io/projected/7866608b-328f-49f4-b8c0-1cfb595e820d-kube-api-access-vsbhr\") pod \"isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx\" (UID: \"7866608b-328f-49f4-b8c0-1cfb595e820d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx"
Jul 02 12:53:35.486397 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:35.486379    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7866608b-328f-49f4-b8c0-1cfb595e820d-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx\" (UID: \"7866608b-328f-49f4-b8c0-1cfb595e820d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx"
Jul 02 12:53:35.486816 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:35.486791    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7866608b-328f-49f4-b8c0-1cfb595e820d-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx\" (UID: \"7866608b-328f-49f4-b8c0-1cfb595e820d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx"
Jul 02 12:53:35.487003 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:35.486984    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7866608b-328f-49f4-b8c0-1cfb595e820d-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx\" (UID: \"7866608b-328f-49f4-b8c0-1cfb595e820d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx"
Jul 02 12:53:35.487003 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:35.486990    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7866608b-328f-49f4-b8c0-1cfb595e820d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx\" (UID: \"7866608b-328f-49f4-b8c0-1cfb595e820d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx"
Jul 02 12:53:35.488614 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:35.488590    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7866608b-328f-49f4-b8c0-1cfb595e820d-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx\" (UID: \"7866608b-328f-49f4-b8c0-1cfb595e820d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx"
Jul 02 12:53:35.495833 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:35.495781    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsbhr\" (UniqueName: \"kubernetes.io/projected/7866608b-328f-49f4-b8c0-1cfb595e820d-kube-api-access-vsbhr\") pod \"isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx\" (UID: \"7866608b-328f-49f4-b8c0-1cfb595e820d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx"
Jul 02 12:53:35.564777 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:35.564754    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx"
Jul 02 12:53:35.706374 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:35.706353    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx"]
Jul 02 12:53:35.708242 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:53:35.708213    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7866608b_328f_49f4_b8c0_1cfb595e820d.slice/crio-474b885456f1b212d45300681f6164282658fe14c201e0443ad580bfb63ca855 WatchSource:0}: Error finding container 474b885456f1b212d45300681f6164282658fe14c201e0443ad580bfb63ca855: Status 404 returned error can't find the container with id 474b885456f1b212d45300681f6164282658fe14c201e0443ad580bfb63ca855
Jul 02 12:53:36.209867 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:36.209830    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx" event={"ID":"7866608b-328f-49f4-b8c0-1cfb595e820d","Type":"ContainerStarted","Data":"268fe4da72a8494e184943ed9149dff4c0fb7805e393329c304e841473597b05"}
Jul 02 12:53:36.210084 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:36.209903    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx" event={"ID":"7866608b-328f-49f4-b8c0-1cfb595e820d","Type":"ContainerStarted","Data":"474b885456f1b212d45300681f6164282658fe14c201e0443ad580bfb63ca855"}
Jul 02 12:53:36.212026 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:36.212000    2576 generic.go:356] "Generic (PLEG): container finished" podID="44bac380-51a6-475a-a4ce-a339dc65a434" containerID="86a584fb8f8e18ba9179f5aba8770bed00b8cddc444e55450e7b9799bb0f2050" exitCode=2
Jul 02 12:53:36.212157 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:36.212078    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns" event={"ID":"44bac380-51a6-475a-a4ce-a339dc65a434","Type":"ContainerDied","Data":"86a584fb8f8e18ba9179f5aba8770bed00b8cddc444e55450e7b9799bb0f2050"}
Jul 02 12:53:36.214245 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:36.214222    2576 generic.go:356] "Generic (PLEG): container finished" podID="c3450b92-228e-4217-8a8c-0f25d12e1938" containerID="4919045eaef12d8c729f8b4229f4de4198bfc714972932fd938f6894ae582348" exitCode=2
Jul 02 12:53:36.214337 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:36.214297    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz" event={"ID":"c3450b92-228e-4217-8a8c-0f25d12e1938","Type":"ContainerDied","Data":"4919045eaef12d8c729f8b4229f4de4198bfc714972932fd938f6894ae582348"}
Jul 02 12:53:36.745682 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:36.745649    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz" podUID="c3450b92-228e-4217-8a8c-0f25d12e1938" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Jul 02 12:53:36.750042 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:36.750021    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz" podUID="c3450b92-228e-4217-8a8c-0f25d12e1938" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 02 12:53:37.634086 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:37.634043    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns" podUID="44bac380-51a6-475a-a4ce-a339dc65a434" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Jul 02 12:53:37.644866 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:37.644835    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns" podUID="44bac380-51a6-475a-a4ce-a339dc65a434" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 02 12:53:38.866023 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:38.866001    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz"
Jul 02 12:53:39.021060 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:39.020972    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9hlc\" (UniqueName: \"kubernetes.io/projected/c3450b92-228e-4217-8a8c-0f25d12e1938-kube-api-access-t9hlc\") pod \"c3450b92-228e-4217-8a8c-0f25d12e1938\" (UID: \"c3450b92-228e-4217-8a8c-0f25d12e1938\") "
Jul 02 12:53:39.021060 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:39.021011    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3450b92-228e-4217-8a8c-0f25d12e1938-kserve-provision-location\") pod \"c3450b92-228e-4217-8a8c-0f25d12e1938\" (UID: \"c3450b92-228e-4217-8a8c-0f25d12e1938\") "
Jul 02 12:53:39.021060 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:39.021051    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3450b92-228e-4217-8a8c-0f25d12e1938-proxy-tls\") pod \"c3450b92-228e-4217-8a8c-0f25d12e1938\" (UID: \"c3450b92-228e-4217-8a8c-0f25d12e1938\") "
Jul 02 12:53:39.021348 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:39.021082    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3450b92-228e-4217-8a8c-0f25d12e1938-cabundle-cert\") pod \"c3450b92-228e-4217-8a8c-0f25d12e1938\" (UID: \"c3450b92-228e-4217-8a8c-0f25d12e1938\") "
Jul 02 12:53:39.021348 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:39.021166    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3450b92-228e-4217-8a8c-0f25d12e1938-kube-rbac-proxy-sar-config\") pod \"c3450b92-228e-4217-8a8c-0f25d12e1938\" (UID: \"c3450b92-228e-4217-8a8c-0f25d12e1938\") "
Jul 02 12:53:39.021968 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:39.021892    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3450b92-228e-4217-8a8c-0f25d12e1938-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c3450b92-228e-4217-8a8c-0f25d12e1938" (UID: "c3450b92-228e-4217-8a8c-0f25d12e1938"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 12:53:39.021968 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:39.021939    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3450b92-228e-4217-8a8c-0f25d12e1938-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c3450b92-228e-4217-8a8c-0f25d12e1938" (UID: "c3450b92-228e-4217-8a8c-0f25d12e1938"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:53:39.022166 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:39.021965    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3450b92-228e-4217-8a8c-0f25d12e1938-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c3450b92-228e-4217-8a8c-0f25d12e1938" (UID: "c3450b92-228e-4217-8a8c-0f25d12e1938"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:53:39.028974 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:39.024125    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3450b92-228e-4217-8a8c-0f25d12e1938-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c3450b92-228e-4217-8a8c-0f25d12e1938" (UID: "c3450b92-228e-4217-8a8c-0f25d12e1938"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 12:53:39.029524 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:39.029396    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3450b92-228e-4217-8a8c-0f25d12e1938-kube-api-access-t9hlc" (OuterVolumeSpecName: "kube-api-access-t9hlc") pod "c3450b92-228e-4217-8a8c-0f25d12e1938" (UID: "c3450b92-228e-4217-8a8c-0f25d12e1938"). InnerVolumeSpecName "kube-api-access-t9hlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 12:53:39.123061 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:39.123018    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3450b92-228e-4217-8a8c-0f25d12e1938-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:53:39.123061 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:39.123052    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9hlc\" (UniqueName: \"kubernetes.io/projected/c3450b92-228e-4217-8a8c-0f25d12e1938-kube-api-access-t9hlc\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:53:39.123061 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:39.123065    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3450b92-228e-4217-8a8c-0f25d12e1938-kserve-provision-location\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:53:39.123301 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:39.123079    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3450b92-228e-4217-8a8c-0f25d12e1938-proxy-tls\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:53:39.123301 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:39.123095    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3450b92-228e-4217-8a8c-0f25d12e1938-cabundle-cert\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:53:39.230404 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:39.230370    2576 generic.go:356] "Generic (PLEG): container finished" podID="c3450b92-228e-4217-8a8c-0f25d12e1938" containerID="c299e5a10c321c4fff62ffda42fdf110291775ae5a252410e739f7d6c7d7846d" exitCode=0
Jul 02 12:53:39.230404 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:39.230408    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz" event={"ID":"c3450b92-228e-4217-8a8c-0f25d12e1938","Type":"ContainerDied","Data":"c299e5a10c321c4fff62ffda42fdf110291775ae5a252410e739f7d6c7d7846d"}
Jul 02 12:53:39.230601 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:39.230443    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz"
Jul 02 12:53:39.230601 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:39.230482    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz" event={"ID":"c3450b92-228e-4217-8a8c-0f25d12e1938","Type":"ContainerDied","Data":"41e218c30cf4dee0ecaa1031115642b9fb97d0dff8efe45f3e3a960a4daca538"}
Jul 02 12:53:39.230601 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:39.230507    2576 scope.go:117] "RemoveContainer" containerID="4919045eaef12d8c729f8b4229f4de4198bfc714972932fd938f6894ae582348"
Jul 02 12:53:39.239335 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:39.239318    2576 scope.go:117] "RemoveContainer" containerID="c299e5a10c321c4fff62ffda42fdf110291775ae5a252410e739f7d6c7d7846d"
Jul 02 12:53:39.246614 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:39.246594    2576 scope.go:117] "RemoveContainer" containerID="ab6519185ff50a23b1784c7bc6a44c2ab1ca81846a3de373fce5e4533c9a6914"
Jul 02 12:53:39.254029 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:39.254009    2576 scope.go:117] "RemoveContainer" containerID="4919045eaef12d8c729f8b4229f4de4198bfc714972932fd938f6894ae582348"
Jul 02 12:53:39.254310 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:53:39.254291    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4919045eaef12d8c729f8b4229f4de4198bfc714972932fd938f6894ae582348\": container with ID starting with 4919045eaef12d8c729f8b4229f4de4198bfc714972932fd938f6894ae582348 not found: ID does not exist" containerID="4919045eaef12d8c729f8b4229f4de4198bfc714972932fd938f6894ae582348"
Jul 02 12:53:39.254382 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:39.254317    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4919045eaef12d8c729f8b4229f4de4198bfc714972932fd938f6894ae582348"} err="failed to get container status \"4919045eaef12d8c729f8b4229f4de4198bfc714972932fd938f6894ae582348\": rpc error: code = NotFound desc = could not find container \"4919045eaef12d8c729f8b4229f4de4198bfc714972932fd938f6894ae582348\": container with ID starting with 4919045eaef12d8c729f8b4229f4de4198bfc714972932fd938f6894ae582348 not found: ID does not exist"
Jul 02 12:53:39.254382 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:39.254337    2576 scope.go:117] "RemoveContainer" containerID="c299e5a10c321c4fff62ffda42fdf110291775ae5a252410e739f7d6c7d7846d"
Jul 02 12:53:39.254562 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:53:39.254543    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c299e5a10c321c4fff62ffda42fdf110291775ae5a252410e739f7d6c7d7846d\": container with ID starting with c299e5a10c321c4fff62ffda42fdf110291775ae5a252410e739f7d6c7d7846d not found: ID does not exist" containerID="c299e5a10c321c4fff62ffda42fdf110291775ae5a252410e739f7d6c7d7846d"
Jul 02 12:53:39.254609 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:39.254573    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c299e5a10c321c4fff62ffda42fdf110291775ae5a252410e739f7d6c7d7846d"} err="failed to get container status \"c299e5a10c321c4fff62ffda42fdf110291775ae5a252410e739f7d6c7d7846d\": rpc error: code = NotFound desc = could not find container \"c299e5a10c321c4fff62ffda42fdf110291775ae5a252410e739f7d6c7d7846d\": container with ID starting with c299e5a10c321c4fff62ffda42fdf110291775ae5a252410e739f7d6c7d7846d not found: ID does not exist"
Jul 02 12:53:39.254609 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:39.254599    2576 scope.go:117] "RemoveContainer" containerID="ab6519185ff50a23b1784c7bc6a44c2ab1ca81846a3de373fce5e4533c9a6914"
Jul 02 12:53:39.254846 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:53:39.254827    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab6519185ff50a23b1784c7bc6a44c2ab1ca81846a3de373fce5e4533c9a6914\": container with ID starting with ab6519185ff50a23b1784c7bc6a44c2ab1ca81846a3de373fce5e4533c9a6914 not found: ID does not exist" containerID="ab6519185ff50a23b1784c7bc6a44c2ab1ca81846a3de373fce5e4533c9a6914"
Jul 02 12:53:39.254895 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:39.254850    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab6519185ff50a23b1784c7bc6a44c2ab1ca81846a3de373fce5e4533c9a6914"} err="failed to get container status \"ab6519185ff50a23b1784c7bc6a44c2ab1ca81846a3de373fce5e4533c9a6914\": rpc error: code = NotFound desc = could not find container \"ab6519185ff50a23b1784c7bc6a44c2ab1ca81846a3de373fce5e4533c9a6914\": container with ID starting with ab6519185ff50a23b1784c7bc6a44c2ab1ca81846a3de373fce5e4533c9a6914 not found: ID does not exist"
Jul 02 12:53:39.255840 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:39.255821    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz"]
Jul 02 12:53:39.267152 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:39.267125    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-e1206-predictor-66d5466ccc-x6dvz"]
Jul 02 12:53:39.817713 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:39.817692    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns"
Jul 02 12:53:39.931309 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:39.931238    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44bac380-51a6-475a-a4ce-a339dc65a434-kserve-provision-location\") pod \"44bac380-51a6-475a-a4ce-a339dc65a434\" (UID: \"44bac380-51a6-475a-a4ce-a339dc65a434\") "
Jul 02 12:53:39.931309 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:39.931280    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44bac380-51a6-475a-a4ce-a339dc65a434-proxy-tls\") pod \"44bac380-51a6-475a-a4ce-a339dc65a434\" (UID: \"44bac380-51a6-475a-a4ce-a339dc65a434\") "
Jul 02 12:53:39.931809 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:39.931361    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z65m7\" (UniqueName: \"kubernetes.io/projected/44bac380-51a6-475a-a4ce-a339dc65a434-kube-api-access-z65m7\") pod \"44bac380-51a6-475a-a4ce-a339dc65a434\" (UID: \"44bac380-51a6-475a-a4ce-a339dc65a434\") "
Jul 02 12:53:39.931809 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:39.931436    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44bac380-51a6-475a-a4ce-a339dc65a434-kube-rbac-proxy-sar-config\") pod \"44bac380-51a6-475a-a4ce-a339dc65a434\" (UID: \"44bac380-51a6-475a-a4ce-a339dc65a434\") "
Jul 02 12:53:39.931809 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:39.931497    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44bac380-51a6-475a-a4ce-a339dc65a434-cabundle-cert\") pod \"44bac380-51a6-475a-a4ce-a339dc65a434\" (UID: \"44bac380-51a6-475a-a4ce-a339dc65a434\") "
Jul 02 12:53:39.931809 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:39.931538    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44bac380-51a6-475a-a4ce-a339dc65a434-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "44bac380-51a6-475a-a4ce-a339dc65a434" (UID: "44bac380-51a6-475a-a4ce-a339dc65a434"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 12:53:39.931809 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:39.931778    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44bac380-51a6-475a-a4ce-a339dc65a434-kserve-provision-location\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:53:39.932094 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:39.931942    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44bac380-51a6-475a-a4ce-a339dc65a434-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "44bac380-51a6-475a-a4ce-a339dc65a434" (UID: "44bac380-51a6-475a-a4ce-a339dc65a434"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:53:39.932094 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:39.931978    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44bac380-51a6-475a-a4ce-a339dc65a434-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "44bac380-51a6-475a-a4ce-a339dc65a434" (UID: "44bac380-51a6-475a-a4ce-a339dc65a434"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:53:39.933458 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:39.933433    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44bac380-51a6-475a-a4ce-a339dc65a434-kube-api-access-z65m7" (OuterVolumeSpecName: "kube-api-access-z65m7") pod "44bac380-51a6-475a-a4ce-a339dc65a434" (UID: "44bac380-51a6-475a-a4ce-a339dc65a434"). InnerVolumeSpecName "kube-api-access-z65m7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 12:53:39.933458 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:39.933446    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44bac380-51a6-475a-a4ce-a339dc65a434-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "44bac380-51a6-475a-a4ce-a339dc65a434" (UID: "44bac380-51a6-475a-a4ce-a339dc65a434"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 12:53:40.033174 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:40.033148    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44bac380-51a6-475a-a4ce-a339dc65a434-cabundle-cert\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:53:40.033174 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:40.033172    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44bac380-51a6-475a-a4ce-a339dc65a434-proxy-tls\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:53:40.033355 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:40.033183    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z65m7\" (UniqueName: \"kubernetes.io/projected/44bac380-51a6-475a-a4ce-a339dc65a434-kube-api-access-z65m7\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:53:40.033355 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:40.033194    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44bac380-51a6-475a-a4ce-a339dc65a434-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:53:40.236959 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:40.236855    2576 generic.go:356] "Generic (PLEG): container finished" podID="7866608b-328f-49f4-b8c0-1cfb595e820d" containerID="268fe4da72a8494e184943ed9149dff4c0fb7805e393329c304e841473597b05" exitCode=0
Jul 02 12:53:40.237124 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:40.236949    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx" event={"ID":"7866608b-328f-49f4-b8c0-1cfb595e820d","Type":"ContainerDied","Data":"268fe4da72a8494e184943ed9149dff4c0fb7805e393329c304e841473597b05"}
Jul 02 12:53:40.238974 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:40.238952    2576 generic.go:356] "Generic (PLEG): container finished" podID="44bac380-51a6-475a-a4ce-a339dc65a434" containerID="3f7c1d27c709df18b858438451fb636cf8789b3b3ff250ddfab055b1f8c85a12" exitCode=0
Jul 02 12:53:40.239084 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:40.239017    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns"
Jul 02 12:53:40.239084 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:40.239028    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns" event={"ID":"44bac380-51a6-475a-a4ce-a339dc65a434","Type":"ContainerDied","Data":"3f7c1d27c709df18b858438451fb636cf8789b3b3ff250ddfab055b1f8c85a12"}
Jul 02 12:53:40.239171 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:40.239086    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns" event={"ID":"44bac380-51a6-475a-a4ce-a339dc65a434","Type":"ContainerDied","Data":"83e0403aca8f363ed98c8c63eb94fcb5b1080dca815d00d871468da9a74b1435"}
Jul 02 12:53:40.239171 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:40.239127    2576 scope.go:117] "RemoveContainer" containerID="86a584fb8f8e18ba9179f5aba8770bed00b8cddc444e55450e7b9799bb0f2050"
Jul 02 12:53:40.247957 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:40.247941    2576 scope.go:117] "RemoveContainer" containerID="3f7c1d27c709df18b858438451fb636cf8789b3b3ff250ddfab055b1f8c85a12"
Jul 02 12:53:40.255516 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:40.255492    2576 scope.go:117] "RemoveContainer" containerID="c65ef4f92a4a54f26c13b3038f566cf710f4f5f06b8171672083e7f840006ac5"
Jul 02 12:53:40.268189 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:40.268175    2576 scope.go:117] "RemoveContainer" containerID="86a584fb8f8e18ba9179f5aba8770bed00b8cddc444e55450e7b9799bb0f2050"
Jul 02 12:53:40.268441 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:53:40.268422    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86a584fb8f8e18ba9179f5aba8770bed00b8cddc444e55450e7b9799bb0f2050\": container with ID starting with 86a584fb8f8e18ba9179f5aba8770bed00b8cddc444e55450e7b9799bb0f2050 not found: ID does not exist" containerID="86a584fb8f8e18ba9179f5aba8770bed00b8cddc444e55450e7b9799bb0f2050"
Jul 02 12:53:40.268488 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:40.268450    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86a584fb8f8e18ba9179f5aba8770bed00b8cddc444e55450e7b9799bb0f2050"} err="failed to get container status \"86a584fb8f8e18ba9179f5aba8770bed00b8cddc444e55450e7b9799bb0f2050\": rpc error: code = NotFound desc = could not find container \"86a584fb8f8e18ba9179f5aba8770bed00b8cddc444e55450e7b9799bb0f2050\": container with ID starting with 86a584fb8f8e18ba9179f5aba8770bed00b8cddc444e55450e7b9799bb0f2050 not found: ID does not exist"
Jul 02 12:53:40.268488 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:40.268467    2576 scope.go:117] "RemoveContainer" containerID="3f7c1d27c709df18b858438451fb636cf8789b3b3ff250ddfab055b1f8c85a12"
Jul 02 12:53:40.268706 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:53:40.268689    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f7c1d27c709df18b858438451fb636cf8789b3b3ff250ddfab055b1f8c85a12\": container with ID starting with 3f7c1d27c709df18b858438451fb636cf8789b3b3ff250ddfab055b1f8c85a12 not found: ID does not exist" containerID="3f7c1d27c709df18b858438451fb636cf8789b3b3ff250ddfab055b1f8c85a12"
Jul 02 12:53:40.268746 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:40.268712    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f7c1d27c709df18b858438451fb636cf8789b3b3ff250ddfab055b1f8c85a12"} err="failed to get container status \"3f7c1d27c709df18b858438451fb636cf8789b3b3ff250ddfab055b1f8c85a12\": rpc error: code = NotFound desc = could not find container \"3f7c1d27c709df18b858438451fb636cf8789b3b3ff250ddfab055b1f8c85a12\": container with ID starting with 3f7c1d27c709df18b858438451fb636cf8789b3b3ff250ddfab055b1f8c85a12 not found: ID does not exist"
Jul 02 12:53:40.268746 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:40.268727    2576 scope.go:117] "RemoveContainer" containerID="c65ef4f92a4a54f26c13b3038f566cf710f4f5f06b8171672083e7f840006ac5"
Jul 02 12:53:40.269006 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:53:40.268986    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c65ef4f92a4a54f26c13b3038f566cf710f4f5f06b8171672083e7f840006ac5\": container with ID starting with c65ef4f92a4a54f26c13b3038f566cf710f4f5f06b8171672083e7f840006ac5 not found: ID does not exist" containerID="c65ef4f92a4a54f26c13b3038f566cf710f4f5f06b8171672083e7f840006ac5"
Jul 02 12:53:40.269081 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:40.269007    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c65ef4f92a4a54f26c13b3038f566cf710f4f5f06b8171672083e7f840006ac5"} err="failed to get container status \"c65ef4f92a4a54f26c13b3038f566cf710f4f5f06b8171672083e7f840006ac5\": rpc error: code = NotFound desc = could not find container \"c65ef4f92a4a54f26c13b3038f566cf710f4f5f06b8171672083e7f840006ac5\": container with ID starting with c65ef4f92a4a54f26c13b3038f566cf710f4f5f06b8171672083e7f840006ac5 not found: ID does not exist"
Jul 02 12:53:40.278047 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:40.278022    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns"]
Jul 02 12:53:40.287141 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:40.287122    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-e1206-predictor-666bd7dbf4-lqjns"]
Jul 02 12:53:40.863161 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:40.863128    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44bac380-51a6-475a-a4ce-a339dc65a434" path="/var/lib/kubelet/pods/44bac380-51a6-475a-a4ce-a339dc65a434/volumes"
Jul 02 12:53:40.863633 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:40.863618    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3450b92-228e-4217-8a8c-0f25d12e1938" path="/var/lib/kubelet/pods/c3450b92-228e-4217-8a8c-0f25d12e1938/volumes"
Jul 02 12:53:41.249949 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:41.249902    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx" event={"ID":"7866608b-328f-49f4-b8c0-1cfb595e820d","Type":"ContainerStarted","Data":"03e78bc223850f3055642ac47bfc89772cec831931009f581b87d502d8aa10e5"}
Jul 02 12:53:41.250411 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:41.249982    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx" event={"ID":"7866608b-328f-49f4-b8c0-1cfb595e820d","Type":"ContainerStarted","Data":"94a99676ceafc4435c3fdaf4413339ed991239a1417b4b7fd2c095cc0a337e23"}
Jul 02 12:53:41.250411 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:41.250296    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx"
Jul 02 12:53:41.250411 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:41.250353    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx"
Jul 02 12:53:41.251955 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:41.251929    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx" podUID="7866608b-328f-49f4-b8c0-1cfb595e820d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 02 12:53:41.290070 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:41.290020    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx" podStartSLOduration=2.653163464 podStartE2EDuration="6.290006652s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.636843188s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:53:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:53:41.288334794 +0000 UTC m=+909.895797817" watchObservedRunningTime="2026-07-02 12:53:41.290006652 +0000 UTC m=+909.897469732"
Jul 02 12:53:42.255732 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:42.255695    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx" podUID="7866608b-328f-49f4-b8c0-1cfb595e820d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 02 12:53:47.260440 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:47.260411    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx"
Jul 02 12:53:47.260903 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:47.260868    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx" podUID="7866608b-328f-49f4-b8c0-1cfb595e820d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 02 12:53:57.261467 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:53:57.261428    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx" podUID="7866608b-328f-49f4-b8c0-1cfb595e820d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 02 12:54:07.260955 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:54:07.260875    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx" podUID="7866608b-328f-49f4-b8c0-1cfb595e820d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 02 12:54:17.261671 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:54:17.261625    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx" podUID="7866608b-328f-49f4-b8c0-1cfb595e820d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 02 12:54:27.261637 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:54:27.261590    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx" podUID="7866608b-328f-49f4-b8c0-1cfb595e820d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 02 12:54:37.261845 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:54:37.261803    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx" podUID="7866608b-328f-49f4-b8c0-1cfb595e820d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 02 12:54:47.261258 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:54:47.261220    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx" podUID="7866608b-328f-49f4-b8c0-1cfb595e820d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 02 12:54:57.262187 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:54:57.262156    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx"
Jul 02 12:55:25.675819 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:25.675781    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv"]
Jul 02 12:55:25.676426 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:25.676404    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3450b92-228e-4217-8a8c-0f25d12e1938" containerName="storage-initializer"
Jul 02 12:55:25.676522 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:25.676430    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3450b92-228e-4217-8a8c-0f25d12e1938" containerName="storage-initializer"
Jul 02 12:55:25.676522 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:25.676449    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44bac380-51a6-475a-a4ce-a339dc65a434" containerName="kserve-container"
Jul 02 12:55:25.676522 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:25.676458    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="44bac380-51a6-475a-a4ce-a339dc65a434" containerName="kserve-container"
Jul 02 12:55:25.676522 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:25.676471    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44bac380-51a6-475a-a4ce-a339dc65a434" containerName="storage-initializer"
Jul 02 12:55:25.676522 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:25.676480    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="44bac380-51a6-475a-a4ce-a339dc65a434" containerName="storage-initializer"
Jul 02 12:55:25.676522 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:25.676502    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44bac380-51a6-475a-a4ce-a339dc65a434" containerName="kube-rbac-proxy"
Jul 02 12:55:25.676522 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:25.676510    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="44bac380-51a6-475a-a4ce-a339dc65a434" containerName="kube-rbac-proxy"
Jul 02 12:55:25.676522 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:25.676522    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3450b92-228e-4217-8a8c-0f25d12e1938" containerName="kube-rbac-proxy"
Jul 02 12:55:25.676803 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:25.676531    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3450b92-228e-4217-8a8c-0f25d12e1938" containerName="kube-rbac-proxy"
Jul 02 12:55:25.676803 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:25.676544    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3450b92-228e-4217-8a8c-0f25d12e1938" containerName="kserve-container"
Jul 02 12:55:25.676803 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:25.676552    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3450b92-228e-4217-8a8c-0f25d12e1938" containerName="kserve-container"
Jul 02 12:55:25.676803 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:25.676680    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3450b92-228e-4217-8a8c-0f25d12e1938" containerName="kube-rbac-proxy"
Jul 02 12:55:25.676803 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:25.676695    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="44bac380-51a6-475a-a4ce-a339dc65a434" containerName="kserve-container"
Jul 02 12:55:25.676803 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:25.676709    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="44bac380-51a6-475a-a4ce-a339dc65a434" containerName="kube-rbac-proxy"
Jul 02 12:55:25.676803 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:25.676732    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3450b92-228e-4217-8a8c-0f25d12e1938" containerName="kserve-container"
Jul 02 12:55:25.680659 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:25.680639    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv"
Jul 02 12:55:25.686211 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:25.686168    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-c99c3-predictor-serving-cert\""
Jul 02 12:55:25.686334 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:25.686211    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-c99c3-kube-rbac-proxy-sar-config\""
Jul 02 12:55:25.701311 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:25.701289    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv"]
Jul 02 12:55:25.710108 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:25.710082    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx"]
Jul 02 12:55:25.710434 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:25.710373    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx" podUID="7866608b-328f-49f4-b8c0-1cfb595e820d" containerName="kserve-container" containerID="cri-o://94a99676ceafc4435c3fdaf4413339ed991239a1417b4b7fd2c095cc0a337e23" gracePeriod=30
Jul 02 12:55:25.710549 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:25.710441    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx" podUID="7866608b-328f-49f4-b8c0-1cfb595e820d" containerName="kube-rbac-proxy" containerID="cri-o://03e78bc223850f3055642ac47bfc89772cec831931009f581b87d502d8aa10e5" gracePeriod=30
Jul 02 12:55:25.779710 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:25.779675    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxndt\" (UniqueName: \"kubernetes.io/projected/15064fb3-c271-4243-9c62-6bb1be858fc2-kube-api-access-xxndt\") pod \"message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv\" (UID: \"15064fb3-c271-4243-9c62-6bb1be858fc2\") " pod="kserve-ci-e2e-test/message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv"
Jul 02 12:55:25.779882 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:25.779749    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15064fb3-c271-4243-9c62-6bb1be858fc2-proxy-tls\") pod \"message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv\" (UID: \"15064fb3-c271-4243-9c62-6bb1be858fc2\") " pod="kserve-ci-e2e-test/message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv"
Jul 02 12:55:25.779882 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:25.779768    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15064fb3-c271-4243-9c62-6bb1be858fc2-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv\" (UID: \"15064fb3-c271-4243-9c62-6bb1be858fc2\") " pod="kserve-ci-e2e-test/message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv"
Jul 02 12:55:25.881330 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:25.881303    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15064fb3-c271-4243-9c62-6bb1be858fc2-proxy-tls\") pod \"message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv\" (UID: \"15064fb3-c271-4243-9c62-6bb1be858fc2\") " pod="kserve-ci-e2e-test/message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv"
Jul 02 12:55:25.881466 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:25.881338    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15064fb3-c271-4243-9c62-6bb1be858fc2-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv\" (UID: \"15064fb3-c271-4243-9c62-6bb1be858fc2\") " pod="kserve-ci-e2e-test/message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv"
Jul 02 12:55:25.881466 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:25.881399    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxndt\" (UniqueName: \"kubernetes.io/projected/15064fb3-c271-4243-9c62-6bb1be858fc2-kube-api-access-xxndt\") pod \"message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv\" (UID: \"15064fb3-c271-4243-9c62-6bb1be858fc2\") " pod="kserve-ci-e2e-test/message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv"
Jul 02 12:55:25.881466 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:55:25.881429    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-c99c3-predictor-serving-cert: secret "message-dumper-raw-c99c3-predictor-serving-cert" not found
Jul 02 12:55:25.881589 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:55:25.881505    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15064fb3-c271-4243-9c62-6bb1be858fc2-proxy-tls podName:15064fb3-c271-4243-9c62-6bb1be858fc2 nodeName:}" failed. No retries permitted until 2026-07-02 12:55:26.381483802 +0000 UTC m=+1014.988946803 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/15064fb3-c271-4243-9c62-6bb1be858fc2-proxy-tls") pod "message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv" (UID: "15064fb3-c271-4243-9c62-6bb1be858fc2") : secret "message-dumper-raw-c99c3-predictor-serving-cert" not found
Jul 02 12:55:25.882801 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:25.882758    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15064fb3-c271-4243-9c62-6bb1be858fc2-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv\" (UID: \"15064fb3-c271-4243-9c62-6bb1be858fc2\") " pod="kserve-ci-e2e-test/message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv"
Jul 02 12:55:25.890653 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:25.890628    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxndt\" (UniqueName: \"kubernetes.io/projected/15064fb3-c271-4243-9c62-6bb1be858fc2-kube-api-access-xxndt\") pod \"message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv\" (UID: \"15064fb3-c271-4243-9c62-6bb1be858fc2\") " pod="kserve-ci-e2e-test/message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv"
Jul 02 12:55:26.384493 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:26.384450    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15064fb3-c271-4243-9c62-6bb1be858fc2-proxy-tls\") pod \"message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv\" (UID: \"15064fb3-c271-4243-9c62-6bb1be858fc2\") " pod="kserve-ci-e2e-test/message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv"
Jul 02 12:55:26.386873 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:26.386853    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15064fb3-c271-4243-9c62-6bb1be858fc2-proxy-tls\") pod \"message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv\" (UID: \"15064fb3-c271-4243-9c62-6bb1be858fc2\") " pod="kserve-ci-e2e-test/message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv"
Jul 02 12:55:26.595203 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:26.595168    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv"
Jul 02 12:55:26.726370 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:26.726340    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv"]
Jul 02 12:55:26.727801 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:55:26.727770    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15064fb3_c271_4243_9c62_6bb1be858fc2.slice/crio-4b37e00eda5e400da8deb420579477103881b8c087d20f2e0c6cbcd8ae9eb15f WatchSource:0}: Error finding container 4b37e00eda5e400da8deb420579477103881b8c087d20f2e0c6cbcd8ae9eb15f: Status 404 returned error can't find the container with id 4b37e00eda5e400da8deb420579477103881b8c087d20f2e0c6cbcd8ae9eb15f
Jul 02 12:55:26.729754 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:26.729737    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 12:55:26.749683 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:26.749657    2576 generic.go:356] "Generic (PLEG): container finished" podID="7866608b-328f-49f4-b8c0-1cfb595e820d" containerID="03e78bc223850f3055642ac47bfc89772cec831931009f581b87d502d8aa10e5" exitCode=2
Jul 02 12:55:26.749798 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:26.749728    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx" event={"ID":"7866608b-328f-49f4-b8c0-1cfb595e820d","Type":"ContainerDied","Data":"03e78bc223850f3055642ac47bfc89772cec831931009f581b87d502d8aa10e5"}
Jul 02 12:55:26.750885 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:26.750868    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv" event={"ID":"15064fb3-c271-4243-9c62-6bb1be858fc2","Type":"ContainerStarted","Data":"4b37e00eda5e400da8deb420579477103881b8c087d20f2e0c6cbcd8ae9eb15f"}
Jul 02 12:55:27.257075 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:27.257033    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx" podUID="7866608b-328f-49f4-b8c0-1cfb595e820d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Jul 02 12:55:27.261408 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:27.261377    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx" podUID="7866608b-328f-49f4-b8c0-1cfb595e820d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 02 12:55:28.763650 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:28.763612    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv" event={"ID":"15064fb3-c271-4243-9c62-6bb1be858fc2","Type":"ContainerStarted","Data":"87d148ddcef96fab893881c1d18b862187b169d884fa65ffaadf87a893bbd3cd"}
Jul 02 12:55:28.764063 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:28.763765    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv"
Jul 02 12:55:28.764063 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:28.763813    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv" event={"ID":"15064fb3-c271-4243-9c62-6bb1be858fc2","Type":"ContainerStarted","Data":"48aa19715c13ef82c8b458b7a57a0a140f2accc4d6e8bf9c461a910fe1dc0648"}
Jul 02 12:55:28.764063 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:28.763848    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv"
Jul 02 12:55:28.765436 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:28.765417    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv"
Jul 02 12:55:28.789667 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:28.789612    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv" podStartSLOduration=2.503739079 podStartE2EDuration="3.789596958s" totalImagesPullingTime="1.285857879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:55:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:55:28.78834657 +0000 UTC m=+1017.395809594" watchObservedRunningTime="2026-07-02 12:55:28.789596958 +0000 UTC m=+1017.397059980"
Jul 02 12:55:30.365632 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:30.365606    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx"
Jul 02 12:55:30.419244 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:30.419214    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7866608b-328f-49f4-b8c0-1cfb595e820d-kserve-provision-location\") pod \"7866608b-328f-49f4-b8c0-1cfb595e820d\" (UID: \"7866608b-328f-49f4-b8c0-1cfb595e820d\") "
Jul 02 12:55:30.419410 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:30.419252    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7866608b-328f-49f4-b8c0-1cfb595e820d-kube-rbac-proxy-sar-config\") pod \"7866608b-328f-49f4-b8c0-1cfb595e820d\" (UID: \"7866608b-328f-49f4-b8c0-1cfb595e820d\") "
Jul 02 12:55:30.419410 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:30.419312    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vsbhr\" (UniqueName: \"kubernetes.io/projected/7866608b-328f-49f4-b8c0-1cfb595e820d-kube-api-access-vsbhr\") pod \"7866608b-328f-49f4-b8c0-1cfb595e820d\" (UID: \"7866608b-328f-49f4-b8c0-1cfb595e820d\") "
Jul 02 12:55:30.419410 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:30.419343    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7866608b-328f-49f4-b8c0-1cfb595e820d-cabundle-cert\") pod \"7866608b-328f-49f4-b8c0-1cfb595e820d\" (UID: \"7866608b-328f-49f4-b8c0-1cfb595e820d\") "
Jul 02 12:55:30.419410 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:30.419368    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7866608b-328f-49f4-b8c0-1cfb595e820d-proxy-tls\") pod \"7866608b-328f-49f4-b8c0-1cfb595e820d\" (UID: \"7866608b-328f-49f4-b8c0-1cfb595e820d\") "
Jul 02 12:55:30.419621 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:30.419529    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7866608b-328f-49f4-b8c0-1cfb595e820d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7866608b-328f-49f4-b8c0-1cfb595e820d" (UID: "7866608b-328f-49f4-b8c0-1cfb595e820d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 12:55:30.419687 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:30.419662    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7866608b-328f-49f4-b8c0-1cfb595e820d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7866608b-328f-49f4-b8c0-1cfb595e820d" (UID: "7866608b-328f-49f4-b8c0-1cfb595e820d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:55:30.419752 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:30.419684    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7866608b-328f-49f4-b8c0-1cfb595e820d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7866608b-328f-49f4-b8c0-1cfb595e820d" (UID: "7866608b-328f-49f4-b8c0-1cfb595e820d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:55:30.419752 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:30.419722    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7866608b-328f-49f4-b8c0-1cfb595e820d-kserve-provision-location\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:55:30.421479 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:30.421458    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7866608b-328f-49f4-b8c0-1cfb595e820d-kube-api-access-vsbhr" (OuterVolumeSpecName: "kube-api-access-vsbhr") pod "7866608b-328f-49f4-b8c0-1cfb595e820d" (UID: "7866608b-328f-49f4-b8c0-1cfb595e820d"). InnerVolumeSpecName "kube-api-access-vsbhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 12:55:30.421479 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:30.421471    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7866608b-328f-49f4-b8c0-1cfb595e820d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7866608b-328f-49f4-b8c0-1cfb595e820d" (UID: "7866608b-328f-49f4-b8c0-1cfb595e820d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 12:55:30.520938 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:30.520804    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7866608b-328f-49f4-b8c0-1cfb595e820d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:55:30.520938 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:30.520840    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsbhr\" (UniqueName: \"kubernetes.io/projected/7866608b-328f-49f4-b8c0-1cfb595e820d-kube-api-access-vsbhr\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:55:30.520938 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:30.520853    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7866608b-328f-49f4-b8c0-1cfb595e820d-cabundle-cert\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:55:30.520938 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:30.520865    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7866608b-328f-49f4-b8c0-1cfb595e820d-proxy-tls\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:55:30.778162 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:30.778074    2576 generic.go:356] "Generic (PLEG): container finished" podID="7866608b-328f-49f4-b8c0-1cfb595e820d" containerID="94a99676ceafc4435c3fdaf4413339ed991239a1417b4b7fd2c095cc0a337e23" exitCode=0
Jul 02 12:55:30.778162 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:30.778137    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx"
Jul 02 12:55:30.778162 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:30.778149    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx" event={"ID":"7866608b-328f-49f4-b8c0-1cfb595e820d","Type":"ContainerDied","Data":"94a99676ceafc4435c3fdaf4413339ed991239a1417b4b7fd2c095cc0a337e23"}
Jul 02 12:55:30.778436 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:30.778207    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx" event={"ID":"7866608b-328f-49f4-b8c0-1cfb595e820d","Type":"ContainerDied","Data":"474b885456f1b212d45300681f6164282658fe14c201e0443ad580bfb63ca855"}
Jul 02 12:55:30.778436 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:30.778234    2576 scope.go:117] "RemoveContainer" containerID="03e78bc223850f3055642ac47bfc89772cec831931009f581b87d502d8aa10e5"
Jul 02 12:55:30.787500 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:30.787480    2576 scope.go:117] "RemoveContainer" containerID="94a99676ceafc4435c3fdaf4413339ed991239a1417b4b7fd2c095cc0a337e23"
Jul 02 12:55:30.795155 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:30.795139    2576 scope.go:117] "RemoveContainer" containerID="268fe4da72a8494e184943ed9149dff4c0fb7805e393329c304e841473597b05"
Jul 02 12:55:30.802453 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:30.802431    2576 scope.go:117] "RemoveContainer" containerID="03e78bc223850f3055642ac47bfc89772cec831931009f581b87d502d8aa10e5"
Jul 02 12:55:30.802715 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:55:30.802693    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03e78bc223850f3055642ac47bfc89772cec831931009f581b87d502d8aa10e5\": container with ID starting with 03e78bc223850f3055642ac47bfc89772cec831931009f581b87d502d8aa10e5 not found: ID does not exist" containerID="03e78bc223850f3055642ac47bfc89772cec831931009f581b87d502d8aa10e5"
Jul 02 12:55:30.802754 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:30.802725    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03e78bc223850f3055642ac47bfc89772cec831931009f581b87d502d8aa10e5"} err="failed to get container status \"03e78bc223850f3055642ac47bfc89772cec831931009f581b87d502d8aa10e5\": rpc error: code = NotFound desc = could not find container \"03e78bc223850f3055642ac47bfc89772cec831931009f581b87d502d8aa10e5\": container with ID starting with 03e78bc223850f3055642ac47bfc89772cec831931009f581b87d502d8aa10e5 not found: ID does not exist"
Jul 02 12:55:30.802754 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:30.802743    2576 scope.go:117] "RemoveContainer" containerID="94a99676ceafc4435c3fdaf4413339ed991239a1417b4b7fd2c095cc0a337e23"
Jul 02 12:55:30.802993 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:55:30.802975    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94a99676ceafc4435c3fdaf4413339ed991239a1417b4b7fd2c095cc0a337e23\": container with ID starting with 94a99676ceafc4435c3fdaf4413339ed991239a1417b4b7fd2c095cc0a337e23 not found: ID does not exist" containerID="94a99676ceafc4435c3fdaf4413339ed991239a1417b4b7fd2c095cc0a337e23"
Jul 02 12:55:30.803046 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:30.803000    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94a99676ceafc4435c3fdaf4413339ed991239a1417b4b7fd2c095cc0a337e23"} err="failed to get container status \"94a99676ceafc4435c3fdaf4413339ed991239a1417b4b7fd2c095cc0a337e23\": rpc error: code = NotFound desc = could not find container \"94a99676ceafc4435c3fdaf4413339ed991239a1417b4b7fd2c095cc0a337e23\": container with ID starting with 94a99676ceafc4435c3fdaf4413339ed991239a1417b4b7fd2c095cc0a337e23 not found: ID does not exist"
Jul 02 12:55:30.803046 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:30.803016    2576 scope.go:117] "RemoveContainer" containerID="268fe4da72a8494e184943ed9149dff4c0fb7805e393329c304e841473597b05"
Jul 02 12:55:30.803256 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:55:30.803233    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"268fe4da72a8494e184943ed9149dff4c0fb7805e393329c304e841473597b05\": container with ID starting with 268fe4da72a8494e184943ed9149dff4c0fb7805e393329c304e841473597b05 not found: ID does not exist" containerID="268fe4da72a8494e184943ed9149dff4c0fb7805e393329c304e841473597b05"
Jul 02 12:55:30.803297 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:30.803259    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"268fe4da72a8494e184943ed9149dff4c0fb7805e393329c304e841473597b05"} err="failed to get container status \"268fe4da72a8494e184943ed9149dff4c0fb7805e393329c304e841473597b05\": rpc error: code = NotFound desc = could not find container \"268fe4da72a8494e184943ed9149dff4c0fb7805e393329c304e841473597b05\": container with ID starting with 268fe4da72a8494e184943ed9149dff4c0fb7805e393329c304e841473597b05 not found: ID does not exist"
Jul 02 12:55:30.808622 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:30.808603    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx"]
Jul 02 12:55:30.816410 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:30.816388    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-9587b-predictor-54f88c8746-wx5lx"]
Jul 02 12:55:30.864010 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:30.863985    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7866608b-328f-49f4-b8c0-1cfb595e820d" path="/var/lib/kubelet/pods/7866608b-328f-49f4-b8c0-1cfb595e820d/volumes"
Jul 02 12:55:35.782883 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:35.782854    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv"
Jul 02 12:55:45.659291 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:45.659250    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l"]
Jul 02 12:55:45.659801 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:45.659782    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7866608b-328f-49f4-b8c0-1cfb595e820d" containerName="kube-rbac-proxy"
Jul 02 12:55:45.659883 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:45.659804    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7866608b-328f-49f4-b8c0-1cfb595e820d" containerName="kube-rbac-proxy"
Jul 02 12:55:45.659883 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:45.659826    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7866608b-328f-49f4-b8c0-1cfb595e820d" containerName="kserve-container"
Jul 02 12:55:45.659883 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:45.659834    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7866608b-328f-49f4-b8c0-1cfb595e820d" containerName="kserve-container"
Jul 02 12:55:45.659883 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:45.659868    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7866608b-328f-49f4-b8c0-1cfb595e820d" containerName="storage-initializer"
Jul 02 12:55:45.659883 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:45.659877    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7866608b-328f-49f4-b8c0-1cfb595e820d" containerName="storage-initializer"
Jul 02 12:55:45.660158 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:45.660013    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7866608b-328f-49f4-b8c0-1cfb595e820d" containerName="kserve-container"
Jul 02 12:55:45.660158 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:45.660033    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7866608b-328f-49f4-b8c0-1cfb595e820d" containerName="kube-rbac-proxy"
Jul 02 12:55:45.665006 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:45.664987    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l"
Jul 02 12:55:45.669159 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:45.669140    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 12:55:45.669455 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:45.669440    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-c99c3-kube-rbac-proxy-sar-config\""
Jul 02 12:55:45.669538 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:45.669442    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-c99c3-predictor-serving-cert\""
Jul 02 12:55:45.669799 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:45.669784    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 12:55:45.683779 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:45.683758    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l"]
Jul 02 12:55:45.753018 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:45.752983    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6eb83fab-c7af-40e5-8059-88ff3b38cfcb-proxy-tls\") pod \"isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l\" (UID: \"6eb83fab-c7af-40e5-8059-88ff3b38cfcb\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l"
Jul 02 12:55:45.753154 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:45.753038    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6eb83fab-c7af-40e5-8059-88ff3b38cfcb-kserve-provision-location\") pod \"isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l\" (UID: \"6eb83fab-c7af-40e5-8059-88ff3b38cfcb\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l"
Jul 02 12:55:45.753205 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:45.753142    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc22c\" (UniqueName: \"kubernetes.io/projected/6eb83fab-c7af-40e5-8059-88ff3b38cfcb-kube-api-access-pc22c\") pod \"isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l\" (UID: \"6eb83fab-c7af-40e5-8059-88ff3b38cfcb\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l"
Jul 02 12:55:45.753205 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:45.753178    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6eb83fab-c7af-40e5-8059-88ff3b38cfcb-cabundle-cert\") pod \"isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l\" (UID: \"6eb83fab-c7af-40e5-8059-88ff3b38cfcb\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l"
Jul 02 12:55:45.753290 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:45.753222    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6eb83fab-c7af-40e5-8059-88ff3b38cfcb-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l\" (UID: \"6eb83fab-c7af-40e5-8059-88ff3b38cfcb\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l"
Jul 02 12:55:45.854233 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:45.854195    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6eb83fab-c7af-40e5-8059-88ff3b38cfcb-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l\" (UID: \"6eb83fab-c7af-40e5-8059-88ff3b38cfcb\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l"
Jul 02 12:55:45.854423 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:45.854255    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6eb83fab-c7af-40e5-8059-88ff3b38cfcb-proxy-tls\") pod \"isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l\" (UID: \"6eb83fab-c7af-40e5-8059-88ff3b38cfcb\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l"
Jul 02 12:55:45.854423 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:45.854288    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6eb83fab-c7af-40e5-8059-88ff3b38cfcb-kserve-provision-location\") pod \"isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l\" (UID: \"6eb83fab-c7af-40e5-8059-88ff3b38cfcb\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l"
Jul 02 12:55:45.854423 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:45.854321    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc22c\" (UniqueName: \"kubernetes.io/projected/6eb83fab-c7af-40e5-8059-88ff3b38cfcb-kube-api-access-pc22c\") pod \"isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l\" (UID: \"6eb83fab-c7af-40e5-8059-88ff3b38cfcb\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l"
Jul 02 12:55:45.854423 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:45.854342    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6eb83fab-c7af-40e5-8059-88ff3b38cfcb-cabundle-cert\") pod \"isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l\" (UID: \"6eb83fab-c7af-40e5-8059-88ff3b38cfcb\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l"
Jul 02 12:55:45.854778 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:45.854755    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6eb83fab-c7af-40e5-8059-88ff3b38cfcb-kserve-provision-location\") pod \"isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l\" (UID: \"6eb83fab-c7af-40e5-8059-88ff3b38cfcb\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l"
Jul 02 12:55:45.854972 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:45.854945    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6eb83fab-c7af-40e5-8059-88ff3b38cfcb-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l\" (UID: \"6eb83fab-c7af-40e5-8059-88ff3b38cfcb\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l"
Jul 02 12:55:45.854972 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:45.854956    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6eb83fab-c7af-40e5-8059-88ff3b38cfcb-cabundle-cert\") pod \"isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l\" (UID: \"6eb83fab-c7af-40e5-8059-88ff3b38cfcb\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l"
Jul 02 12:55:45.856779 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:45.856760    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6eb83fab-c7af-40e5-8059-88ff3b38cfcb-proxy-tls\") pod \"isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l\" (UID: \"6eb83fab-c7af-40e5-8059-88ff3b38cfcb\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l"
Jul 02 12:55:45.863668 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:45.863645    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc22c\" (UniqueName: \"kubernetes.io/projected/6eb83fab-c7af-40e5-8059-88ff3b38cfcb-kube-api-access-pc22c\") pod \"isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l\" (UID: \"6eb83fab-c7af-40e5-8059-88ff3b38cfcb\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l"
Jul 02 12:55:45.975850 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:45.975764    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l"
Jul 02 12:55:46.117904 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:46.117872    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l"]
Jul 02 12:55:46.122066 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:55:46.122040    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eb83fab_c7af_40e5_8059_88ff3b38cfcb.slice/crio-9beaa9b15181af6a8da8d18064da9e7e22dc60328e5b5107486c197616d8bcb9 WatchSource:0}: Error finding container 9beaa9b15181af6a8da8d18064da9e7e22dc60328e5b5107486c197616d8bcb9: Status 404 returned error can't find the container with id 9beaa9b15181af6a8da8d18064da9e7e22dc60328e5b5107486c197616d8bcb9
Jul 02 12:55:46.855135 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:46.855096    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" event={"ID":"6eb83fab-c7af-40e5-8059-88ff3b38cfcb","Type":"ContainerStarted","Data":"c912dc99119f6523caefcc1e9dfea587e7e5e925842f1357bb0c0d064be0cbce"}
Jul 02 12:55:46.855502 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:46.855162    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" event={"ID":"6eb83fab-c7af-40e5-8059-88ff3b38cfcb","Type":"ContainerStarted","Data":"9beaa9b15181af6a8da8d18064da9e7e22dc60328e5b5107486c197616d8bcb9"}
Jul 02 12:55:50.881356 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:50.881324    2576 generic.go:356] "Generic (PLEG): container finished" podID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerID="c912dc99119f6523caefcc1e9dfea587e7e5e925842f1357bb0c0d064be0cbce" exitCode=0
Jul 02 12:55:50.881703 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:50.881396    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" event={"ID":"6eb83fab-c7af-40e5-8059-88ff3b38cfcb","Type":"ContainerDied","Data":"c912dc99119f6523caefcc1e9dfea587e7e5e925842f1357bb0c0d064be0cbce"}
Jul 02 12:55:51.888879 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:51.888846    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" event={"ID":"6eb83fab-c7af-40e5-8059-88ff3b38cfcb","Type":"ContainerStarted","Data":"9665c5f2e975f9ad0aface865705c4edc26ec0d7e054d842163d7183100c2272"}
Jul 02 12:55:51.889340 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:51.888932    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" event={"ID":"6eb83fab-c7af-40e5-8059-88ff3b38cfcb","Type":"ContainerStarted","Data":"e0ce0245841a77239d76453b6c5a2ba5f585c644f542b0a55c1e3d08af71266f"}
Jul 02 12:55:51.889340 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:51.888957    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" event={"ID":"6eb83fab-c7af-40e5-8059-88ff3b38cfcb","Type":"ContainerStarted","Data":"a3ab1531922569fbde4b2fbaa46d0a7a845a776e01ce6e7d21b6fa065b6760e9"}
Jul 02 12:55:51.889455 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:51.889389    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l"
Jul 02 12:55:51.889455 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:51.889449    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l"
Jul 02 12:55:51.889519 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:51.889486    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l"
Jul 02 12:55:51.890959 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:51.890937    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 02 12:55:51.891706 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:51.891679    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:55:52.894886 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:52.894841    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 02 12:55:52.895307 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:52.895279    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:55:57.900173 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:57.900145    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l"
Jul 02 12:55:57.900718 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:57.900687    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 02 12:55:57.901131 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:57.901106    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:55:57.932247 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:55:57.932199    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" podStartSLOduration=9.182260974 podStartE2EDuration="12.93218424s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.749923266s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:55:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:55:51.923678858 +0000 UTC m=+1040.531141882" watchObservedRunningTime="2026-07-02 12:55:57.93218424 +0000 UTC m=+1046.539647262"
Jul 02 12:56:07.901267 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:56:07.901226    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 02 12:56:07.901695 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:56:07.901670    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:56:17.901171 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:56:17.901117    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 02 12:56:17.901605 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:56:17.901576    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:56:27.900897 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:56:27.900810    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 02 12:56:27.901384 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:56:27.901360    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:56:37.901144 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:56:37.901103    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 02 12:56:37.901659 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:56:37.901622    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:56:47.901377 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:56:47.901332    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 02 12:56:47.901856 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:56:47.901829    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:56:57.901114 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:56:57.901076    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 02 12:56:57.901976 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:56:57.901949    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:57:07.901310 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:07.901279    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l"
Jul 02 12:57:07.901814 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:07.901789    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l"
Jul 02 12:57:20.633834 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:20.633801    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv_15064fb3-c271-4243-9c62-6bb1be858fc2/kserve-container/0.log"
Jul 02 12:57:20.956441 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:20.956410    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l"]
Jul 02 12:57:20.956723 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:20.956697    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="kserve-container" containerID="cri-o://a3ab1531922569fbde4b2fbaa46d0a7a845a776e01ce6e7d21b6fa065b6760e9" gracePeriod=30
Jul 02 12:57:20.956808 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:20.956766    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="agent" containerID="cri-o://9665c5f2e975f9ad0aface865705c4edc26ec0d7e054d842163d7183100c2272" gracePeriod=30
Jul 02 12:57:20.956996 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:20.956935    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="kube-rbac-proxy" containerID="cri-o://e0ce0245841a77239d76453b6c5a2ba5f585c644f542b0a55c1e3d08af71266f" gracePeriod=30
Jul 02 12:57:20.967463 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:20.967438    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc"]
Jul 02 12:57:20.971583 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:20.971564    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc"
Jul 02 12:57:20.976249 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:20.976018    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-56395-kube-rbac-proxy-sar-config\""
Jul 02 12:57:20.976249 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:20.976063    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-56395-predictor-serving-cert\""
Jul 02 12:57:20.985393 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:20.985365    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4774483e-4db5-4cd4-ac2c-cd33a7c33331-cabundle-cert\") pod \"isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc\" (UID: \"4774483e-4db5-4cd4-ac2c-cd33a7c33331\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc"
Jul 02 12:57:20.985549 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:20.985417    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4774483e-4db5-4cd4-ac2c-cd33a7c33331-proxy-tls\") pod \"isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc\" (UID: \"4774483e-4db5-4cd4-ac2c-cd33a7c33331\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc"
Jul 02 12:57:20.985549 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:20.985468    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wjqb\" (UniqueName: \"kubernetes.io/projected/4774483e-4db5-4cd4-ac2c-cd33a7c33331-kube-api-access-8wjqb\") pod \"isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc\" (UID: \"4774483e-4db5-4cd4-ac2c-cd33a7c33331\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc"
Jul 02 12:57:20.985549 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:20.985525    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4774483e-4db5-4cd4-ac2c-cd33a7c33331-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc\" (UID: \"4774483e-4db5-4cd4-ac2c-cd33a7c33331\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc"
Jul 02 12:57:20.985817 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:20.985615    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4774483e-4db5-4cd4-ac2c-cd33a7c33331-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc\" (UID: \"4774483e-4db5-4cd4-ac2c-cd33a7c33331\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc"
Jul 02 12:57:20.991329 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:20.991305    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc"]
Jul 02 12:57:21.000802 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.000782    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv"]
Jul 02 12:57:21.001031 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.000995    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv" podUID="15064fb3-c271-4243-9c62-6bb1be858fc2" containerName="kserve-container" containerID="cri-o://48aa19715c13ef82c8b458b7a57a0a140f2accc4d6e8bf9c461a910fe1dc0648" gracePeriod=30
Jul 02 12:57:21.001111 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.001082    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv" podUID="15064fb3-c271-4243-9c62-6bb1be858fc2" containerName="kube-rbac-proxy" containerID="cri-o://87d148ddcef96fab893881c1d18b862187b169d884fa65ffaadf87a893bbd3cd" gracePeriod=30
Jul 02 12:57:21.087105 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.087074    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4774483e-4db5-4cd4-ac2c-cd33a7c33331-cabundle-cert\") pod \"isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc\" (UID: \"4774483e-4db5-4cd4-ac2c-cd33a7c33331\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc"
Jul 02 12:57:21.087308 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.087112    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4774483e-4db5-4cd4-ac2c-cd33a7c33331-proxy-tls\") pod \"isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc\" (UID: \"4774483e-4db5-4cd4-ac2c-cd33a7c33331\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc"
Jul 02 12:57:21.087308 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.087145    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wjqb\" (UniqueName: \"kubernetes.io/projected/4774483e-4db5-4cd4-ac2c-cd33a7c33331-kube-api-access-8wjqb\") pod \"isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc\" (UID: \"4774483e-4db5-4cd4-ac2c-cd33a7c33331\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc"
Jul 02 12:57:21.087308 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.087185    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4774483e-4db5-4cd4-ac2c-cd33a7c33331-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc\" (UID: \"4774483e-4db5-4cd4-ac2c-cd33a7c33331\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc"
Jul 02 12:57:21.087308 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.087222    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4774483e-4db5-4cd4-ac2c-cd33a7c33331-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc\" (UID: \"4774483e-4db5-4cd4-ac2c-cd33a7c33331\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc"
Jul 02 12:57:21.087308 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:57:21.087247    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-serving-cert: secret "isvc-sklearn-scale-raw-56395-predictor-serving-cert" not found
Jul 02 12:57:21.087591 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:57:21.087317    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4774483e-4db5-4cd4-ac2c-cd33a7c33331-proxy-tls podName:4774483e-4db5-4cd4-ac2c-cd33a7c33331 nodeName:}" failed. No retries permitted until 2026-07-02 12:57:21.587294163 +0000 UTC m=+1130.194757165 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4774483e-4db5-4cd4-ac2c-cd33a7c33331-proxy-tls") pod "isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc" (UID: "4774483e-4db5-4cd4-ac2c-cd33a7c33331") : secret "isvc-sklearn-scale-raw-56395-predictor-serving-cert" not found
Jul 02 12:57:21.087843 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.087812    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4774483e-4db5-4cd4-ac2c-cd33a7c33331-cabundle-cert\") pod \"isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc\" (UID: \"4774483e-4db5-4cd4-ac2c-cd33a7c33331\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc"
Jul 02 12:57:21.087946 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.087843    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4774483e-4db5-4cd4-ac2c-cd33a7c33331-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc\" (UID: \"4774483e-4db5-4cd4-ac2c-cd33a7c33331\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc"
Jul 02 12:57:21.088000 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.087988    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4774483e-4db5-4cd4-ac2c-cd33a7c33331-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc\" (UID: \"4774483e-4db5-4cd4-ac2c-cd33a7c33331\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc"
Jul 02 12:57:21.096097 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.096076    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wjqb\" (UniqueName: \"kubernetes.io/projected/4774483e-4db5-4cd4-ac2c-cd33a7c33331-kube-api-access-8wjqb\") pod \"isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc\" (UID: \"4774483e-4db5-4cd4-ac2c-cd33a7c33331\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc"
Jul 02 12:57:21.251195 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.251174    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv"
Jul 02 12:57:21.289586 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.289562    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15064fb3-c271-4243-9c62-6bb1be858fc2-kube-rbac-proxy-sar-config\") pod \"15064fb3-c271-4243-9c62-6bb1be858fc2\" (UID: \"15064fb3-c271-4243-9c62-6bb1be858fc2\") "
Jul 02 12:57:21.289749 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.289729    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxndt\" (UniqueName: \"kubernetes.io/projected/15064fb3-c271-4243-9c62-6bb1be858fc2-kube-api-access-xxndt\") pod \"15064fb3-c271-4243-9c62-6bb1be858fc2\" (UID: \"15064fb3-c271-4243-9c62-6bb1be858fc2\") "
Jul 02 12:57:21.289818 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.289771    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15064fb3-c271-4243-9c62-6bb1be858fc2-proxy-tls\") pod \"15064fb3-c271-4243-9c62-6bb1be858fc2\" (UID: \"15064fb3-c271-4243-9c62-6bb1be858fc2\") "
Jul 02 12:57:21.289902 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.289874    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15064fb3-c271-4243-9c62-6bb1be858fc2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "15064fb3-c271-4243-9c62-6bb1be858fc2" (UID: "15064fb3-c271-4243-9c62-6bb1be858fc2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:57:21.290121 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.290103    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15064fb3-c271-4243-9c62-6bb1be858fc2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:57:21.291768 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.291737    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15064fb3-c271-4243-9c62-6bb1be858fc2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "15064fb3-c271-4243-9c62-6bb1be858fc2" (UID: "15064fb3-c271-4243-9c62-6bb1be858fc2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 12:57:21.291928 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.291777    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15064fb3-c271-4243-9c62-6bb1be858fc2-kube-api-access-xxndt" (OuterVolumeSpecName: "kube-api-access-xxndt") pod "15064fb3-c271-4243-9c62-6bb1be858fc2" (UID: "15064fb3-c271-4243-9c62-6bb1be858fc2"). InnerVolumeSpecName "kube-api-access-xxndt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 12:57:21.338453 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.338422    2576 generic.go:356] "Generic (PLEG): container finished" podID="15064fb3-c271-4243-9c62-6bb1be858fc2" containerID="87d148ddcef96fab893881c1d18b862187b169d884fa65ffaadf87a893bbd3cd" exitCode=2
Jul 02 12:57:21.338453 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.338445    2576 generic.go:356] "Generic (PLEG): container finished" podID="15064fb3-c271-4243-9c62-6bb1be858fc2" containerID="48aa19715c13ef82c8b458b7a57a0a140f2accc4d6e8bf9c461a910fe1dc0648" exitCode=2
Jul 02 12:57:21.338678 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.338475    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv"
Jul 02 12:57:21.338678 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.338479    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv" event={"ID":"15064fb3-c271-4243-9c62-6bb1be858fc2","Type":"ContainerDied","Data":"87d148ddcef96fab893881c1d18b862187b169d884fa65ffaadf87a893bbd3cd"}
Jul 02 12:57:21.338678 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.338545    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv" event={"ID":"15064fb3-c271-4243-9c62-6bb1be858fc2","Type":"ContainerDied","Data":"48aa19715c13ef82c8b458b7a57a0a140f2accc4d6e8bf9c461a910fe1dc0648"}
Jul 02 12:57:21.338678 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.338580    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv" event={"ID":"15064fb3-c271-4243-9c62-6bb1be858fc2","Type":"ContainerDied","Data":"4b37e00eda5e400da8deb420579477103881b8c087d20f2e0c6cbcd8ae9eb15f"}
Jul 02 12:57:21.338678 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.338616    2576 scope.go:117] "RemoveContainer" containerID="87d148ddcef96fab893881c1d18b862187b169d884fa65ffaadf87a893bbd3cd"
Jul 02 12:57:21.341328 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.341303    2576 generic.go:356] "Generic (PLEG): container finished" podID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerID="e0ce0245841a77239d76453b6c5a2ba5f585c644f542b0a55c1e3d08af71266f" exitCode=2
Jul 02 12:57:21.341436 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.341349    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" event={"ID":"6eb83fab-c7af-40e5-8059-88ff3b38cfcb","Type":"ContainerDied","Data":"e0ce0245841a77239d76453b6c5a2ba5f585c644f542b0a55c1e3d08af71266f"}
Jul 02 12:57:21.347777 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.347761    2576 scope.go:117] "RemoveContainer" containerID="48aa19715c13ef82c8b458b7a57a0a140f2accc4d6e8bf9c461a910fe1dc0648"
Jul 02 12:57:21.357005 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.356985    2576 scope.go:117] "RemoveContainer" containerID="87d148ddcef96fab893881c1d18b862187b169d884fa65ffaadf87a893bbd3cd"
Jul 02 12:57:21.357450 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:57:21.357430    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87d148ddcef96fab893881c1d18b862187b169d884fa65ffaadf87a893bbd3cd\": container with ID starting with 87d148ddcef96fab893881c1d18b862187b169d884fa65ffaadf87a893bbd3cd not found: ID does not exist" containerID="87d148ddcef96fab893881c1d18b862187b169d884fa65ffaadf87a893bbd3cd"
Jul 02 12:57:21.357520 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.357459    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87d148ddcef96fab893881c1d18b862187b169d884fa65ffaadf87a893bbd3cd"} err="failed to get container status \"87d148ddcef96fab893881c1d18b862187b169d884fa65ffaadf87a893bbd3cd\": rpc error: code = NotFound desc = could not find container \"87d148ddcef96fab893881c1d18b862187b169d884fa65ffaadf87a893bbd3cd\": container with ID starting with 87d148ddcef96fab893881c1d18b862187b169d884fa65ffaadf87a893bbd3cd not found: ID does not exist"
Jul 02 12:57:21.357520 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.357484    2576 scope.go:117] "RemoveContainer" containerID="48aa19715c13ef82c8b458b7a57a0a140f2accc4d6e8bf9c461a910fe1dc0648"
Jul 02 12:57:21.358041 ip-10-0-143-88 kubenswrapper[2576]: E0702 12:57:21.358016    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48aa19715c13ef82c8b458b7a57a0a140f2accc4d6e8bf9c461a910fe1dc0648\": container with ID starting with 48aa19715c13ef82c8b458b7a57a0a140f2accc4d6e8bf9c461a910fe1dc0648 not found: ID does not exist" containerID="48aa19715c13ef82c8b458b7a57a0a140f2accc4d6e8bf9c461a910fe1dc0648"
Jul 02 12:57:21.358155 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.358048    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48aa19715c13ef82c8b458b7a57a0a140f2accc4d6e8bf9c461a910fe1dc0648"} err="failed to get container status \"48aa19715c13ef82c8b458b7a57a0a140f2accc4d6e8bf9c461a910fe1dc0648\": rpc error: code = NotFound desc = could not find container \"48aa19715c13ef82c8b458b7a57a0a140f2accc4d6e8bf9c461a910fe1dc0648\": container with ID starting with 48aa19715c13ef82c8b458b7a57a0a140f2accc4d6e8bf9c461a910fe1dc0648 not found: ID does not exist"
Jul 02 12:57:21.358155 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.358065    2576 scope.go:117] "RemoveContainer" containerID="87d148ddcef96fab893881c1d18b862187b169d884fa65ffaadf87a893bbd3cd"
Jul 02 12:57:21.358363 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.358333    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87d148ddcef96fab893881c1d18b862187b169d884fa65ffaadf87a893bbd3cd"} err="failed to get container status \"87d148ddcef96fab893881c1d18b862187b169d884fa65ffaadf87a893bbd3cd\": rpc error: code = NotFound desc = could not find container \"87d148ddcef96fab893881c1d18b862187b169d884fa65ffaadf87a893bbd3cd\": container with ID starting with 87d148ddcef96fab893881c1d18b862187b169d884fa65ffaadf87a893bbd3cd not found: ID does not exist"
Jul 02 12:57:21.358363 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.358363    2576 scope.go:117] "RemoveContainer" containerID="48aa19715c13ef82c8b458b7a57a0a140f2accc4d6e8bf9c461a910fe1dc0648"
Jul 02 12:57:21.358670 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.358651    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48aa19715c13ef82c8b458b7a57a0a140f2accc4d6e8bf9c461a910fe1dc0648"} err="failed to get container status \"48aa19715c13ef82c8b458b7a57a0a140f2accc4d6e8bf9c461a910fe1dc0648\": rpc error: code = NotFound desc = could not find container \"48aa19715c13ef82c8b458b7a57a0a140f2accc4d6e8bf9c461a910fe1dc0648\": container with ID starting with 48aa19715c13ef82c8b458b7a57a0a140f2accc4d6e8bf9c461a910fe1dc0648 not found: ID does not exist"
Jul 02 12:57:21.360728 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.360709    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv"]
Jul 02 12:57:21.366480 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.366460    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-c99c3-predictor-7f55f4f59c-vm9qv"]
Jul 02 12:57:21.390981 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.390962    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxndt\" (UniqueName: \"kubernetes.io/projected/15064fb3-c271-4243-9c62-6bb1be858fc2-kube-api-access-xxndt\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:57:21.391091 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.390983    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15064fb3-c271-4243-9c62-6bb1be858fc2-proxy-tls\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:57:21.592765 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.592680    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4774483e-4db5-4cd4-ac2c-cd33a7c33331-proxy-tls\") pod \"isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc\" (UID: \"4774483e-4db5-4cd4-ac2c-cd33a7c33331\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc"
Jul 02 12:57:21.595066 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.595042    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4774483e-4db5-4cd4-ac2c-cd33a7c33331-proxy-tls\") pod \"isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc\" (UID: \"4774483e-4db5-4cd4-ac2c-cd33a7c33331\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc"
Jul 02 12:57:21.884579 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:21.884493    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc"
Jul 02 12:57:22.029335 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:22.029296    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc"]
Jul 02 12:57:22.031723 ip-10-0-143-88 kubenswrapper[2576]: W0702 12:57:22.031697    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4774483e_4db5_4cd4_ac2c_cd33a7c33331.slice/crio-1beb7198ca9301b5ea144668da53150ff85133204af23ef02795833a0c7e7b4b WatchSource:0}: Error finding container 1beb7198ca9301b5ea144668da53150ff85133204af23ef02795833a0c7e7b4b: Status 404 returned error can't find the container with id 1beb7198ca9301b5ea144668da53150ff85133204af23ef02795833a0c7e7b4b
Jul 02 12:57:22.349531 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:22.349497    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc" event={"ID":"4774483e-4db5-4cd4-ac2c-cd33a7c33331","Type":"ContainerStarted","Data":"2c9ecb24fd2bc7632c2722d9f01eddf1de1a3f58b9aedfbf19ac6e191854f865"}
Jul 02 12:57:22.349747 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:22.349568    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc" event={"ID":"4774483e-4db5-4cd4-ac2c-cd33a7c33331","Type":"ContainerStarted","Data":"1beb7198ca9301b5ea144668da53150ff85133204af23ef02795833a0c7e7b4b"}
Jul 02 12:57:22.864026 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:22.863998    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15064fb3-c271-4243-9c62-6bb1be858fc2" path="/var/lib/kubelet/pods/15064fb3-c271-4243-9c62-6bb1be858fc2/volumes"
Jul 02 12:57:22.895076 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:22.895039    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Jul 02 12:57:26.373016 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:26.372931    2576 generic.go:356] "Generic (PLEG): container finished" podID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerID="a3ab1531922569fbde4b2fbaa46d0a7a845a776e01ce6e7d21b6fa065b6760e9" exitCode=0
Jul 02 12:57:26.373016 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:26.372986    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" event={"ID":"6eb83fab-c7af-40e5-8059-88ff3b38cfcb","Type":"ContainerDied","Data":"a3ab1531922569fbde4b2fbaa46d0a7a845a776e01ce6e7d21b6fa065b6760e9"}
Jul 02 12:57:26.374466 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:26.374444    2576 generic.go:356] "Generic (PLEG): container finished" podID="4774483e-4db5-4cd4-ac2c-cd33a7c33331" containerID="2c9ecb24fd2bc7632c2722d9f01eddf1de1a3f58b9aedfbf19ac6e191854f865" exitCode=0
Jul 02 12:57:26.374573 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:26.374492    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc" event={"ID":"4774483e-4db5-4cd4-ac2c-cd33a7c33331","Type":"ContainerDied","Data":"2c9ecb24fd2bc7632c2722d9f01eddf1de1a3f58b9aedfbf19ac6e191854f865"}
Jul 02 12:57:27.382034 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:27.381999    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc" event={"ID":"4774483e-4db5-4cd4-ac2c-cd33a7c33331","Type":"ContainerStarted","Data":"b3118bf4bd952b28c510227dea6029848b6a19938b4235f6c1980c6f21ff04ba"}
Jul 02 12:57:27.382451 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:27.382060    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc" event={"ID":"4774483e-4db5-4cd4-ac2c-cd33a7c33331","Type":"ContainerStarted","Data":"962561d63ced00ab0a83f4205008259241cc3ba0823c6be244dbdd4c9a248770"}
Jul 02 12:57:27.382451 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:27.382316    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc"
Jul 02 12:57:27.382551 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:27.382460    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc"
Jul 02 12:57:27.383687 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:27.383661    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc" podUID="4774483e-4db5-4cd4-ac2c-cd33a7c33331" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 02 12:57:27.401602 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:27.401559    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc" podStartSLOduration=3.529310166 podStartE2EDuration="7.401546326s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.87223616s" isStatefulPod=false podCreationTimestamp="2026-07-02 12:57:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 12:57:27.400119107 +0000 UTC m=+1136.007582130" watchObservedRunningTime="2026-07-02 12:57:27.401546326 +0000 UTC m=+1136.009009348"
Jul 02 12:57:27.895478 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:27.895438    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Jul 02 12:57:27.901181 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:27.901146    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:57:27.902371 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:27.902351    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 02 12:57:28.386633 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:28.386545    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc" podUID="4774483e-4db5-4cd4-ac2c-cd33a7c33331" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 02 12:57:32.895044 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:32.895006    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Jul 02 12:57:32.895431 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:32.895135    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l"
Jul 02 12:57:33.392170 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:33.392150    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc"
Jul 02 12:57:33.392625 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:33.392599    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc" podUID="4774483e-4db5-4cd4-ac2c-cd33a7c33331" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 02 12:57:37.895707 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:37.895659    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Jul 02 12:57:37.901541 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:37.901509    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:57:37.902417 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:37.902393    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 02 12:57:42.895313 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:42.895276    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Jul 02 12:57:43.393473 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:43.393438    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc" podUID="4774483e-4db5-4cd4-ac2c-cd33a7c33331" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 02 12:57:47.895738 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:47.895696    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Jul 02 12:57:47.901534 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:47.901497    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 12:57:47.901655 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:47.901634    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l"
Jul 02 12:57:47.901925 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:47.901892    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 02 12:57:47.902006 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:47.901995    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l"
Jul 02 12:57:51.502864 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:51.502817    2576 generic.go:356] "Generic (PLEG): container finished" podID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerID="9665c5f2e975f9ad0aface865705c4edc26ec0d7e054d842163d7183100c2272" exitCode=0
Jul 02 12:57:51.503316 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:51.502939    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" event={"ID":"6eb83fab-c7af-40e5-8059-88ff3b38cfcb","Type":"ContainerDied","Data":"9665c5f2e975f9ad0aface865705c4edc26ec0d7e054d842163d7183100c2272"}
Jul 02 12:57:51.608698 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:51.608677    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l"
Jul 02 12:57:51.757873 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:51.757780    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6eb83fab-c7af-40e5-8059-88ff3b38cfcb-kserve-provision-location\") pod \"6eb83fab-c7af-40e5-8059-88ff3b38cfcb\" (UID: \"6eb83fab-c7af-40e5-8059-88ff3b38cfcb\") "
Jul 02 12:57:51.757873 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:51.757843    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6eb83fab-c7af-40e5-8059-88ff3b38cfcb-cabundle-cert\") pod \"6eb83fab-c7af-40e5-8059-88ff3b38cfcb\" (UID: \"6eb83fab-c7af-40e5-8059-88ff3b38cfcb\") "
Jul 02 12:57:51.757873 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:51.757874    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pc22c\" (UniqueName: \"kubernetes.io/projected/6eb83fab-c7af-40e5-8059-88ff3b38cfcb-kube-api-access-pc22c\") pod \"6eb83fab-c7af-40e5-8059-88ff3b38cfcb\" (UID: \"6eb83fab-c7af-40e5-8059-88ff3b38cfcb\") "
Jul 02 12:57:51.758113 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:51.757966    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6eb83fab-c7af-40e5-8059-88ff3b38cfcb-kube-rbac-proxy-sar-config\") pod \"6eb83fab-c7af-40e5-8059-88ff3b38cfcb\" (UID: \"6eb83fab-c7af-40e5-8059-88ff3b38cfcb\") "
Jul 02 12:57:51.758113 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:51.758056    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6eb83fab-c7af-40e5-8059-88ff3b38cfcb-proxy-tls\") pod \"6eb83fab-c7af-40e5-8059-88ff3b38cfcb\" (UID: \"6eb83fab-c7af-40e5-8059-88ff3b38cfcb\") "
Jul 02 12:57:51.758196 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:51.758134    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eb83fab-c7af-40e5-8059-88ff3b38cfcb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6eb83fab-c7af-40e5-8059-88ff3b38cfcb" (UID: "6eb83fab-c7af-40e5-8059-88ff3b38cfcb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 12:57:51.758314 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:51.758290    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eb83fab-c7af-40e5-8059-88ff3b38cfcb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6eb83fab-c7af-40e5-8059-88ff3b38cfcb" (UID: "6eb83fab-c7af-40e5-8059-88ff3b38cfcb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:57:51.758379 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:51.758347    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eb83fab-c7af-40e5-8059-88ff3b38cfcb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6eb83fab-c7af-40e5-8059-88ff3b38cfcb" (UID: "6eb83fab-c7af-40e5-8059-88ff3b38cfcb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 12:57:51.758379 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:51.758362    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6eb83fab-c7af-40e5-8059-88ff3b38cfcb-kserve-provision-location\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:57:51.758471 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:51.758378    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6eb83fab-c7af-40e5-8059-88ff3b38cfcb-cabundle-cert\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:57:51.760084 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:51.760060    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6eb83fab-c7af-40e5-8059-88ff3b38cfcb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6eb83fab-c7af-40e5-8059-88ff3b38cfcb" (UID: "6eb83fab-c7af-40e5-8059-88ff3b38cfcb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 12:57:51.760198 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:51.760182    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eb83fab-c7af-40e5-8059-88ff3b38cfcb-kube-api-access-pc22c" (OuterVolumeSpecName: "kube-api-access-pc22c") pod "6eb83fab-c7af-40e5-8059-88ff3b38cfcb" (UID: "6eb83fab-c7af-40e5-8059-88ff3b38cfcb"). InnerVolumeSpecName "kube-api-access-pc22c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 12:57:51.858831 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:51.858797    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6eb83fab-c7af-40e5-8059-88ff3b38cfcb-proxy-tls\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:57:51.858831 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:51.858826    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc22c\" (UniqueName: \"kubernetes.io/projected/6eb83fab-c7af-40e5-8059-88ff3b38cfcb-kube-api-access-pc22c\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:57:51.858831 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:51.858836    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6eb83fab-c7af-40e5-8059-88ff3b38cfcb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 12:57:52.509451 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:52.509421    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l" event={"ID":"6eb83fab-c7af-40e5-8059-88ff3b38cfcb","Type":"ContainerDied","Data":"9beaa9b15181af6a8da8d18064da9e7e22dc60328e5b5107486c197616d8bcb9"}
Jul 02 12:57:52.509936 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:52.509461    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l"
Jul 02 12:57:52.509936 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:52.509501    2576 scope.go:117] "RemoveContainer" containerID="9665c5f2e975f9ad0aface865705c4edc26ec0d7e054d842163d7183100c2272"
Jul 02 12:57:52.518757 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:52.518741    2576 scope.go:117] "RemoveContainer" containerID="e0ce0245841a77239d76453b6c5a2ba5f585c644f542b0a55c1e3d08af71266f"
Jul 02 12:57:52.526422 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:52.526402    2576 scope.go:117] "RemoveContainer" containerID="a3ab1531922569fbde4b2fbaa46d0a7a845a776e01ce6e7d21b6fa065b6760e9"
Jul 02 12:57:52.532504 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:52.532479    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l"]
Jul 02 12:57:52.534492 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:52.534476    2576 scope.go:117] "RemoveContainer" containerID="c912dc99119f6523caefcc1e9dfea587e7e5e925842f1357bb0c0d064be0cbce"
Jul 02 12:57:52.538079 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:52.538058    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-c99c3-predictor-77cbd7b758-qd74l"]
Jul 02 12:57:52.869655 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:52.869571    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" path="/var/lib/kubelet/pods/6eb83fab-c7af-40e5-8059-88ff3b38cfcb/volumes"
Jul 02 12:57:53.393425 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:57:53.393383    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc" podUID="4774483e-4db5-4cd4-ac2c-cd33a7c33331" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 02 12:58:03.392769 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:58:03.392726    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc" podUID="4774483e-4db5-4cd4-ac2c-cd33a7c33331" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 02 12:58:13.392803 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:58:13.392762    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc" podUID="4774483e-4db5-4cd4-ac2c-cd33a7c33331" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 02 12:58:23.393153 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:58:23.393116    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc" podUID="4774483e-4db5-4cd4-ac2c-cd33a7c33331" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 02 12:58:32.275967 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:58:32.275941    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2c7ks_116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a/ovn-acl-logging/0.log"
Jul 02 12:58:32.900367 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:58:32.900343    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2c7ks_116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a/ovn-acl-logging/0.log"
Jul 02 12:58:33.393103 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:58:33.393066    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc" podUID="4774483e-4db5-4cd4-ac2c-cd33a7c33331" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 02 12:58:34.857332 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:58:34.857290    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc" podUID="4774483e-4db5-4cd4-ac2c-cd33a7c33331" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 02 12:58:44.857490 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:58:44.857443    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc" podUID="4774483e-4db5-4cd4-ac2c-cd33a7c33331" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 02 12:58:54.857423 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:58:54.857381    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc" podUID="4774483e-4db5-4cd4-ac2c-cd33a7c33331" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 02 12:59:04.858135 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:59:04.858095    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc" podUID="4774483e-4db5-4cd4-ac2c-cd33a7c33331" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 02 12:59:14.857282 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:59:14.857240    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc" podUID="4774483e-4db5-4cd4-ac2c-cd33a7c33331" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 02 12:59:24.857480 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:59:24.857381    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc" podUID="4774483e-4db5-4cd4-ac2c-cd33a7c33331" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 02 12:59:34.857677 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:59:34.857630    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc" podUID="4774483e-4db5-4cd4-ac2c-cd33a7c33331" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 02 12:59:39.858274 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:59:39.858233    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc" podUID="4774483e-4db5-4cd4-ac2c-cd33a7c33331" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 02 12:59:49.858931 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:59:49.858873    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc" podUID="4774483e-4db5-4cd4-ac2c-cd33a7c33331" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 02 12:59:59.859111 ip-10-0-143-88 kubenswrapper[2576]: I0702 12:59:59.859077    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc"
Jul 02 13:00:01.136090 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:01.136054    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc"]
Jul 02 13:00:01.136500 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:01.136321    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc" podUID="4774483e-4db5-4cd4-ac2c-cd33a7c33331" containerName="kserve-container" containerID="cri-o://962561d63ced00ab0a83f4205008259241cc3ba0823c6be244dbdd4c9a248770" gracePeriod=30
Jul 02 13:00:01.136500 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:01.136366    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc" podUID="4774483e-4db5-4cd4-ac2c-cd33a7c33331" containerName="kube-rbac-proxy" containerID="cri-o://b3118bf4bd952b28c510227dea6029848b6a19938b4235f6c1980c6f21ff04ba" gracePeriod=30
Jul 02 13:00:02.136181 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:02.136148    2576 generic.go:356] "Generic (PLEG): container finished" podID="4774483e-4db5-4cd4-ac2c-cd33a7c33331" containerID="b3118bf4bd952b28c510227dea6029848b6a19938b4235f6c1980c6f21ff04ba" exitCode=2
Jul 02 13:00:02.136549 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:02.136223    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc" event={"ID":"4774483e-4db5-4cd4-ac2c-cd33a7c33331","Type":"ContainerDied","Data":"b3118bf4bd952b28c510227dea6029848b6a19938b4235f6c1980c6f21ff04ba"}
Jul 02 13:00:03.387584 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:03.387536    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc" podUID="4774483e-4db5-4cd4-ac2c-cd33a7c33331" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Jul 02 13:00:06.244038 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:06.244002    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt"]
Jul 02 13:00:06.244417 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:06.244403    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15064fb3-c271-4243-9c62-6bb1be858fc2" containerName="kserve-container"
Jul 02 13:00:06.244460 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:06.244418    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="15064fb3-c271-4243-9c62-6bb1be858fc2" containerName="kserve-container"
Jul 02 13:00:06.244460 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:06.244434    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="agent"
Jul 02 13:00:06.244460 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:06.244439    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="agent"
Jul 02 13:00:06.244460 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:06.244448    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="kserve-container"
Jul 02 13:00:06.244460 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:06.244453    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="kserve-container"
Jul 02 13:00:06.244460 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:06.244460    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15064fb3-c271-4243-9c62-6bb1be858fc2" containerName="kube-rbac-proxy"
Jul 02 13:00:06.244643 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:06.244465    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="15064fb3-c271-4243-9c62-6bb1be858fc2" containerName="kube-rbac-proxy"
Jul 02 13:00:06.244643 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:06.244471    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="storage-initializer"
Jul 02 13:00:06.244643 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:06.244476    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="storage-initializer"
Jul 02 13:00:06.244643 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:06.244488    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="kube-rbac-proxy"
Jul 02 13:00:06.244643 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:06.244493    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="kube-rbac-proxy"
Jul 02 13:00:06.244643 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:06.244558    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="kube-rbac-proxy"
Jul 02 13:00:06.244643 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:06.244566    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="kserve-container"
Jul 02 13:00:06.244643 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:06.244572    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="15064fb3-c271-4243-9c62-6bb1be858fc2" containerName="kserve-container"
Jul 02 13:00:06.244643 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:06.244580    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6eb83fab-c7af-40e5-8059-88ff3b38cfcb" containerName="agent"
Jul 02 13:00:06.244643 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:06.244587    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="15064fb3-c271-4243-9c62-6bb1be858fc2" containerName="kube-rbac-proxy"
Jul 02 13:00:06.247955 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:06.247933    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt"
Jul 02 13:00:06.250596 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:06.250577    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-c9e60f-predictor-serving-cert\""
Jul 02 13:00:06.250749 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:06.250730    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-c9e60f-kube-rbac-proxy-sar-config\""
Jul 02 13:00:06.259475 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:06.259450    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt"]
Jul 02 13:00:06.298190 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:06.298148    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnpth\" (UniqueName: \"kubernetes.io/projected/80847d7c-5d90-449c-816c-70cdb1548320-kube-api-access-tnpth\") pod \"isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt\" (UID: \"80847d7c-5d90-449c-816c-70cdb1548320\") " pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt"
Jul 02 13:00:06.298391 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:06.298202    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80847d7c-5d90-449c-816c-70cdb1548320-kserve-provision-location\") pod \"isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt\" (UID: \"80847d7c-5d90-449c-816c-70cdb1548320\") " pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt"
Jul 02 13:00:06.298391 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:06.298294    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80847d7c-5d90-449c-816c-70cdb1548320-kube-rbac-proxy-sar-config\") pod \"isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt\" (UID: \"80847d7c-5d90-449c-816c-70cdb1548320\") " pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt"
Jul 02 13:00:06.298478 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:06.298422    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80847d7c-5d90-449c-816c-70cdb1548320-proxy-tls\") pod \"isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt\" (UID: \"80847d7c-5d90-449c-816c-70cdb1548320\") " pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt"
Jul 02 13:00:06.298515 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:06.298477    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80847d7c-5d90-449c-816c-70cdb1548320-cabundle-cert\") pod \"isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt\" (UID: \"80847d7c-5d90-449c-816c-70cdb1548320\") " pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt"
Jul 02 13:00:06.399198 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:06.399160    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80847d7c-5d90-449c-816c-70cdb1548320-cabundle-cert\") pod \"isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt\" (UID: \"80847d7c-5d90-449c-816c-70cdb1548320\") " pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt"
Jul 02 13:00:06.399393 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:06.399206    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnpth\" (UniqueName: \"kubernetes.io/projected/80847d7c-5d90-449c-816c-70cdb1548320-kube-api-access-tnpth\") pod \"isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt\" (UID: \"80847d7c-5d90-449c-816c-70cdb1548320\") " pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt"
Jul 02 13:00:06.399393 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:06.399272    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80847d7c-5d90-449c-816c-70cdb1548320-kserve-provision-location\") pod \"isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt\" (UID: \"80847d7c-5d90-449c-816c-70cdb1548320\") " pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt"
Jul 02 13:00:06.399393 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:06.399321    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80847d7c-5d90-449c-816c-70cdb1548320-kube-rbac-proxy-sar-config\") pod \"isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt\" (UID: \"80847d7c-5d90-449c-816c-70cdb1548320\") " pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt"
Jul 02 13:00:06.399563 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:06.399426    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80847d7c-5d90-449c-816c-70cdb1548320-proxy-tls\") pod \"isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt\" (UID: \"80847d7c-5d90-449c-816c-70cdb1548320\") " pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt"
Jul 02 13:00:06.399563 ip-10-0-143-88 kubenswrapper[2576]: E0702 13:00:06.399550    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-serving-cert: secret "isvc-primary-c9e60f-predictor-serving-cert" not found
Jul 02 13:00:06.399666 ip-10-0-143-88 kubenswrapper[2576]: E0702 13:00:06.399629    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80847d7c-5d90-449c-816c-70cdb1548320-proxy-tls podName:80847d7c-5d90-449c-816c-70cdb1548320 nodeName:}" failed. No retries permitted until 2026-07-02 13:00:06.899607453 +0000 UTC m=+1295.507070460 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/80847d7c-5d90-449c-816c-70cdb1548320-proxy-tls") pod "isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt" (UID: "80847d7c-5d90-449c-816c-70cdb1548320") : secret "isvc-primary-c9e60f-predictor-serving-cert" not found
Jul 02 13:00:06.399729 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:06.399675    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80847d7c-5d90-449c-816c-70cdb1548320-kserve-provision-location\") pod \"isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt\" (UID: \"80847d7c-5d90-449c-816c-70cdb1548320\") " pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt"
Jul 02 13:00:06.400053 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:06.400028    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80847d7c-5d90-449c-816c-70cdb1548320-cabundle-cert\") pod \"isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt\" (UID: \"80847d7c-5d90-449c-816c-70cdb1548320\") " pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt"
Jul 02 13:00:06.400102 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:06.400093    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80847d7c-5d90-449c-816c-70cdb1548320-kube-rbac-proxy-sar-config\") pod \"isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt\" (UID: \"80847d7c-5d90-449c-816c-70cdb1548320\") " pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt"
Jul 02 13:00:06.408225 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:06.408200    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnpth\" (UniqueName: \"kubernetes.io/projected/80847d7c-5d90-449c-816c-70cdb1548320-kube-api-access-tnpth\") pod \"isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt\" (UID: \"80847d7c-5d90-449c-816c-70cdb1548320\") " pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt"
Jul 02 13:00:06.902873 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:06.902841    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80847d7c-5d90-449c-816c-70cdb1548320-proxy-tls\") pod \"isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt\" (UID: \"80847d7c-5d90-449c-816c-70cdb1548320\") " pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt"
Jul 02 13:00:06.905332 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:06.905312    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80847d7c-5d90-449c-816c-70cdb1548320-proxy-tls\") pod \"isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt\" (UID: \"80847d7c-5d90-449c-816c-70cdb1548320\") " pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt"
Jul 02 13:00:07.158175 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:07.158086    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt"
Jul 02 13:00:07.301051 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:07.301021    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt"]
Jul 02 13:00:07.305364 ip-10-0-143-88 kubenswrapper[2576]: W0702 13:00:07.305334    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80847d7c_5d90_449c_816c_70cdb1548320.slice/crio-1edac906768e905d76f51fe707ec45dc6f5895468c89f92a0f8d021d7a463b8e WatchSource:0}: Error finding container 1edac906768e905d76f51fe707ec45dc6f5895468c89f92a0f8d021d7a463b8e: Status 404 returned error can't find the container with id 1edac906768e905d76f51fe707ec45dc6f5895468c89f92a0f8d021d7a463b8e
Jul 02 13:00:08.166344 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:08.166309    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt" event={"ID":"80847d7c-5d90-449c-816c-70cdb1548320","Type":"ContainerStarted","Data":"55fb31d57c5316cd069d836812c993ca68c185d80c435271b268361095159575"}
Jul 02 13:00:08.166535 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:08.166364    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt" event={"ID":"80847d7c-5d90-449c-816c-70cdb1548320","Type":"ContainerStarted","Data":"1edac906768e905d76f51fe707ec45dc6f5895468c89f92a0f8d021d7a463b8e"}
Jul 02 13:00:08.387506 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:08.387463    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc" podUID="4774483e-4db5-4cd4-ac2c-cd33a7c33331" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Jul 02 13:00:09.858934 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:09.858875    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc" podUID="4774483e-4db5-4cd4-ac2c-cd33a7c33331" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 02 13:00:10.780519 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:10.780495    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc"
Jul 02 13:00:10.837889 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:10.837850    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4774483e-4db5-4cd4-ac2c-cd33a7c33331-kube-rbac-proxy-sar-config\") pod \"4774483e-4db5-4cd4-ac2c-cd33a7c33331\" (UID: \"4774483e-4db5-4cd4-ac2c-cd33a7c33331\") "
Jul 02 13:00:10.838087 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:10.837927    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4774483e-4db5-4cd4-ac2c-cd33a7c33331-cabundle-cert\") pod \"4774483e-4db5-4cd4-ac2c-cd33a7c33331\" (UID: \"4774483e-4db5-4cd4-ac2c-cd33a7c33331\") "
Jul 02 13:00:10.838087 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:10.837954    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4774483e-4db5-4cd4-ac2c-cd33a7c33331-proxy-tls\") pod \"4774483e-4db5-4cd4-ac2c-cd33a7c33331\" (UID: \"4774483e-4db5-4cd4-ac2c-cd33a7c33331\") "
Jul 02 13:00:10.838087 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:10.837995    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wjqb\" (UniqueName: \"kubernetes.io/projected/4774483e-4db5-4cd4-ac2c-cd33a7c33331-kube-api-access-8wjqb\") pod \"4774483e-4db5-4cd4-ac2c-cd33a7c33331\" (UID: \"4774483e-4db5-4cd4-ac2c-cd33a7c33331\") "
Jul 02 13:00:10.838087 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:10.838037    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4774483e-4db5-4cd4-ac2c-cd33a7c33331-kserve-provision-location\") pod \"4774483e-4db5-4cd4-ac2c-cd33a7c33331\" (UID: \"4774483e-4db5-4cd4-ac2c-cd33a7c33331\") "
Jul 02 13:00:10.838300 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:10.838228    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4774483e-4db5-4cd4-ac2c-cd33a7c33331-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4774483e-4db5-4cd4-ac2c-cd33a7c33331" (UID: "4774483e-4db5-4cd4-ac2c-cd33a7c33331"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:00:10.838374 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:10.838359    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4774483e-4db5-4cd4-ac2c-cd33a7c33331-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 13:00:10.838428 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:10.838355    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4774483e-4db5-4cd4-ac2c-cd33a7c33331-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4774483e-4db5-4cd4-ac2c-cd33a7c33331" (UID: "4774483e-4db5-4cd4-ac2c-cd33a7c33331"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:00:10.838487 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:10.838436    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4774483e-4db5-4cd4-ac2c-cd33a7c33331-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4774483e-4db5-4cd4-ac2c-cd33a7c33331" (UID: "4774483e-4db5-4cd4-ac2c-cd33a7c33331"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:00:10.840134 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:10.840103    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4774483e-4db5-4cd4-ac2c-cd33a7c33331-kube-api-access-8wjqb" (OuterVolumeSpecName: "kube-api-access-8wjqb") pod "4774483e-4db5-4cd4-ac2c-cd33a7c33331" (UID: "4774483e-4db5-4cd4-ac2c-cd33a7c33331"). InnerVolumeSpecName "kube-api-access-8wjqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:00:10.840134 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:10.840127    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4774483e-4db5-4cd4-ac2c-cd33a7c33331-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4774483e-4db5-4cd4-ac2c-cd33a7c33331" (UID: "4774483e-4db5-4cd4-ac2c-cd33a7c33331"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:00:10.939593 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:10.939565    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4774483e-4db5-4cd4-ac2c-cd33a7c33331-cabundle-cert\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 13:00:10.939593 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:10.939589    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4774483e-4db5-4cd4-ac2c-cd33a7c33331-proxy-tls\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 13:00:10.939593 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:10.939599    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wjqb\" (UniqueName: \"kubernetes.io/projected/4774483e-4db5-4cd4-ac2c-cd33a7c33331-kube-api-access-8wjqb\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 13:00:10.940011 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:10.939607    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4774483e-4db5-4cd4-ac2c-cd33a7c33331-kserve-provision-location\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 13:00:11.182088 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:11.182057    2576 generic.go:356] "Generic (PLEG): container finished" podID="80847d7c-5d90-449c-816c-70cdb1548320" containerID="55fb31d57c5316cd069d836812c993ca68c185d80c435271b268361095159575" exitCode=0
Jul 02 13:00:11.182260 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:11.182130    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt" event={"ID":"80847d7c-5d90-449c-816c-70cdb1548320","Type":"ContainerDied","Data":"55fb31d57c5316cd069d836812c993ca68c185d80c435271b268361095159575"}
Jul 02 13:00:11.183927 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:11.183894    2576 generic.go:356] "Generic (PLEG): container finished" podID="4774483e-4db5-4cd4-ac2c-cd33a7c33331" containerID="962561d63ced00ab0a83f4205008259241cc3ba0823c6be244dbdd4c9a248770" exitCode=0
Jul 02 13:00:11.184034 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:11.183938    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc" event={"ID":"4774483e-4db5-4cd4-ac2c-cd33a7c33331","Type":"ContainerDied","Data":"962561d63ced00ab0a83f4205008259241cc3ba0823c6be244dbdd4c9a248770"}
Jul 02 13:00:11.184034 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:11.183968    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc"
Jul 02 13:00:11.184034 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:11.183995    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc" event={"ID":"4774483e-4db5-4cd4-ac2c-cd33a7c33331","Type":"ContainerDied","Data":"1beb7198ca9301b5ea144668da53150ff85133204af23ef02795833a0c7e7b4b"}
Jul 02 13:00:11.184034 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:11.184027    2576 scope.go:117] "RemoveContainer" containerID="b3118bf4bd952b28c510227dea6029848b6a19938b4235f6c1980c6f21ff04ba"
Jul 02 13:00:11.194719 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:11.194703    2576 scope.go:117] "RemoveContainer" containerID="962561d63ced00ab0a83f4205008259241cc3ba0823c6be244dbdd4c9a248770"
Jul 02 13:00:11.202251 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:11.202231    2576 scope.go:117] "RemoveContainer" containerID="2c9ecb24fd2bc7632c2722d9f01eddf1de1a3f58b9aedfbf19ac6e191854f865"
Jul 02 13:00:11.217997 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:11.217978    2576 scope.go:117] "RemoveContainer" containerID="b3118bf4bd952b28c510227dea6029848b6a19938b4235f6c1980c6f21ff04ba"
Jul 02 13:00:11.218312 ip-10-0-143-88 kubenswrapper[2576]: E0702 13:00:11.218281    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3118bf4bd952b28c510227dea6029848b6a19938b4235f6c1980c6f21ff04ba\": container with ID starting with b3118bf4bd952b28c510227dea6029848b6a19938b4235f6c1980c6f21ff04ba not found: ID does not exist" containerID="b3118bf4bd952b28c510227dea6029848b6a19938b4235f6c1980c6f21ff04ba"
Jul 02 13:00:11.218415 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:11.218322    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3118bf4bd952b28c510227dea6029848b6a19938b4235f6c1980c6f21ff04ba"} err="failed to get container status \"b3118bf4bd952b28c510227dea6029848b6a19938b4235f6c1980c6f21ff04ba\": rpc error: code = NotFound desc = could not find container \"b3118bf4bd952b28c510227dea6029848b6a19938b4235f6c1980c6f21ff04ba\": container with ID starting with b3118bf4bd952b28c510227dea6029848b6a19938b4235f6c1980c6f21ff04ba not found: ID does not exist"
Jul 02 13:00:11.218415 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:11.218345    2576 scope.go:117] "RemoveContainer" containerID="962561d63ced00ab0a83f4205008259241cc3ba0823c6be244dbdd4c9a248770"
Jul 02 13:00:11.218657 ip-10-0-143-88 kubenswrapper[2576]: E0702 13:00:11.218636    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"962561d63ced00ab0a83f4205008259241cc3ba0823c6be244dbdd4c9a248770\": container with ID starting with 962561d63ced00ab0a83f4205008259241cc3ba0823c6be244dbdd4c9a248770 not found: ID does not exist" containerID="962561d63ced00ab0a83f4205008259241cc3ba0823c6be244dbdd4c9a248770"
Jul 02 13:00:11.218704 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:11.218666    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"962561d63ced00ab0a83f4205008259241cc3ba0823c6be244dbdd4c9a248770"} err="failed to get container status \"962561d63ced00ab0a83f4205008259241cc3ba0823c6be244dbdd4c9a248770\": rpc error: code = NotFound desc = could not find container \"962561d63ced00ab0a83f4205008259241cc3ba0823c6be244dbdd4c9a248770\": container with ID starting with 962561d63ced00ab0a83f4205008259241cc3ba0823c6be244dbdd4c9a248770 not found: ID does not exist"
Jul 02 13:00:11.218704 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:11.218682    2576 scope.go:117] "RemoveContainer" containerID="2c9ecb24fd2bc7632c2722d9f01eddf1de1a3f58b9aedfbf19ac6e191854f865"
Jul 02 13:00:11.218965 ip-10-0-143-88 kubenswrapper[2576]: E0702 13:00:11.218947    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c9ecb24fd2bc7632c2722d9f01eddf1de1a3f58b9aedfbf19ac6e191854f865\": container with ID starting with 2c9ecb24fd2bc7632c2722d9f01eddf1de1a3f58b9aedfbf19ac6e191854f865 not found: ID does not exist" containerID="2c9ecb24fd2bc7632c2722d9f01eddf1de1a3f58b9aedfbf19ac6e191854f865"
Jul 02 13:00:11.219016 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:11.218971    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c9ecb24fd2bc7632c2722d9f01eddf1de1a3f58b9aedfbf19ac6e191854f865"} err="failed to get container status \"2c9ecb24fd2bc7632c2722d9f01eddf1de1a3f58b9aedfbf19ac6e191854f865\": rpc error: code = NotFound desc = could not find container \"2c9ecb24fd2bc7632c2722d9f01eddf1de1a3f58b9aedfbf19ac6e191854f865\": container with ID starting with 2c9ecb24fd2bc7632c2722d9f01eddf1de1a3f58b9aedfbf19ac6e191854f865 not found: ID does not exist"
Jul 02 13:00:11.230741 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:11.230718    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc"]
Jul 02 13:00:11.239852 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:11.239832    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-56395-predictor-5845d646b4-nxtmc"]
Jul 02 13:00:12.190805 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:12.190769    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt" event={"ID":"80847d7c-5d90-449c-816c-70cdb1548320","Type":"ContainerStarted","Data":"b637383f42527f395dcfd4cc2e645dd32d3c30469d3e0d5ad5ea94537bf48895"}
Jul 02 13:00:12.191292 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:12.190873    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt" event={"ID":"80847d7c-5d90-449c-816c-70cdb1548320","Type":"ContainerStarted","Data":"e8b48a057858649c8c75fb26e0678616bc65f4106521ca640305b0827aefcf24"}
Jul 02 13:00:12.191292 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:12.191236    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt"
Jul 02 13:00:12.191438 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:12.191394    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt"
Jul 02 13:00:12.192944 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:12.192893    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt" podUID="80847d7c-5d90-449c-816c-70cdb1548320" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 02 13:00:12.213109 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:12.213053    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt" podStartSLOduration=2.697036856 podStartE2EDuration="6.213040403s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.516003547s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:00:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:00:12.212005418 +0000 UTC m=+1300.819468442" watchObservedRunningTime="2026-07-02 13:00:12.213040403 +0000 UTC m=+1300.820503426"
Jul 02 13:00:12.862626 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:12.862594    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4774483e-4db5-4cd4-ac2c-cd33a7c33331" path="/var/lib/kubelet/pods/4774483e-4db5-4cd4-ac2c-cd33a7c33331/volumes"
Jul 02 13:00:13.197067 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:13.197028    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt" podUID="80847d7c-5d90-449c-816c-70cdb1548320" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 02 13:00:18.202620 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:18.202595    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt"
Jul 02 13:00:18.203219 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:18.203193    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt" podUID="80847d7c-5d90-449c-816c-70cdb1548320" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 02 13:00:28.203525 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:28.203479    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt" podUID="80847d7c-5d90-449c-816c-70cdb1548320" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 02 13:00:38.203712 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:38.203678    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt" podUID="80847d7c-5d90-449c-816c-70cdb1548320" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 02 13:00:48.203498 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:48.203457    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt" podUID="80847d7c-5d90-449c-816c-70cdb1548320" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 02 13:00:58.204067 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:00:58.204025    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt" podUID="80847d7c-5d90-449c-816c-70cdb1548320" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 02 13:01:08.204073 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:08.204032    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt" podUID="80847d7c-5d90-449c-816c-70cdb1548320" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 02 13:01:18.203337 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:18.203299    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt" podUID="80847d7c-5d90-449c-816c-70cdb1548320" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 02 13:01:28.204062 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:28.204033    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt"
Jul 02 13:01:36.519821 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:36.519780    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq"]
Jul 02 13:01:36.520395 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:36.520369    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4774483e-4db5-4cd4-ac2c-cd33a7c33331" containerName="kserve-container"
Jul 02 13:01:36.520395 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:36.520389    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4774483e-4db5-4cd4-ac2c-cd33a7c33331" containerName="kserve-container"
Jul 02 13:01:36.520520 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:36.520414    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4774483e-4db5-4cd4-ac2c-cd33a7c33331" containerName="storage-initializer"
Jul 02 13:01:36.520520 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:36.520423    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4774483e-4db5-4cd4-ac2c-cd33a7c33331" containerName="storage-initializer"
Jul 02 13:01:36.520520 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:36.520460    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4774483e-4db5-4cd4-ac2c-cd33a7c33331" containerName="kube-rbac-proxy"
Jul 02 13:01:36.520520 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:36.520469    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4774483e-4db5-4cd4-ac2c-cd33a7c33331" containerName="kube-rbac-proxy"
Jul 02 13:01:36.520726 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:36.520552    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4774483e-4db5-4cd4-ac2c-cd33a7c33331" containerName="kube-rbac-proxy"
Jul 02 13:01:36.520726 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:36.520566    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4774483e-4db5-4cd4-ac2c-cd33a7c33331" containerName="kserve-container"
Jul 02 13:01:36.523952 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:36.523931    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq"
Jul 02 13:01:36.526579 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:36.526558    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-c9e60f-kube-rbac-proxy-sar-config\""
Jul 02 13:01:36.526705 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:36.526588    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-c9e60f-predictor-serving-cert\""
Jul 02 13:01:36.526705 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:36.526587    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-c9e60f\""
Jul 02 13:01:36.526830 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:36.526817    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-c9e60f-dockercfg-j944n\""
Jul 02 13:01:36.540738 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:36.540709    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq"]
Jul 02 13:01:36.705711 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:36.705676    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86gjd\" (UniqueName: \"kubernetes.io/projected/c698ea02-a654-4343-86af-86adba47fbd8-kube-api-access-86gjd\") pod \"isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq\" (UID: \"c698ea02-a654-4343-86af-86adba47fbd8\") " pod="kserve-ci-e2e-test/isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq"
Jul 02 13:01:36.705875 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:36.705739    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c698ea02-a654-4343-86af-86adba47fbd8-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq\" (UID: \"c698ea02-a654-4343-86af-86adba47fbd8\") " pod="kserve-ci-e2e-test/isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq"
Jul 02 13:01:36.705875 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:36.705797    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c698ea02-a654-4343-86af-86adba47fbd8-cabundle-cert\") pod \"isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq\" (UID: \"c698ea02-a654-4343-86af-86adba47fbd8\") " pod="kserve-ci-e2e-test/isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq"
Jul 02 13:01:36.705875 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:36.705848    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c698ea02-a654-4343-86af-86adba47fbd8-proxy-tls\") pod \"isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq\" (UID: \"c698ea02-a654-4343-86af-86adba47fbd8\") " pod="kserve-ci-e2e-test/isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq"
Jul 02 13:01:36.705875 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:36.705871    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c698ea02-a654-4343-86af-86adba47fbd8-kserve-provision-location\") pod \"isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq\" (UID: \"c698ea02-a654-4343-86af-86adba47fbd8\") " pod="kserve-ci-e2e-test/isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq"
Jul 02 13:01:36.807372 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:36.807291    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c698ea02-a654-4343-86af-86adba47fbd8-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq\" (UID: \"c698ea02-a654-4343-86af-86adba47fbd8\") " pod="kserve-ci-e2e-test/isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq"
Jul 02 13:01:36.807372 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:36.807328    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c698ea02-a654-4343-86af-86adba47fbd8-cabundle-cert\") pod \"isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq\" (UID: \"c698ea02-a654-4343-86af-86adba47fbd8\") " pod="kserve-ci-e2e-test/isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq"
Jul 02 13:01:36.807372 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:36.807365    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c698ea02-a654-4343-86af-86adba47fbd8-proxy-tls\") pod \"isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq\" (UID: \"c698ea02-a654-4343-86af-86adba47fbd8\") " pod="kserve-ci-e2e-test/isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq"
Jul 02 13:01:36.807628 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:36.807384    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c698ea02-a654-4343-86af-86adba47fbd8-kserve-provision-location\") pod \"isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq\" (UID: \"c698ea02-a654-4343-86af-86adba47fbd8\") " pod="kserve-ci-e2e-test/isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq"
Jul 02 13:01:36.807628 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:36.807422    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86gjd\" (UniqueName: \"kubernetes.io/projected/c698ea02-a654-4343-86af-86adba47fbd8-kube-api-access-86gjd\") pod \"isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq\" (UID: \"c698ea02-a654-4343-86af-86adba47fbd8\") " pod="kserve-ci-e2e-test/isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq"
Jul 02 13:01:36.807840 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:36.807816    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c698ea02-a654-4343-86af-86adba47fbd8-kserve-provision-location\") pod \"isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq\" (UID: \"c698ea02-a654-4343-86af-86adba47fbd8\") " pod="kserve-ci-e2e-test/isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq"
Jul 02 13:01:36.808005 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:36.807984    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c698ea02-a654-4343-86af-86adba47fbd8-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq\" (UID: \"c698ea02-a654-4343-86af-86adba47fbd8\") " pod="kserve-ci-e2e-test/isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq"
Jul 02 13:01:36.808067 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:36.808050    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c698ea02-a654-4343-86af-86adba47fbd8-cabundle-cert\") pod \"isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq\" (UID: \"c698ea02-a654-4343-86af-86adba47fbd8\") " pod="kserve-ci-e2e-test/isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq"
Jul 02 13:01:36.809876 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:36.809859    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c698ea02-a654-4343-86af-86adba47fbd8-proxy-tls\") pod \"isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq\" (UID: \"c698ea02-a654-4343-86af-86adba47fbd8\") " pod="kserve-ci-e2e-test/isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq"
Jul 02 13:01:36.817296 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:36.817273    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86gjd\" (UniqueName: \"kubernetes.io/projected/c698ea02-a654-4343-86af-86adba47fbd8-kube-api-access-86gjd\") pod \"isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq\" (UID: \"c698ea02-a654-4343-86af-86adba47fbd8\") " pod="kserve-ci-e2e-test/isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq"
Jul 02 13:01:36.834230 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:36.834207    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq"
Jul 02 13:01:36.968613 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:36.968586    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq"]
Jul 02 13:01:36.970849 ip-10-0-143-88 kubenswrapper[2576]: W0702 13:01:36.970820    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc698ea02_a654_4343_86af_86adba47fbd8.slice/crio-949931410c38787c270d5c9ad00bd273aebeed3efdd2b0e1bdb8b3d98488880b WatchSource:0}: Error finding container 949931410c38787c270d5c9ad00bd273aebeed3efdd2b0e1bdb8b3d98488880b: Status 404 returned error can't find the container with id 949931410c38787c270d5c9ad00bd273aebeed3efdd2b0e1bdb8b3d98488880b
Jul 02 13:01:37.592129 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:37.592089    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq" event={"ID":"c698ea02-a654-4343-86af-86adba47fbd8","Type":"ContainerStarted","Data":"536a6ae12ee1939a8184b4f4976a8ba5ea9cb171df18ce89ea4290b5a1316408"}
Jul 02 13:01:37.592541 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:37.592150    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq" event={"ID":"c698ea02-a654-4343-86af-86adba47fbd8","Type":"ContainerStarted","Data":"949931410c38787c270d5c9ad00bd273aebeed3efdd2b0e1bdb8b3d98488880b"}
Jul 02 13:01:43.624525 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:43.624497    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq_c698ea02-a654-4343-86af-86adba47fbd8/storage-initializer/0.log"
Jul 02 13:01:43.624887 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:43.624535    2576 generic.go:356] "Generic (PLEG): container finished" podID="c698ea02-a654-4343-86af-86adba47fbd8" containerID="536a6ae12ee1939a8184b4f4976a8ba5ea9cb171df18ce89ea4290b5a1316408" exitCode=1
Jul 02 13:01:43.624887 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:43.624620    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq" event={"ID":"c698ea02-a654-4343-86af-86adba47fbd8","Type":"ContainerDied","Data":"536a6ae12ee1939a8184b4f4976a8ba5ea9cb171df18ce89ea4290b5a1316408"}
Jul 02 13:01:44.630959 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:44.630937    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq_c698ea02-a654-4343-86af-86adba47fbd8/storage-initializer/0.log"
Jul 02 13:01:44.631336 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:44.631007    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq" event={"ID":"c698ea02-a654-4343-86af-86adba47fbd8","Type":"ContainerStarted","Data":"13ad0bd76ef076c39ff071d113144296cc1b7014f546caea3dd1d7a023a6e773"}
Jul 02 13:01:48.651207 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:48.651181    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq_c698ea02-a654-4343-86af-86adba47fbd8/storage-initializer/1.log"
Jul 02 13:01:48.651612 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:48.651593    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq_c698ea02-a654-4343-86af-86adba47fbd8/storage-initializer/0.log"
Jul 02 13:01:48.651691 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:48.651636    2576 generic.go:356] "Generic (PLEG): container finished" podID="c698ea02-a654-4343-86af-86adba47fbd8" containerID="13ad0bd76ef076c39ff071d113144296cc1b7014f546caea3dd1d7a023a6e773" exitCode=1
Jul 02 13:01:48.651745 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:48.651708    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq" event={"ID":"c698ea02-a654-4343-86af-86adba47fbd8","Type":"ContainerDied","Data":"13ad0bd76ef076c39ff071d113144296cc1b7014f546caea3dd1d7a023a6e773"}
Jul 02 13:01:48.651799 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:48.651775    2576 scope.go:117] "RemoveContainer" containerID="536a6ae12ee1939a8184b4f4976a8ba5ea9cb171df18ce89ea4290b5a1316408"
Jul 02 13:01:48.652145 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:48.652124    2576 scope.go:117] "RemoveContainer" containerID="536a6ae12ee1939a8184b4f4976a8ba5ea9cb171df18ce89ea4290b5a1316408"
Jul 02 13:01:48.662471 ip-10-0-143-88 kubenswrapper[2576]: E0702 13:01:48.662448    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq_kserve-ci-e2e-test(c698ea02-a654-4343-86af-86adba47fbd8)\"" pod="kserve-ci-e2e-test/isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq" podUID="c698ea02-a654-4343-86af-86adba47fbd8"
Jul 02 13:01:48.662556 ip-10-0-143-88 kubenswrapper[2576]: E0702 13:01:48.662538    2576 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq_kserve-ci-e2e-test_c698ea02-a654-4343-86af-86adba47fbd8_0 in pod sandbox 949931410c38787c270d5c9ad00bd273aebeed3efdd2b0e1bdb8b3d98488880b from index: no such id: '536a6ae12ee1939a8184b4f4976a8ba5ea9cb171df18ce89ea4290b5a1316408'" containerID="536a6ae12ee1939a8184b4f4976a8ba5ea9cb171df18ce89ea4290b5a1316408"
Jul 02 13:01:48.662604 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:48.662569    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"536a6ae12ee1939a8184b4f4976a8ba5ea9cb171df18ce89ea4290b5a1316408"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq_kserve-ci-e2e-test_c698ea02-a654-4343-86af-86adba47fbd8_0 in pod sandbox 949931410c38787c270d5c9ad00bd273aebeed3efdd2b0e1bdb8b3d98488880b from index: no such id: '536a6ae12ee1939a8184b4f4976a8ba5ea9cb171df18ce89ea4290b5a1316408'"
Jul 02 13:01:49.657968 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:49.657939    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq_c698ea02-a654-4343-86af-86adba47fbd8/storage-initializer/1.log"
Jul 02 13:01:54.568973 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:54.568938    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq"]
Jul 02 13:01:54.656301 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:54.656265    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt"]
Jul 02 13:01:54.656743 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:54.656538    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt" podUID="80847d7c-5d90-449c-816c-70cdb1548320" containerName="kserve-container" containerID="cri-o://e8b48a057858649c8c75fb26e0678616bc65f4106521ca640305b0827aefcf24" gracePeriod=30
Jul 02 13:01:54.656743 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:54.656594    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt" podUID="80847d7c-5d90-449c-816c-70cdb1548320" containerName="kube-rbac-proxy" containerID="cri-o://b637383f42527f395dcfd4cc2e645dd32d3c30469d3e0d5ad5ea94537bf48895" gracePeriod=30
Jul 02 13:01:54.778556 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:54.778537    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq_c698ea02-a654-4343-86af-86adba47fbd8/storage-initializer/1.log"
Jul 02 13:01:54.780649 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:54.780633    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq"
Jul 02 13:01:54.860521 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:54.860448    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c698ea02-a654-4343-86af-86adba47fbd8-kube-rbac-proxy-sar-config\") pod \"c698ea02-a654-4343-86af-86adba47fbd8\" (UID: \"c698ea02-a654-4343-86af-86adba47fbd8\") "
Jul 02 13:01:54.860521 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:54.860520    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c698ea02-a654-4343-86af-86adba47fbd8-proxy-tls\") pod \"c698ea02-a654-4343-86af-86adba47fbd8\" (UID: \"c698ea02-a654-4343-86af-86adba47fbd8\") "
Jul 02 13:01:54.860720 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:54.860546    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86gjd\" (UniqueName: \"kubernetes.io/projected/c698ea02-a654-4343-86af-86adba47fbd8-kube-api-access-86gjd\") pod \"c698ea02-a654-4343-86af-86adba47fbd8\" (UID: \"c698ea02-a654-4343-86af-86adba47fbd8\") "
Jul 02 13:01:54.860720 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:54.860589    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c698ea02-a654-4343-86af-86adba47fbd8-cabundle-cert\") pod \"c698ea02-a654-4343-86af-86adba47fbd8\" (UID: \"c698ea02-a654-4343-86af-86adba47fbd8\") "
Jul 02 13:01:54.860720 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:54.860631    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c698ea02-a654-4343-86af-86adba47fbd8-kserve-provision-location\") pod \"c698ea02-a654-4343-86af-86adba47fbd8\" (UID: \"c698ea02-a654-4343-86af-86adba47fbd8\") "
Jul 02 13:01:54.860894 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:54.860815    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c698ea02-a654-4343-86af-86adba47fbd8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c698ea02-a654-4343-86af-86adba47fbd8" (UID: "c698ea02-a654-4343-86af-86adba47fbd8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:01:54.860972 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:54.860952    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c698ea02-a654-4343-86af-86adba47fbd8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 13:01:54.861018 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:54.860994    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c698ea02-a654-4343-86af-86adba47fbd8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c698ea02-a654-4343-86af-86adba47fbd8" (UID: "c698ea02-a654-4343-86af-86adba47fbd8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:01:54.861057 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:54.861022    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c698ea02-a654-4343-86af-86adba47fbd8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c698ea02-a654-4343-86af-86adba47fbd8" (UID: "c698ea02-a654-4343-86af-86adba47fbd8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:01:54.862826 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:54.862796    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c698ea02-a654-4343-86af-86adba47fbd8-kube-api-access-86gjd" (OuterVolumeSpecName: "kube-api-access-86gjd") pod "c698ea02-a654-4343-86af-86adba47fbd8" (UID: "c698ea02-a654-4343-86af-86adba47fbd8"). InnerVolumeSpecName "kube-api-access-86gjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:01:54.862982 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:54.862885    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c698ea02-a654-4343-86af-86adba47fbd8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c698ea02-a654-4343-86af-86adba47fbd8" (UID: "c698ea02-a654-4343-86af-86adba47fbd8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:01:54.962397 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:54.962364    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c698ea02-a654-4343-86af-86adba47fbd8-proxy-tls\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 13:01:54.962397 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:54.962392    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86gjd\" (UniqueName: \"kubernetes.io/projected/c698ea02-a654-4343-86af-86adba47fbd8-kube-api-access-86gjd\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 13:01:54.962397 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:54.962401    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c698ea02-a654-4343-86af-86adba47fbd8-cabundle-cert\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 13:01:54.962595 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:54.962411    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c698ea02-a654-4343-86af-86adba47fbd8-kserve-provision-location\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 13:01:55.689471 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:55.689442    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq_c698ea02-a654-4343-86af-86adba47fbd8/storage-initializer/1.log"
Jul 02 13:01:55.689896 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:55.689541    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq"
Jul 02 13:01:55.689896 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:55.689578    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq" event={"ID":"c698ea02-a654-4343-86af-86adba47fbd8","Type":"ContainerDied","Data":"949931410c38787c270d5c9ad00bd273aebeed3efdd2b0e1bdb8b3d98488880b"}
Jul 02 13:01:55.689896 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:55.689644    2576 scope.go:117] "RemoveContainer" containerID="13ad0bd76ef076c39ff071d113144296cc1b7014f546caea3dd1d7a023a6e773"
Jul 02 13:01:55.692339 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:55.692312    2576 generic.go:356] "Generic (PLEG): container finished" podID="80847d7c-5d90-449c-816c-70cdb1548320" containerID="b637383f42527f395dcfd4cc2e645dd32d3c30469d3e0d5ad5ea94537bf48895" exitCode=2
Jul 02 13:01:55.692478 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:55.692394    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt" event={"ID":"80847d7c-5d90-449c-816c-70cdb1548320","Type":"ContainerDied","Data":"b637383f42527f395dcfd4cc2e645dd32d3c30469d3e0d5ad5ea94537bf48895"}
Jul 02 13:01:55.724230 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:55.724207    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq"]
Jul 02 13:01:55.731114 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:55.731093    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-c9e60f-predictor-d9496c8c6-fhbcq"]
Jul 02 13:01:56.868754 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:56.868720    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c698ea02-a654-4343-86af-86adba47fbd8" path="/var/lib/kubelet/pods/c698ea02-a654-4343-86af-86adba47fbd8/volumes"
Jul 02 13:01:58.197442 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:58.197402    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt" podUID="80847d7c-5d90-449c-816c-70cdb1548320" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Jul 02 13:01:58.203205 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:58.203175    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt" podUID="80847d7c-5d90-449c-816c-70cdb1548320" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 02 13:01:59.214778 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.214757    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt"
Jul 02 13:01:59.300563 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.300483    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80847d7c-5d90-449c-816c-70cdb1548320-proxy-tls\") pod \"80847d7c-5d90-449c-816c-70cdb1548320\" (UID: \"80847d7c-5d90-449c-816c-70cdb1548320\") "
Jul 02 13:01:59.300563 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.300517    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80847d7c-5d90-449c-816c-70cdb1548320-cabundle-cert\") pod \"80847d7c-5d90-449c-816c-70cdb1548320\" (UID: \"80847d7c-5d90-449c-816c-70cdb1548320\") "
Jul 02 13:01:59.300563 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.300552    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80847d7c-5d90-449c-816c-70cdb1548320-kube-rbac-proxy-sar-config\") pod \"80847d7c-5d90-449c-816c-70cdb1548320\" (UID: \"80847d7c-5d90-449c-816c-70cdb1548320\") "
Jul 02 13:01:59.300828 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.300585    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnpth\" (UniqueName: \"kubernetes.io/projected/80847d7c-5d90-449c-816c-70cdb1548320-kube-api-access-tnpth\") pod \"80847d7c-5d90-449c-816c-70cdb1548320\" (UID: \"80847d7c-5d90-449c-816c-70cdb1548320\") "
Jul 02 13:01:59.300828 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.300663    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80847d7c-5d90-449c-816c-70cdb1548320-kserve-provision-location\") pod \"80847d7c-5d90-449c-816c-70cdb1548320\" (UID: \"80847d7c-5d90-449c-816c-70cdb1548320\") "
Jul 02 13:01:59.300962 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.300926    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80847d7c-5d90-449c-816c-70cdb1548320-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "80847d7c-5d90-449c-816c-70cdb1548320" (UID: "80847d7c-5d90-449c-816c-70cdb1548320"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:01:59.300962 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.300948    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80847d7c-5d90-449c-816c-70cdb1548320-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "80847d7c-5d90-449c-816c-70cdb1548320" (UID: "80847d7c-5d90-449c-816c-70cdb1548320"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:01:59.301102 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.301078    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80847d7c-5d90-449c-816c-70cdb1548320-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "80847d7c-5d90-449c-816c-70cdb1548320" (UID: "80847d7c-5d90-449c-816c-70cdb1548320"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:01:59.302682 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.302654    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80847d7c-5d90-449c-816c-70cdb1548320-kube-api-access-tnpth" (OuterVolumeSpecName: "kube-api-access-tnpth") pod "80847d7c-5d90-449c-816c-70cdb1548320" (UID: "80847d7c-5d90-449c-816c-70cdb1548320"). InnerVolumeSpecName "kube-api-access-tnpth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:01:59.302764 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.302698    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80847d7c-5d90-449c-816c-70cdb1548320-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "80847d7c-5d90-449c-816c-70cdb1548320" (UID: "80847d7c-5d90-449c-816c-70cdb1548320"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:01:59.402021 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.401992    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80847d7c-5d90-449c-816c-70cdb1548320-kserve-provision-location\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 13:01:59.402021 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.402014    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80847d7c-5d90-449c-816c-70cdb1548320-proxy-tls\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 13:01:59.402021 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.402026    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80847d7c-5d90-449c-816c-70cdb1548320-cabundle-cert\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 13:01:59.402214 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.402034    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80847d7c-5d90-449c-816c-70cdb1548320-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 13:01:59.402214 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.402042    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnpth\" (UniqueName: \"kubernetes.io/projected/80847d7c-5d90-449c-816c-70cdb1548320-kube-api-access-tnpth\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 13:01:59.715605 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.715569    2576 generic.go:356] "Generic (PLEG): container finished" podID="80847d7c-5d90-449c-816c-70cdb1548320" containerID="e8b48a057858649c8c75fb26e0678616bc65f4106521ca640305b0827aefcf24" exitCode=0
Jul 02 13:01:59.715769 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.715612    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt" event={"ID":"80847d7c-5d90-449c-816c-70cdb1548320","Type":"ContainerDied","Data":"e8b48a057858649c8c75fb26e0678616bc65f4106521ca640305b0827aefcf24"}
Jul 02 13:01:59.715769 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.715627    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt"
Jul 02 13:01:59.715769 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.715658    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt" event={"ID":"80847d7c-5d90-449c-816c-70cdb1548320","Type":"ContainerDied","Data":"1edac906768e905d76f51fe707ec45dc6f5895468c89f92a0f8d021d7a463b8e"}
Jul 02 13:01:59.715769 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.715684    2576 scope.go:117] "RemoveContainer" containerID="b637383f42527f395dcfd4cc2e645dd32d3c30469d3e0d5ad5ea94537bf48895"
Jul 02 13:01:59.725274 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.725257    2576 scope.go:117] "RemoveContainer" containerID="e8b48a057858649c8c75fb26e0678616bc65f4106521ca640305b0827aefcf24"
Jul 02 13:01:59.732604 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.732587    2576 scope.go:117] "RemoveContainer" containerID="55fb31d57c5316cd069d836812c993ca68c185d80c435271b268361095159575"
Jul 02 13:01:59.740471 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.740346    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt"]
Jul 02 13:01:59.740471 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.740430    2576 scope.go:117] "RemoveContainer" containerID="b637383f42527f395dcfd4cc2e645dd32d3c30469d3e0d5ad5ea94537bf48895"
Jul 02 13:01:59.740669 ip-10-0-143-88 kubenswrapper[2576]: E0702 13:01:59.740652    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b637383f42527f395dcfd4cc2e645dd32d3c30469d3e0d5ad5ea94537bf48895\": container with ID starting with b637383f42527f395dcfd4cc2e645dd32d3c30469d3e0d5ad5ea94537bf48895 not found: ID does not exist" containerID="b637383f42527f395dcfd4cc2e645dd32d3c30469d3e0d5ad5ea94537bf48895"
Jul 02 13:01:59.740707 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.740676    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b637383f42527f395dcfd4cc2e645dd32d3c30469d3e0d5ad5ea94537bf48895"} err="failed to get container status \"b637383f42527f395dcfd4cc2e645dd32d3c30469d3e0d5ad5ea94537bf48895\": rpc error: code = NotFound desc = could not find container \"b637383f42527f395dcfd4cc2e645dd32d3c30469d3e0d5ad5ea94537bf48895\": container with ID starting with b637383f42527f395dcfd4cc2e645dd32d3c30469d3e0d5ad5ea94537bf48895 not found: ID does not exist"
Jul 02 13:01:59.740707 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.740692    2576 scope.go:117] "RemoveContainer" containerID="e8b48a057858649c8c75fb26e0678616bc65f4106521ca640305b0827aefcf24"
Jul 02 13:01:59.740957 ip-10-0-143-88 kubenswrapper[2576]: E0702 13:01:59.740937    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8b48a057858649c8c75fb26e0678616bc65f4106521ca640305b0827aefcf24\": container with ID starting with e8b48a057858649c8c75fb26e0678616bc65f4106521ca640305b0827aefcf24 not found: ID does not exist" containerID="e8b48a057858649c8c75fb26e0678616bc65f4106521ca640305b0827aefcf24"
Jul 02 13:01:59.741021 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.740962    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8b48a057858649c8c75fb26e0678616bc65f4106521ca640305b0827aefcf24"} err="failed to get container status \"e8b48a057858649c8c75fb26e0678616bc65f4106521ca640305b0827aefcf24\": rpc error: code = NotFound desc = could not find container \"e8b48a057858649c8c75fb26e0678616bc65f4106521ca640305b0827aefcf24\": container with ID starting with e8b48a057858649c8c75fb26e0678616bc65f4106521ca640305b0827aefcf24 not found: ID does not exist"
Jul 02 13:01:59.741021 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.740975    2576 scope.go:117] "RemoveContainer" containerID="55fb31d57c5316cd069d836812c993ca68c185d80c435271b268361095159575"
Jul 02 13:01:59.741204 ip-10-0-143-88 kubenswrapper[2576]: E0702 13:01:59.741186    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55fb31d57c5316cd069d836812c993ca68c185d80c435271b268361095159575\": container with ID starting with 55fb31d57c5316cd069d836812c993ca68c185d80c435271b268361095159575 not found: ID does not exist" containerID="55fb31d57c5316cd069d836812c993ca68c185d80c435271b268361095159575"
Jul 02 13:01:59.741257 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.741213    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55fb31d57c5316cd069d836812c993ca68c185d80c435271b268361095159575"} err="failed to get container status \"55fb31d57c5316cd069d836812c993ca68c185d80c435271b268361095159575\": rpc error: code = NotFound desc = could not find container \"55fb31d57c5316cd069d836812c993ca68c185d80c435271b268361095159575\": container with ID starting with 55fb31d57c5316cd069d836812c993ca68c185d80c435271b268361095159575 not found: ID does not exist"
Jul 02 13:01:59.744472 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.744451    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-c9e60f-predictor-6d4687f95f-6qmdt"]
Jul 02 13:01:59.764595 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.764573    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j"]
Jul 02 13:01:59.764974 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.764961    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c698ea02-a654-4343-86af-86adba47fbd8" containerName="storage-initializer"
Jul 02 13:01:59.765075 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.764975    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c698ea02-a654-4343-86af-86adba47fbd8" containerName="storage-initializer"
Jul 02 13:01:59.765075 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.764993    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c698ea02-a654-4343-86af-86adba47fbd8" containerName="storage-initializer"
Jul 02 13:01:59.765075 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.764998    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c698ea02-a654-4343-86af-86adba47fbd8" containerName="storage-initializer"
Jul 02 13:01:59.765075 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.765005    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80847d7c-5d90-449c-816c-70cdb1548320" containerName="storage-initializer"
Jul 02 13:01:59.765075 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.765011    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="80847d7c-5d90-449c-816c-70cdb1548320" containerName="storage-initializer"
Jul 02 13:01:59.765075 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.765016    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80847d7c-5d90-449c-816c-70cdb1548320" containerName="kserve-container"
Jul 02 13:01:59.765075 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.765020    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="80847d7c-5d90-449c-816c-70cdb1548320" containerName="kserve-container"
Jul 02 13:01:59.765075 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.765025    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80847d7c-5d90-449c-816c-70cdb1548320" containerName="kube-rbac-proxy"
Jul 02 13:01:59.765075 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.765030    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="80847d7c-5d90-449c-816c-70cdb1548320" containerName="kube-rbac-proxy"
Jul 02 13:01:59.765335 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.765103    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c698ea02-a654-4343-86af-86adba47fbd8" containerName="storage-initializer"
Jul 02 13:01:59.765335 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.765115    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="80847d7c-5d90-449c-816c-70cdb1548320" containerName="kube-rbac-proxy"
Jul 02 13:01:59.765335 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.765126    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="80847d7c-5d90-449c-816c-70cdb1548320" containerName="kserve-container"
Jul 02 13:01:59.765335 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.765253    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c698ea02-a654-4343-86af-86adba47fbd8" containerName="storage-initializer"
Jul 02 13:01:59.770527 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.770512    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j"
Jul 02 13:01:59.773104 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.773084    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 13:01:59.773204 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.773096    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-f3d602-kube-rbac-proxy-sar-config\""
Jul 02 13:01:59.773204 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.773091    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 13:01:59.773204 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.773109    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-f3d602-predictor-serving-cert\""
Jul 02 13:01:59.773385 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.773265    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-f3d602-dockercfg-nkg5p\""
Jul 02 13:01:59.773499 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.773483    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-f3d602\""
Jul 02 13:01:59.773735 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.773721    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 13:01:59.779806 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.779786    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j"]
Jul 02 13:01:59.906735 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.906702    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9faee4ca-523f-4a0c-80a4-1373ba260eaf-kserve-provision-location\") pod \"isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j\" (UID: \"9faee4ca-523f-4a0c-80a4-1373ba260eaf\") " pod="kserve-ci-e2e-test/isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j"
Jul 02 13:01:59.906887 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.906746    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szc6s\" (UniqueName: \"kubernetes.io/projected/9faee4ca-523f-4a0c-80a4-1373ba260eaf-kube-api-access-szc6s\") pod \"isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j\" (UID: \"9faee4ca-523f-4a0c-80a4-1373ba260eaf\") " pod="kserve-ci-e2e-test/isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j"
Jul 02 13:01:59.906887 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.906853    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9faee4ca-523f-4a0c-80a4-1373ba260eaf-proxy-tls\") pod \"isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j\" (UID: \"9faee4ca-523f-4a0c-80a4-1373ba260eaf\") " pod="kserve-ci-e2e-test/isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j"
Jul 02 13:01:59.906887 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.906881    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9faee4ca-523f-4a0c-80a4-1373ba260eaf-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j\" (UID: \"9faee4ca-523f-4a0c-80a4-1373ba260eaf\") " pod="kserve-ci-e2e-test/isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j"
Jul 02 13:01:59.907020 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:01:59.906928    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9faee4ca-523f-4a0c-80a4-1373ba260eaf-cabundle-cert\") pod \"isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j\" (UID: \"9faee4ca-523f-4a0c-80a4-1373ba260eaf\") " pod="kserve-ci-e2e-test/isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j"
Jul 02 13:02:00.008183 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:00.008104    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szc6s\" (UniqueName: \"kubernetes.io/projected/9faee4ca-523f-4a0c-80a4-1373ba260eaf-kube-api-access-szc6s\") pod \"isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j\" (UID: \"9faee4ca-523f-4a0c-80a4-1373ba260eaf\") " pod="kserve-ci-e2e-test/isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j"
Jul 02 13:02:00.008183 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:00.008169    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9faee4ca-523f-4a0c-80a4-1373ba260eaf-proxy-tls\") pod \"isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j\" (UID: \"9faee4ca-523f-4a0c-80a4-1373ba260eaf\") " pod="kserve-ci-e2e-test/isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j"
Jul 02 13:02:00.008408 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:00.008190    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9faee4ca-523f-4a0c-80a4-1373ba260eaf-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j\" (UID: \"9faee4ca-523f-4a0c-80a4-1373ba260eaf\") " pod="kserve-ci-e2e-test/isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j"
Jul 02 13:02:00.008408 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:00.008208    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9faee4ca-523f-4a0c-80a4-1373ba260eaf-cabundle-cert\") pod \"isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j\" (UID: \"9faee4ca-523f-4a0c-80a4-1373ba260eaf\") " pod="kserve-ci-e2e-test/isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j"
Jul 02 13:02:00.008408 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:00.008245    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9faee4ca-523f-4a0c-80a4-1373ba260eaf-kserve-provision-location\") pod \"isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j\" (UID: \"9faee4ca-523f-4a0c-80a4-1373ba260eaf\") " pod="kserve-ci-e2e-test/isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j"
Jul 02 13:02:00.008655 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:00.008633    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9faee4ca-523f-4a0c-80a4-1373ba260eaf-kserve-provision-location\") pod \"isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j\" (UID: \"9faee4ca-523f-4a0c-80a4-1373ba260eaf\") " pod="kserve-ci-e2e-test/isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j"
Jul 02 13:02:00.008969 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:00.008951    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9faee4ca-523f-4a0c-80a4-1373ba260eaf-cabundle-cert\") pod \"isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j\" (UID: \"9faee4ca-523f-4a0c-80a4-1373ba260eaf\") " pod="kserve-ci-e2e-test/isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j"
Jul 02 13:02:00.009026 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:00.008969    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9faee4ca-523f-4a0c-80a4-1373ba260eaf-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j\" (UID: \"9faee4ca-523f-4a0c-80a4-1373ba260eaf\") " pod="kserve-ci-e2e-test/isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j"
Jul 02 13:02:00.010656 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:00.010634    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9faee4ca-523f-4a0c-80a4-1373ba260eaf-proxy-tls\") pod \"isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j\" (UID: \"9faee4ca-523f-4a0c-80a4-1373ba260eaf\") " pod="kserve-ci-e2e-test/isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j"
Jul 02 13:02:00.016772 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:00.016742    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szc6s\" (UniqueName: \"kubernetes.io/projected/9faee4ca-523f-4a0c-80a4-1373ba260eaf-kube-api-access-szc6s\") pod \"isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j\" (UID: \"9faee4ca-523f-4a0c-80a4-1373ba260eaf\") " pod="kserve-ci-e2e-test/isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j"
Jul 02 13:02:00.082146 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:00.082105    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j"
Jul 02 13:02:00.217109 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:00.217089    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j"]
Jul 02 13:02:00.219332 ip-10-0-143-88 kubenswrapper[2576]: W0702 13:02:00.219309    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9faee4ca_523f_4a0c_80a4_1373ba260eaf.slice/crio-91e3131a918199ea89dfdb792614a8abb895aaa8c6eb5089389c2588f3fce67e WatchSource:0}: Error finding container 91e3131a918199ea89dfdb792614a8abb895aaa8c6eb5089389c2588f3fce67e: Status 404 returned error can't find the container with id 91e3131a918199ea89dfdb792614a8abb895aaa8c6eb5089389c2588f3fce67e
Jul 02 13:02:00.723864 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:00.723825    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j" event={"ID":"9faee4ca-523f-4a0c-80a4-1373ba260eaf","Type":"ContainerStarted","Data":"9bcb7ace2cfb233775f09101ec56aab70af9384096fc2b6ecdc9058fb00ee728"}
Jul 02 13:02:00.724049 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:00.723890    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j" event={"ID":"9faee4ca-523f-4a0c-80a4-1373ba260eaf","Type":"ContainerStarted","Data":"91e3131a918199ea89dfdb792614a8abb895aaa8c6eb5089389c2588f3fce67e"}
Jul 02 13:02:00.864975 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:00.864938    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80847d7c-5d90-449c-816c-70cdb1548320" path="/var/lib/kubelet/pods/80847d7c-5d90-449c-816c-70cdb1548320/volumes"
Jul 02 13:02:03.740855 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:03.740827    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j_9faee4ca-523f-4a0c-80a4-1373ba260eaf/storage-initializer/0.log"
Jul 02 13:02:03.741261 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:03.740866    2576 generic.go:356] "Generic (PLEG): container finished" podID="9faee4ca-523f-4a0c-80a4-1373ba260eaf" containerID="9bcb7ace2cfb233775f09101ec56aab70af9384096fc2b6ecdc9058fb00ee728" exitCode=1
Jul 02 13:02:03.741261 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:03.740949    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j" event={"ID":"9faee4ca-523f-4a0c-80a4-1373ba260eaf","Type":"ContainerDied","Data":"9bcb7ace2cfb233775f09101ec56aab70af9384096fc2b6ecdc9058fb00ee728"}
Jul 02 13:02:04.748200 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:04.748170    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j_9faee4ca-523f-4a0c-80a4-1373ba260eaf/storage-initializer/0.log"
Jul 02 13:02:04.748549 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:04.748259    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j" event={"ID":"9faee4ca-523f-4a0c-80a4-1373ba260eaf","Type":"ContainerStarted","Data":"161c08d1866124750e04728a2916a491b5bd105e3c7eba37def78cead380b453"}
Jul 02 13:02:04.748660 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:04.748643    2576 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="kserve-ci-e2e-test/isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j" secret="" err="secret \"fail-s3-sa-f3d602-dockercfg-nkg5p\" not found"
Jul 02 13:02:04.798466 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:04.798432    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j"]
Jul 02 13:02:04.854030 ip-10-0-143-88 kubenswrapper[2576]: E0702 13:02:04.853997    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-f3d602-predictor-serving-cert: secret "isvc-init-fail-f3d602-predictor-serving-cert" not found
Jul 02 13:02:04.854188 ip-10-0-143-88 kubenswrapper[2576]: E0702 13:02:04.854095    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9faee4ca-523f-4a0c-80a4-1373ba260eaf-proxy-tls podName:9faee4ca-523f-4a0c-80a4-1373ba260eaf nodeName:}" failed. No retries permitted until 2026-07-02 13:02:05.354071447 +0000 UTC m=+1413.961534462 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9faee4ca-523f-4a0c-80a4-1373ba260eaf-proxy-tls") pod "isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j" (UID: "9faee4ca-523f-4a0c-80a4-1373ba260eaf") : secret "isvc-init-fail-f3d602-predictor-serving-cert" not found
Jul 02 13:02:05.358263 ip-10-0-143-88 kubenswrapper[2576]: E0702 13:02:05.358229    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-f3d602-predictor-serving-cert: secret "isvc-init-fail-f3d602-predictor-serving-cert" not found
Jul 02 13:02:05.358423 ip-10-0-143-88 kubenswrapper[2576]: E0702 13:02:05.358317    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9faee4ca-523f-4a0c-80a4-1373ba260eaf-proxy-tls podName:9faee4ca-523f-4a0c-80a4-1373ba260eaf nodeName:}" failed. No retries permitted until 2026-07-02 13:02:06.358296106 +0000 UTC m=+1414.965759125 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9faee4ca-523f-4a0c-80a4-1373ba260eaf-proxy-tls") pod "isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j" (UID: "9faee4ca-523f-4a0c-80a4-1373ba260eaf") : secret "isvc-init-fail-f3d602-predictor-serving-cert" not found
Jul 02 13:02:05.753457 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:05.753417    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j" podUID="9faee4ca-523f-4a0c-80a4-1373ba260eaf" containerName="storage-initializer" containerID="cri-o://161c08d1866124750e04728a2916a491b5bd105e3c7eba37def78cead380b453" gracePeriod=30
Jul 02 13:02:06.369426 ip-10-0-143-88 kubenswrapper[2576]: E0702 13:02:06.369392    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-f3d602-predictor-serving-cert: secret "isvc-init-fail-f3d602-predictor-serving-cert" not found
Jul 02 13:02:06.369608 ip-10-0-143-88 kubenswrapper[2576]: E0702 13:02:06.369457    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9faee4ca-523f-4a0c-80a4-1373ba260eaf-proxy-tls podName:9faee4ca-523f-4a0c-80a4-1373ba260eaf nodeName:}" failed. No retries permitted until 2026-07-02 13:02:08.369443966 +0000 UTC m=+1416.976906967 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9faee4ca-523f-4a0c-80a4-1373ba260eaf-proxy-tls") pod "isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j" (UID: "9faee4ca-523f-4a0c-80a4-1373ba260eaf") : secret "isvc-init-fail-f3d602-predictor-serving-cert" not found
Jul 02 13:02:07.898074 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:07.898046    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j_9faee4ca-523f-4a0c-80a4-1373ba260eaf/storage-initializer/1.log"
Jul 02 13:02:07.898441 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:07.898427    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j_9faee4ca-523f-4a0c-80a4-1373ba260eaf/storage-initializer/0.log"
Jul 02 13:02:07.900735 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:07.900718    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j"
Jul 02 13:02:07.983185 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:07.983118    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9faee4ca-523f-4a0c-80a4-1373ba260eaf-proxy-tls\") pod \"9faee4ca-523f-4a0c-80a4-1373ba260eaf\" (UID: \"9faee4ca-523f-4a0c-80a4-1373ba260eaf\") "
Jul 02 13:02:07.983185 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:07.983152    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9faee4ca-523f-4a0c-80a4-1373ba260eaf-kserve-provision-location\") pod \"9faee4ca-523f-4a0c-80a4-1373ba260eaf\" (UID: \"9faee4ca-523f-4a0c-80a4-1373ba260eaf\") "
Jul 02 13:02:07.983185 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:07.983176    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9faee4ca-523f-4a0c-80a4-1373ba260eaf-cabundle-cert\") pod \"9faee4ca-523f-4a0c-80a4-1373ba260eaf\" (UID: \"9faee4ca-523f-4a0c-80a4-1373ba260eaf\") "
Jul 02 13:02:07.983440 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:07.983212    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szc6s\" (UniqueName: \"kubernetes.io/projected/9faee4ca-523f-4a0c-80a4-1373ba260eaf-kube-api-access-szc6s\") pod \"9faee4ca-523f-4a0c-80a4-1373ba260eaf\" (UID: \"9faee4ca-523f-4a0c-80a4-1373ba260eaf\") "
Jul 02 13:02:07.983440 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:07.983241    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9faee4ca-523f-4a0c-80a4-1373ba260eaf-kube-rbac-proxy-sar-config\") pod \"9faee4ca-523f-4a0c-80a4-1373ba260eaf\" (UID: \"9faee4ca-523f-4a0c-80a4-1373ba260eaf\") "
Jul 02 13:02:07.983604 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:07.983442    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9faee4ca-523f-4a0c-80a4-1373ba260eaf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9faee4ca-523f-4a0c-80a4-1373ba260eaf" (UID: "9faee4ca-523f-4a0c-80a4-1373ba260eaf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:02:07.983661 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:07.983629    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9faee4ca-523f-4a0c-80a4-1373ba260eaf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9faee4ca-523f-4a0c-80a4-1373ba260eaf" (UID: "9faee4ca-523f-4a0c-80a4-1373ba260eaf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:02:07.983766 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:07.983715    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9faee4ca-523f-4a0c-80a4-1373ba260eaf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9faee4ca-523f-4a0c-80a4-1373ba260eaf" (UID: "9faee4ca-523f-4a0c-80a4-1373ba260eaf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:02:07.983858 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:07.983800    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9faee4ca-523f-4a0c-80a4-1373ba260eaf-kserve-provision-location\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 13:02:07.983858 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:07.983823    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9faee4ca-523f-4a0c-80a4-1373ba260eaf-cabundle-cert\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 13:02:07.983858 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:07.983838    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9faee4ca-523f-4a0c-80a4-1373ba260eaf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 13:02:07.985466 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:07.985445    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9faee4ca-523f-4a0c-80a4-1373ba260eaf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9faee4ca-523f-4a0c-80a4-1373ba260eaf" (UID: "9faee4ca-523f-4a0c-80a4-1373ba260eaf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:02:07.985547 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:07.985471    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9faee4ca-523f-4a0c-80a4-1373ba260eaf-kube-api-access-szc6s" (OuterVolumeSpecName: "kube-api-access-szc6s") pod "9faee4ca-523f-4a0c-80a4-1373ba260eaf" (UID: "9faee4ca-523f-4a0c-80a4-1373ba260eaf"). InnerVolumeSpecName "kube-api-access-szc6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:02:08.084475 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:08.084447    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9faee4ca-523f-4a0c-80a4-1373ba260eaf-proxy-tls\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 13:02:08.084475 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:08.084472    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szc6s\" (UniqueName: \"kubernetes.io/projected/9faee4ca-523f-4a0c-80a4-1373ba260eaf-kube-api-access-szc6s\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 13:02:08.767829 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:08.767803    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j_9faee4ca-523f-4a0c-80a4-1373ba260eaf/storage-initializer/1.log"
Jul 02 13:02:08.768232 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:08.768216    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j_9faee4ca-523f-4a0c-80a4-1373ba260eaf/storage-initializer/0.log"
Jul 02 13:02:08.768293 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:08.768252    2576 generic.go:356] "Generic (PLEG): container finished" podID="9faee4ca-523f-4a0c-80a4-1373ba260eaf" containerID="161c08d1866124750e04728a2916a491b5bd105e3c7eba37def78cead380b453" exitCode=1
Jul 02 13:02:08.768330 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:08.768307    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j"
Jul 02 13:02:08.768364 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:08.768335    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j" event={"ID":"9faee4ca-523f-4a0c-80a4-1373ba260eaf","Type":"ContainerDied","Data":"161c08d1866124750e04728a2916a491b5bd105e3c7eba37def78cead380b453"}
Jul 02 13:02:08.768407 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:08.768396    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j" event={"ID":"9faee4ca-523f-4a0c-80a4-1373ba260eaf","Type":"ContainerDied","Data":"91e3131a918199ea89dfdb792614a8abb895aaa8c6eb5089389c2588f3fce67e"}
Jul 02 13:02:08.768442 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:08.768426    2576 scope.go:117] "RemoveContainer" containerID="161c08d1866124750e04728a2916a491b5bd105e3c7eba37def78cead380b453"
Jul 02 13:02:08.777181 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:08.777164    2576 scope.go:117] "RemoveContainer" containerID="9bcb7ace2cfb233775f09101ec56aab70af9384096fc2b6ecdc9058fb00ee728"
Jul 02 13:02:08.784835 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:08.784821    2576 scope.go:117] "RemoveContainer" containerID="161c08d1866124750e04728a2916a491b5bd105e3c7eba37def78cead380b453"
Jul 02 13:02:08.785108 ip-10-0-143-88 kubenswrapper[2576]: E0702 13:02:08.785090    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"161c08d1866124750e04728a2916a491b5bd105e3c7eba37def78cead380b453\": container with ID starting with 161c08d1866124750e04728a2916a491b5bd105e3c7eba37def78cead380b453 not found: ID does not exist" containerID="161c08d1866124750e04728a2916a491b5bd105e3c7eba37def78cead380b453"
Jul 02 13:02:08.785179 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:08.785115    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"161c08d1866124750e04728a2916a491b5bd105e3c7eba37def78cead380b453"} err="failed to get container status \"161c08d1866124750e04728a2916a491b5bd105e3c7eba37def78cead380b453\": rpc error: code = NotFound desc = could not find container \"161c08d1866124750e04728a2916a491b5bd105e3c7eba37def78cead380b453\": container with ID starting with 161c08d1866124750e04728a2916a491b5bd105e3c7eba37def78cead380b453 not found: ID does not exist"
Jul 02 13:02:08.785179 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:08.785131    2576 scope.go:117] "RemoveContainer" containerID="9bcb7ace2cfb233775f09101ec56aab70af9384096fc2b6ecdc9058fb00ee728"
Jul 02 13:02:08.785359 ip-10-0-143-88 kubenswrapper[2576]: E0702 13:02:08.785342    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bcb7ace2cfb233775f09101ec56aab70af9384096fc2b6ecdc9058fb00ee728\": container with ID starting with 9bcb7ace2cfb233775f09101ec56aab70af9384096fc2b6ecdc9058fb00ee728 not found: ID does not exist" containerID="9bcb7ace2cfb233775f09101ec56aab70af9384096fc2b6ecdc9058fb00ee728"
Jul 02 13:02:08.785397 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:08.785364    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bcb7ace2cfb233775f09101ec56aab70af9384096fc2b6ecdc9058fb00ee728"} err="failed to get container status \"9bcb7ace2cfb233775f09101ec56aab70af9384096fc2b6ecdc9058fb00ee728\": rpc error: code = NotFound desc = could not find container \"9bcb7ace2cfb233775f09101ec56aab70af9384096fc2b6ecdc9058fb00ee728\": container with ID starting with 9bcb7ace2cfb233775f09101ec56aab70af9384096fc2b6ecdc9058fb00ee728 not found: ID does not exist"
Jul 02 13:02:08.810907 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:08.810878    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j"]
Jul 02 13:02:08.816002 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:08.815980    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-f3d602-predictor-7bbdc69c6d-v895j"]
Jul 02 13:02:08.862471 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:08.862448    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9faee4ca-523f-4a0c-80a4-1373ba260eaf" path="/var/lib/kubelet/pods/9faee4ca-523f-4a0c-80a4-1373ba260eaf/volumes"
Jul 02 13:02:09.988341 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:09.988311    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q"]
Jul 02 13:02:09.988691 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:09.988678    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9faee4ca-523f-4a0c-80a4-1373ba260eaf" containerName="storage-initializer"
Jul 02 13:02:09.988691 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:09.988691    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9faee4ca-523f-4a0c-80a4-1373ba260eaf" containerName="storage-initializer"
Jul 02 13:02:09.988776 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:09.988699    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9faee4ca-523f-4a0c-80a4-1373ba260eaf" containerName="storage-initializer"
Jul 02 13:02:09.988776 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:09.988705    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9faee4ca-523f-4a0c-80a4-1373ba260eaf" containerName="storage-initializer"
Jul 02 13:02:09.988852 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:09.988780    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9faee4ca-523f-4a0c-80a4-1373ba260eaf" containerName="storage-initializer"
Jul 02 13:02:09.988852 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:09.988790    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9faee4ca-523f-4a0c-80a4-1373ba260eaf" containerName="storage-initializer"
Jul 02 13:02:09.993733 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:09.993713    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q"
Jul 02 13:02:09.996902 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:09.996872    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-abd6e-predictor-serving-cert\""
Jul 02 13:02:09.997114 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:09.997098    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-abd6e-kube-rbac-proxy-sar-config\""
Jul 02 13:02:09.997172 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:09.997101    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gnkzk\""
Jul 02 13:02:09.997998 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:09.997984    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 13:02:09.997998 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:09.997993    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 13:02:09.998149 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:09.998005    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 13:02:09.998149 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:09.998042    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 13:02:10.005130 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:10.005109    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q"]
Jul 02 13:02:10.101336 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:10.101310    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b240a6aa-163e-41b8-9423-e4e5cd8ed447-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q\" (UID: \"b240a6aa-163e-41b8-9423-e4e5cd8ed447\") " pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q"
Jul 02 13:02:10.101471 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:10.101346    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n59sj\" (UniqueName: \"kubernetes.io/projected/b240a6aa-163e-41b8-9423-e4e5cd8ed447-kube-api-access-n59sj\") pod \"raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q\" (UID: \"b240a6aa-163e-41b8-9423-e4e5cd8ed447\") " pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q"
Jul 02 13:02:10.101471 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:10.101367    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b240a6aa-163e-41b8-9423-e4e5cd8ed447-kserve-provision-location\") pod \"raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q\" (UID: \"b240a6aa-163e-41b8-9423-e4e5cd8ed447\") " pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q"
Jul 02 13:02:10.101471 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:10.101424    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b240a6aa-163e-41b8-9423-e4e5cd8ed447-cabundle-cert\") pod \"raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q\" (UID: \"b240a6aa-163e-41b8-9423-e4e5cd8ed447\") " pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q"
Jul 02 13:02:10.101471 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:10.101450    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b240a6aa-163e-41b8-9423-e4e5cd8ed447-proxy-tls\") pod \"raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q\" (UID: \"b240a6aa-163e-41b8-9423-e4e5cd8ed447\") " pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q"
Jul 02 13:02:10.202427 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:10.202345    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b240a6aa-163e-41b8-9423-e4e5cd8ed447-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q\" (UID: \"b240a6aa-163e-41b8-9423-e4e5cd8ed447\") " pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q"
Jul 02 13:02:10.202427 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:10.202392    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n59sj\" (UniqueName: \"kubernetes.io/projected/b240a6aa-163e-41b8-9423-e4e5cd8ed447-kube-api-access-n59sj\") pod \"raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q\" (UID: \"b240a6aa-163e-41b8-9423-e4e5cd8ed447\") " pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q"
Jul 02 13:02:10.202427 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:10.202413    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b240a6aa-163e-41b8-9423-e4e5cd8ed447-kserve-provision-location\") pod \"raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q\" (UID: \"b240a6aa-163e-41b8-9423-e4e5cd8ed447\") " pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q"
Jul 02 13:02:10.202694 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:10.202443    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b240a6aa-163e-41b8-9423-e4e5cd8ed447-cabundle-cert\") pod \"raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q\" (UID: \"b240a6aa-163e-41b8-9423-e4e5cd8ed447\") " pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q"
Jul 02 13:02:10.202694 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:10.202469    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b240a6aa-163e-41b8-9423-e4e5cd8ed447-proxy-tls\") pod \"raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q\" (UID: \"b240a6aa-163e-41b8-9423-e4e5cd8ed447\") " pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q"
Jul 02 13:02:10.202694 ip-10-0-143-88 kubenswrapper[2576]: E0702 13:02:10.202569    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-serving-cert: secret "raw-sklearn-abd6e-predictor-serving-cert" not found
Jul 02 13:02:10.202694 ip-10-0-143-88 kubenswrapper[2576]: E0702 13:02:10.202633    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b240a6aa-163e-41b8-9423-e4e5cd8ed447-proxy-tls podName:b240a6aa-163e-41b8-9423-e4e5cd8ed447 nodeName:}" failed. No retries permitted until 2026-07-02 13:02:10.702617172 +0000 UTC m=+1419.310080173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b240a6aa-163e-41b8-9423-e4e5cd8ed447-proxy-tls") pod "raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q" (UID: "b240a6aa-163e-41b8-9423-e4e5cd8ed447") : secret "raw-sklearn-abd6e-predictor-serving-cert" not found
Jul 02 13:02:10.202878 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:10.202855    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b240a6aa-163e-41b8-9423-e4e5cd8ed447-kserve-provision-location\") pod \"raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q\" (UID: \"b240a6aa-163e-41b8-9423-e4e5cd8ed447\") " pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q"
Jul 02 13:02:10.203224 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:10.203202    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b240a6aa-163e-41b8-9423-e4e5cd8ed447-cabundle-cert\") pod \"raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q\" (UID: \"b240a6aa-163e-41b8-9423-e4e5cd8ed447\") " pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q"
Jul 02 13:02:10.203275 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:10.203208    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b240a6aa-163e-41b8-9423-e4e5cd8ed447-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q\" (UID: \"b240a6aa-163e-41b8-9423-e4e5cd8ed447\") " pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q"
Jul 02 13:02:10.211235 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:10.211214    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n59sj\" (UniqueName: \"kubernetes.io/projected/b240a6aa-163e-41b8-9423-e4e5cd8ed447-kube-api-access-n59sj\") pod \"raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q\" (UID: \"b240a6aa-163e-41b8-9423-e4e5cd8ed447\") " pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q"
Jul 02 13:02:10.707080 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:10.707048    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b240a6aa-163e-41b8-9423-e4e5cd8ed447-proxy-tls\") pod \"raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q\" (UID: \"b240a6aa-163e-41b8-9423-e4e5cd8ed447\") " pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q"
Jul 02 13:02:10.709464 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:10.709435    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b240a6aa-163e-41b8-9423-e4e5cd8ed447-proxy-tls\") pod \"raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q\" (UID: \"b240a6aa-163e-41b8-9423-e4e5cd8ed447\") " pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q"
Jul 02 13:02:10.904702 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:10.904667    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q"
Jul 02 13:02:11.037543 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:11.037519    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q"]
Jul 02 13:02:11.039557 ip-10-0-143-88 kubenswrapper[2576]: W0702 13:02:11.039531    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb240a6aa_163e_41b8_9423_e4e5cd8ed447.slice/crio-dfff26c83d95b583b18b6cd8bf640132b558c52feeb74329ebb57dd292a052bd WatchSource:0}: Error finding container dfff26c83d95b583b18b6cd8bf640132b558c52feeb74329ebb57dd292a052bd: Status 404 returned error can't find the container with id dfff26c83d95b583b18b6cd8bf640132b558c52feeb74329ebb57dd292a052bd
Jul 02 13:02:11.785616 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:11.785582    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q" event={"ID":"b240a6aa-163e-41b8-9423-e4e5cd8ed447","Type":"ContainerStarted","Data":"ab7e67f6ade94503144ff8b1b6180864545c8d36d7227f86e0b53ca92122a37e"}
Jul 02 13:02:11.785789 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:11.785639    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q" event={"ID":"b240a6aa-163e-41b8-9423-e4e5cd8ed447","Type":"ContainerStarted","Data":"dfff26c83d95b583b18b6cd8bf640132b558c52feeb74329ebb57dd292a052bd"}
Jul 02 13:02:14.801661 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:14.801627    2576 generic.go:356] "Generic (PLEG): container finished" podID="b240a6aa-163e-41b8-9423-e4e5cd8ed447" containerID="ab7e67f6ade94503144ff8b1b6180864545c8d36d7227f86e0b53ca92122a37e" exitCode=0
Jul 02 13:02:14.802045 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:14.801672    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q" event={"ID":"b240a6aa-163e-41b8-9423-e4e5cd8ed447","Type":"ContainerDied","Data":"ab7e67f6ade94503144ff8b1b6180864545c8d36d7227f86e0b53ca92122a37e"}
Jul 02 13:02:15.808121 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:15.808084    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q" event={"ID":"b240a6aa-163e-41b8-9423-e4e5cd8ed447","Type":"ContainerStarted","Data":"0b55853380889084e21fb18fae5a5790a1ea167f42ffa5fcf063ae61c0ed06ae"}
Jul 02 13:02:15.808541 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:15.808147    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q" event={"ID":"b240a6aa-163e-41b8-9423-e4e5cd8ed447","Type":"ContainerStarted","Data":"5b625c5d14c1b4f6aded775262f3ecfc863e87ad94b50b75e28e378370bc8d88"}
Jul 02 13:02:15.808541 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:15.808428    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q"
Jul 02 13:02:15.808541 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:15.808473    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q"
Jul 02 13:02:15.809994 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:15.809969    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q" podUID="b240a6aa-163e-41b8-9423-e4e5cd8ed447" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 02 13:02:15.829166 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:15.829122    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q" podStartSLOduration=3.301165265 podStartE2EDuration="6.829108099s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.527942834s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:02:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:02:15.827365778 +0000 UTC m=+1424.434828826" watchObservedRunningTime="2026-07-02 13:02:15.829108099 +0000 UTC m=+1424.436571123"
Jul 02 13:02:16.814821 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:16.814778    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q" podUID="b240a6aa-163e-41b8-9423-e4e5cd8ed447" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 02 13:02:21.820450 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:21.820423    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q"
Jul 02 13:02:21.820965 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:21.820938    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q" podUID="b240a6aa-163e-41b8-9423-e4e5cd8ed447" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 02 13:02:31.821296 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:31.821205    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q" podUID="b240a6aa-163e-41b8-9423-e4e5cd8ed447" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 02 13:02:41.821319 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:41.821274    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q" podUID="b240a6aa-163e-41b8-9423-e4e5cd8ed447" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 02 13:02:51.821893 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:02:51.821850    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q" podUID="b240a6aa-163e-41b8-9423-e4e5cd8ed447" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 02 13:03:01.821036 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:01.820995    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q" podUID="b240a6aa-163e-41b8-9423-e4e5cd8ed447" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 02 13:03:11.821640 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:11.821599    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q" podUID="b240a6aa-163e-41b8-9423-e4e5cd8ed447" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 02 13:03:21.821621 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:21.821584    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q" podUID="b240a6aa-163e-41b8-9423-e4e5cd8ed447" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 02 13:03:31.822339 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:31.822307    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q"
Jul 02 13:03:32.309488 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:32.309464    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2c7ks_116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a/ovn-acl-logging/0.log"
Jul 02 13:03:32.929833 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:32.929807    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2c7ks_116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a/ovn-acl-logging/0.log"
Jul 02 13:03:40.173686 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:40.173654    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q"]
Jul 02 13:03:40.174163 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:40.173969    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q" podUID="b240a6aa-163e-41b8-9423-e4e5cd8ed447" containerName="kserve-container" containerID="cri-o://5b625c5d14c1b4f6aded775262f3ecfc863e87ad94b50b75e28e378370bc8d88" gracePeriod=30
Jul 02 13:03:40.174163 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:40.174040    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q" podUID="b240a6aa-163e-41b8-9423-e4e5cd8ed447" containerName="kube-rbac-proxy" containerID="cri-o://0b55853380889084e21fb18fae5a5790a1ea167f42ffa5fcf063ae61c0ed06ae" gracePeriod=30
Jul 02 13:03:41.225628 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:41.225587    2576 generic.go:356] "Generic (PLEG): container finished" podID="b240a6aa-163e-41b8-9423-e4e5cd8ed447" containerID="0b55853380889084e21fb18fae5a5790a1ea167f42ffa5fcf063ae61c0ed06ae" exitCode=2
Jul 02 13:03:41.226032 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:41.225660    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q" event={"ID":"b240a6aa-163e-41b8-9423-e4e5cd8ed447","Type":"ContainerDied","Data":"0b55853380889084e21fb18fae5a5790a1ea167f42ffa5fcf063ae61c0ed06ae"}
Jul 02 13:03:41.815519 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:41.815473    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q" podUID="b240a6aa-163e-41b8-9423-e4e5cd8ed447" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Jul 02 13:03:41.821898 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:41.821867    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q" podUID="b240a6aa-163e-41b8-9423-e4e5cd8ed447" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 02 13:03:44.830977 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:44.830954    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q"
Jul 02 13:03:44.927886 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:44.927858    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b240a6aa-163e-41b8-9423-e4e5cd8ed447-cabundle-cert\") pod \"b240a6aa-163e-41b8-9423-e4e5cd8ed447\" (UID: \"b240a6aa-163e-41b8-9423-e4e5cd8ed447\") "
Jul 02 13:03:44.928088 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:44.927902    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b240a6aa-163e-41b8-9423-e4e5cd8ed447-kserve-provision-location\") pod \"b240a6aa-163e-41b8-9423-e4e5cd8ed447\" (UID: \"b240a6aa-163e-41b8-9423-e4e5cd8ed447\") "
Jul 02 13:03:44.928088 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:44.927948    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b240a6aa-163e-41b8-9423-e4e5cd8ed447-proxy-tls\") pod \"b240a6aa-163e-41b8-9423-e4e5cd8ed447\" (UID: \"b240a6aa-163e-41b8-9423-e4e5cd8ed447\") "
Jul 02 13:03:44.928088 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:44.928051    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n59sj\" (UniqueName: \"kubernetes.io/projected/b240a6aa-163e-41b8-9423-e4e5cd8ed447-kube-api-access-n59sj\") pod \"b240a6aa-163e-41b8-9423-e4e5cd8ed447\" (UID: \"b240a6aa-163e-41b8-9423-e4e5cd8ed447\") "
Jul 02 13:03:44.928279 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:44.928103    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b240a6aa-163e-41b8-9423-e4e5cd8ed447-kube-rbac-proxy-sar-config\") pod \"b240a6aa-163e-41b8-9423-e4e5cd8ed447\" (UID: \"b240a6aa-163e-41b8-9423-e4e5cd8ed447\") "
Jul 02 13:03:44.928279 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:44.928238    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b240a6aa-163e-41b8-9423-e4e5cd8ed447-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b240a6aa-163e-41b8-9423-e4e5cd8ed447" (UID: "b240a6aa-163e-41b8-9423-e4e5cd8ed447"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:03:44.928387 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:44.928285    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b240a6aa-163e-41b8-9423-e4e5cd8ed447-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b240a6aa-163e-41b8-9423-e4e5cd8ed447" (UID: "b240a6aa-163e-41b8-9423-e4e5cd8ed447"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:03:44.928456 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:44.928405    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b240a6aa-163e-41b8-9423-e4e5cd8ed447-cabundle-cert\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 13:03:44.928456 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:44.928422    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b240a6aa-163e-41b8-9423-e4e5cd8ed447-kserve-provision-location\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 13:03:44.928538 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:44.928501    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b240a6aa-163e-41b8-9423-e4e5cd8ed447-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b240a6aa-163e-41b8-9423-e4e5cd8ed447" (UID: "b240a6aa-163e-41b8-9423-e4e5cd8ed447"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:03:44.930187 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:44.930166    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b240a6aa-163e-41b8-9423-e4e5cd8ed447-kube-api-access-n59sj" (OuterVolumeSpecName: "kube-api-access-n59sj") pod "b240a6aa-163e-41b8-9423-e4e5cd8ed447" (UID: "b240a6aa-163e-41b8-9423-e4e5cd8ed447"). InnerVolumeSpecName "kube-api-access-n59sj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:03:44.930255 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:44.930188    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b240a6aa-163e-41b8-9423-e4e5cd8ed447-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b240a6aa-163e-41b8-9423-e4e5cd8ed447" (UID: "b240a6aa-163e-41b8-9423-e4e5cd8ed447"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:03:45.029591 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.029557    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n59sj\" (UniqueName: \"kubernetes.io/projected/b240a6aa-163e-41b8-9423-e4e5cd8ed447-kube-api-access-n59sj\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 13:03:45.029591 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.029586    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b240a6aa-163e-41b8-9423-e4e5cd8ed447-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 13:03:45.029591 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.029595    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b240a6aa-163e-41b8-9423-e4e5cd8ed447-proxy-tls\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 13:03:45.247806 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.247727    2576 generic.go:356] "Generic (PLEG): container finished" podID="b240a6aa-163e-41b8-9423-e4e5cd8ed447" containerID="5b625c5d14c1b4f6aded775262f3ecfc863e87ad94b50b75e28e378370bc8d88" exitCode=0
Jul 02 13:03:45.247806 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.247781    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q"
Jul 02 13:03:45.247806 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.247799    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q" event={"ID":"b240a6aa-163e-41b8-9423-e4e5cd8ed447","Type":"ContainerDied","Data":"5b625c5d14c1b4f6aded775262f3ecfc863e87ad94b50b75e28e378370bc8d88"}
Jul 02 13:03:45.248070 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.247849    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q" event={"ID":"b240a6aa-163e-41b8-9423-e4e5cd8ed447","Type":"ContainerDied","Data":"dfff26c83d95b583b18b6cd8bf640132b558c52feeb74329ebb57dd292a052bd"}
Jul 02 13:03:45.248070 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.247885    2576 scope.go:117] "RemoveContainer" containerID="0b55853380889084e21fb18fae5a5790a1ea167f42ffa5fcf063ae61c0ed06ae"
Jul 02 13:03:45.257094 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.257076    2576 scope.go:117] "RemoveContainer" containerID="5b625c5d14c1b4f6aded775262f3ecfc863e87ad94b50b75e28e378370bc8d88"
Jul 02 13:03:45.264474 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.264459    2576 scope.go:117] "RemoveContainer" containerID="ab7e67f6ade94503144ff8b1b6180864545c8d36d7227f86e0b53ca92122a37e"
Jul 02 13:03:45.271828 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.271810    2576 scope.go:117] "RemoveContainer" containerID="0b55853380889084e21fb18fae5a5790a1ea167f42ffa5fcf063ae61c0ed06ae"
Jul 02 13:03:45.272086 ip-10-0-143-88 kubenswrapper[2576]: E0702 13:03:45.272070    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b55853380889084e21fb18fae5a5790a1ea167f42ffa5fcf063ae61c0ed06ae\": container with ID starting with 0b55853380889084e21fb18fae5a5790a1ea167f42ffa5fcf063ae61c0ed06ae not found: ID does not exist" containerID="0b55853380889084e21fb18fae5a5790a1ea167f42ffa5fcf063ae61c0ed06ae"
Jul 02 13:03:45.272138 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.272094    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b55853380889084e21fb18fae5a5790a1ea167f42ffa5fcf063ae61c0ed06ae"} err="failed to get container status \"0b55853380889084e21fb18fae5a5790a1ea167f42ffa5fcf063ae61c0ed06ae\": rpc error: code = NotFound desc = could not find container \"0b55853380889084e21fb18fae5a5790a1ea167f42ffa5fcf063ae61c0ed06ae\": container with ID starting with 0b55853380889084e21fb18fae5a5790a1ea167f42ffa5fcf063ae61c0ed06ae not found: ID does not exist"
Jul 02 13:03:45.272138 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.272109    2576 scope.go:117] "RemoveContainer" containerID="5b625c5d14c1b4f6aded775262f3ecfc863e87ad94b50b75e28e378370bc8d88"
Jul 02 13:03:45.272374 ip-10-0-143-88 kubenswrapper[2576]: E0702 13:03:45.272356    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b625c5d14c1b4f6aded775262f3ecfc863e87ad94b50b75e28e378370bc8d88\": container with ID starting with 5b625c5d14c1b4f6aded775262f3ecfc863e87ad94b50b75e28e378370bc8d88 not found: ID does not exist" containerID="5b625c5d14c1b4f6aded775262f3ecfc863e87ad94b50b75e28e378370bc8d88"
Jul 02 13:03:45.272430 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.272381    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b625c5d14c1b4f6aded775262f3ecfc863e87ad94b50b75e28e378370bc8d88"} err="failed to get container status \"5b625c5d14c1b4f6aded775262f3ecfc863e87ad94b50b75e28e378370bc8d88\": rpc error: code = NotFound desc = could not find container \"5b625c5d14c1b4f6aded775262f3ecfc863e87ad94b50b75e28e378370bc8d88\": container with ID starting with 5b625c5d14c1b4f6aded775262f3ecfc863e87ad94b50b75e28e378370bc8d88 not found: ID does not exist"
Jul 02 13:03:45.272430 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.272397    2576 scope.go:117] "RemoveContainer" containerID="ab7e67f6ade94503144ff8b1b6180864545c8d36d7227f86e0b53ca92122a37e"
Jul 02 13:03:45.272585 ip-10-0-143-88 kubenswrapper[2576]: E0702 13:03:45.272568    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab7e67f6ade94503144ff8b1b6180864545c8d36d7227f86e0b53ca92122a37e\": container with ID starting with ab7e67f6ade94503144ff8b1b6180864545c8d36d7227f86e0b53ca92122a37e not found: ID does not exist" containerID="ab7e67f6ade94503144ff8b1b6180864545c8d36d7227f86e0b53ca92122a37e"
Jul 02 13:03:45.272624 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.272589    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab7e67f6ade94503144ff8b1b6180864545c8d36d7227f86e0b53ca92122a37e"} err="failed to get container status \"ab7e67f6ade94503144ff8b1b6180864545c8d36d7227f86e0b53ca92122a37e\": rpc error: code = NotFound desc = could not find container \"ab7e67f6ade94503144ff8b1b6180864545c8d36d7227f86e0b53ca92122a37e\": container with ID starting with ab7e67f6ade94503144ff8b1b6180864545c8d36d7227f86e0b53ca92122a37e not found: ID does not exist"
Jul 02 13:03:45.299427 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.299399    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q"]
Jul 02 13:03:45.312657 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.312635    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-abd6e-predictor-568bb8fbf8-d7n2q"]
Jul 02 13:03:45.387152 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.387121    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4"]
Jul 02 13:03:45.387495 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.387479    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b240a6aa-163e-41b8-9423-e4e5cd8ed447" containerName="storage-initializer"
Jul 02 13:03:45.387495 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.387494    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b240a6aa-163e-41b8-9423-e4e5cd8ed447" containerName="storage-initializer"
Jul 02 13:03:45.387495 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.387503    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b240a6aa-163e-41b8-9423-e4e5cd8ed447" containerName="kserve-container"
Jul 02 13:03:45.387495 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.387509    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b240a6aa-163e-41b8-9423-e4e5cd8ed447" containerName="kserve-container"
Jul 02 13:03:45.387668 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.387515    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b240a6aa-163e-41b8-9423-e4e5cd8ed447" containerName="kube-rbac-proxy"
Jul 02 13:03:45.387668 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.387521    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b240a6aa-163e-41b8-9423-e4e5cd8ed447" containerName="kube-rbac-proxy"
Jul 02 13:03:45.387668 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.387582    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b240a6aa-163e-41b8-9423-e4e5cd8ed447" containerName="kserve-container"
Jul 02 13:03:45.387668 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.387590    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b240a6aa-163e-41b8-9423-e4e5cd8ed447" containerName="kube-rbac-proxy"
Jul 02 13:03:45.392279 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.392259    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4"
Jul 02 13:03:45.395281 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.395256    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 13:03:45.395511 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.395492    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 13:03:45.395621 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.395515    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-4a543-predictor-serving-cert\""
Jul 02 13:03:45.395621 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.395545    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 13:03:45.395621 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.395498    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 13:03:45.395621 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.395616    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-4a543-kube-rbac-proxy-sar-config\""
Jul 02 13:03:45.395825 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.395751    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gnkzk\""
Jul 02 13:03:45.402500 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.402480    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4"]
Jul 02 13:03:45.432763 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.432734    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5b25188-d9d1-4261-b781-d93cbfcb2dc7-proxy-tls\") pod \"raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4\" (UID: \"d5b25188-d9d1-4261-b781-d93cbfcb2dc7\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4"
Jul 02 13:03:45.432951 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.432784    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5b25188-d9d1-4261-b781-d93cbfcb2dc7-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4\" (UID: \"d5b25188-d9d1-4261-b781-d93cbfcb2dc7\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4"
Jul 02 13:03:45.432951 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.432856    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgkpn\" (UniqueName: \"kubernetes.io/projected/d5b25188-d9d1-4261-b781-d93cbfcb2dc7-kube-api-access-sgkpn\") pod \"raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4\" (UID: \"d5b25188-d9d1-4261-b781-d93cbfcb2dc7\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4"
Jul 02 13:03:45.432951 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.432891    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5b25188-d9d1-4261-b781-d93cbfcb2dc7-cabundle-cert\") pod \"raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4\" (UID: \"d5b25188-d9d1-4261-b781-d93cbfcb2dc7\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4"
Jul 02 13:03:45.433122 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.432976    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5b25188-d9d1-4261-b781-d93cbfcb2dc7-kserve-provision-location\") pod \"raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4\" (UID: \"d5b25188-d9d1-4261-b781-d93cbfcb2dc7\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4"
Jul 02 13:03:45.534051 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.533965    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5b25188-d9d1-4261-b781-d93cbfcb2dc7-proxy-tls\") pod \"raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4\" (UID: \"d5b25188-d9d1-4261-b781-d93cbfcb2dc7\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4"
Jul 02 13:03:45.534051 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.534013    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5b25188-d9d1-4261-b781-d93cbfcb2dc7-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4\" (UID: \"d5b25188-d9d1-4261-b781-d93cbfcb2dc7\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4"
Jul 02 13:03:45.534256 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.534053    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgkpn\" (UniqueName: \"kubernetes.io/projected/d5b25188-d9d1-4261-b781-d93cbfcb2dc7-kube-api-access-sgkpn\") pod \"raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4\" (UID: \"d5b25188-d9d1-4261-b781-d93cbfcb2dc7\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4"
Jul 02 13:03:45.534256 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.534073    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5b25188-d9d1-4261-b781-d93cbfcb2dc7-cabundle-cert\") pod \"raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4\" (UID: \"d5b25188-d9d1-4261-b781-d93cbfcb2dc7\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4"
Jul 02 13:03:45.534256 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.534108    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5b25188-d9d1-4261-b781-d93cbfcb2dc7-kserve-provision-location\") pod \"raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4\" (UID: \"d5b25188-d9d1-4261-b781-d93cbfcb2dc7\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4"
Jul 02 13:03:45.534553 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.534529    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5b25188-d9d1-4261-b781-d93cbfcb2dc7-kserve-provision-location\") pod \"raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4\" (UID: \"d5b25188-d9d1-4261-b781-d93cbfcb2dc7\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4"
Jul 02 13:03:45.534823 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.534796    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5b25188-d9d1-4261-b781-d93cbfcb2dc7-cabundle-cert\") pod \"raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4\" (UID: \"d5b25188-d9d1-4261-b781-d93cbfcb2dc7\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4"
Jul 02 13:03:45.534890 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.534804    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5b25188-d9d1-4261-b781-d93cbfcb2dc7-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4\" (UID: \"d5b25188-d9d1-4261-b781-d93cbfcb2dc7\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4"
Jul 02 13:03:45.536590 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.536568    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5b25188-d9d1-4261-b781-d93cbfcb2dc7-proxy-tls\") pod \"raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4\" (UID: \"d5b25188-d9d1-4261-b781-d93cbfcb2dc7\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4"
Jul 02 13:03:45.548829 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.548802    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgkpn\" (UniqueName: \"kubernetes.io/projected/d5b25188-d9d1-4261-b781-d93cbfcb2dc7-kube-api-access-sgkpn\") pod \"raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4\" (UID: \"d5b25188-d9d1-4261-b781-d93cbfcb2dc7\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4"
Jul 02 13:03:45.703584 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.703547    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4"
Jul 02 13:03:45.839997 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:45.839973    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4"]
Jul 02 13:03:45.841871 ip-10-0-143-88 kubenswrapper[2576]: W0702 13:03:45.841846    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5b25188_d9d1_4261_b781_d93cbfcb2dc7.slice/crio-7b5ef40421b9ea42000750925a863cf6680a1b7f230b0bb2733128ad9c921337 WatchSource:0}: Error finding container 7b5ef40421b9ea42000750925a863cf6680a1b7f230b0bb2733128ad9c921337: Status 404 returned error can't find the container with id 7b5ef40421b9ea42000750925a863cf6680a1b7f230b0bb2733128ad9c921337
Jul 02 13:03:46.256116 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:46.256087    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4" event={"ID":"d5b25188-d9d1-4261-b781-d93cbfcb2dc7","Type":"ContainerStarted","Data":"5d0a80ba7b0eb8e8efb11da24aed5bbeeefca8c9590815c7bc9d7fed0dad249c"}
Jul 02 13:03:46.256289 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:46.256141    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4" event={"ID":"d5b25188-d9d1-4261-b781-d93cbfcb2dc7","Type":"ContainerStarted","Data":"7b5ef40421b9ea42000750925a863cf6680a1b7f230b0bb2733128ad9c921337"}
Jul 02 13:03:46.863241 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:46.863202    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b240a6aa-163e-41b8-9423-e4e5cd8ed447" path="/var/lib/kubelet/pods/b240a6aa-163e-41b8-9423-e4e5cd8ed447/volumes"
Jul 02 13:03:50.276348 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:50.276311    2576 generic.go:356] "Generic (PLEG): container finished" podID="d5b25188-d9d1-4261-b781-d93cbfcb2dc7" containerID="5d0a80ba7b0eb8e8efb11da24aed5bbeeefca8c9590815c7bc9d7fed0dad249c" exitCode=0
Jul 02 13:03:50.276727 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:50.276371    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4" event={"ID":"d5b25188-d9d1-4261-b781-d93cbfcb2dc7","Type":"ContainerDied","Data":"5d0a80ba7b0eb8e8efb11da24aed5bbeeefca8c9590815c7bc9d7fed0dad249c"}
Jul 02 13:03:51.282810 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:51.282777    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4" event={"ID":"d5b25188-d9d1-4261-b781-d93cbfcb2dc7","Type":"ContainerStarted","Data":"6ec96028a0cb43e8e5af7fc49f4d6feb2323e7e5f40b644a28d3673fa4fea112"}
Jul 02 13:03:51.283258 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:51.282838    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4" event={"ID":"d5b25188-d9d1-4261-b781-d93cbfcb2dc7","Type":"ContainerStarted","Data":"ee74b6167acad1fd631fc89a1ce5ee31e58be3e30cf24ecf9383a116f730dacf"}
Jul 02 13:03:51.283258 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:51.283139    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4"
Jul 02 13:03:51.283386 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:51.283288    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4"
Jul 02 13:03:51.284769 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:51.284742    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4" podUID="d5b25188-d9d1-4261-b781-d93cbfcb2dc7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 02 13:03:51.304477 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:51.304434    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4" podStartSLOduration=2.663784227 podStartE2EDuration="6.304421834s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.640637607s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:03:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:03:51.30236811 +0000 UTC m=+1519.909831136" watchObservedRunningTime="2026-07-02 13:03:51.304421834 +0000 UTC m=+1519.911884924"
Jul 02 13:03:52.287903 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:52.287864    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4" podUID="d5b25188-d9d1-4261-b781-d93cbfcb2dc7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 02 13:03:57.292256 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:57.292228    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4"
Jul 02 13:03:57.292788 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:03:57.292759    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4" podUID="d5b25188-d9d1-4261-b781-d93cbfcb2dc7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 02 13:04:07.292749 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:04:07.292705    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4" podUID="d5b25188-d9d1-4261-b781-d93cbfcb2dc7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 02 13:04:17.292894 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:04:17.292842    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4" podUID="d5b25188-d9d1-4261-b781-d93cbfcb2dc7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 02 13:04:27.293451 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:04:27.293406    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4" podUID="d5b25188-d9d1-4261-b781-d93cbfcb2dc7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 02 13:04:37.293654 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:04:37.293607    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4" podUID="d5b25188-d9d1-4261-b781-d93cbfcb2dc7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 02 13:04:47.293088 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:04:47.293049    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4" podUID="d5b25188-d9d1-4261-b781-d93cbfcb2dc7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 02 13:04:57.292678 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:04:57.292644    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4" podUID="d5b25188-d9d1-4261-b781-d93cbfcb2dc7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 02 13:05:07.294077 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:07.294049    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4"
Jul 02 13:05:15.363551 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:15.363512    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4"]
Jul 02 13:05:15.363965 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:15.363869    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4" podUID="d5b25188-d9d1-4261-b781-d93cbfcb2dc7" containerName="kserve-container" containerID="cri-o://ee74b6167acad1fd631fc89a1ce5ee31e58be3e30cf24ecf9383a116f730dacf" gracePeriod=30
Jul 02 13:05:15.363965 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:15.363904    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4" podUID="d5b25188-d9d1-4261-b781-d93cbfcb2dc7" containerName="kube-rbac-proxy" containerID="cri-o://6ec96028a0cb43e8e5af7fc49f4d6feb2323e7e5f40b644a28d3673fa4fea112" gracePeriod=30
Jul 02 13:05:15.695367 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:15.695329    2576 generic.go:356] "Generic (PLEG): container finished" podID="d5b25188-d9d1-4261-b781-d93cbfcb2dc7" containerID="6ec96028a0cb43e8e5af7fc49f4d6feb2323e7e5f40b644a28d3673fa4fea112" exitCode=2
Jul 02 13:05:15.695565 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:15.695398    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4" event={"ID":"d5b25188-d9d1-4261-b781-d93cbfcb2dc7","Type":"ContainerDied","Data":"6ec96028a0cb43e8e5af7fc49f4d6feb2323e7e5f40b644a28d3673fa4fea112"}
Jul 02 13:05:17.288596 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:17.288547    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4" podUID="d5b25188-d9d1-4261-b781-d93cbfcb2dc7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Jul 02 13:05:17.293017 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:17.292981    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4" podUID="d5b25188-d9d1-4261-b781-d93cbfcb2dc7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 02 13:05:20.113868 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:20.113844    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4"
Jul 02 13:05:20.193657 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:20.193569    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5b25188-d9d1-4261-b781-d93cbfcb2dc7-kube-rbac-proxy-sar-config\") pod \"d5b25188-d9d1-4261-b781-d93cbfcb2dc7\" (UID: \"d5b25188-d9d1-4261-b781-d93cbfcb2dc7\") "
Jul 02 13:05:20.193657 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:20.193629    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgkpn\" (UniqueName: \"kubernetes.io/projected/d5b25188-d9d1-4261-b781-d93cbfcb2dc7-kube-api-access-sgkpn\") pod \"d5b25188-d9d1-4261-b781-d93cbfcb2dc7\" (UID: \"d5b25188-d9d1-4261-b781-d93cbfcb2dc7\") "
Jul 02 13:05:20.193894 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:20.193661    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5b25188-d9d1-4261-b781-d93cbfcb2dc7-kserve-provision-location\") pod \"d5b25188-d9d1-4261-b781-d93cbfcb2dc7\" (UID: \"d5b25188-d9d1-4261-b781-d93cbfcb2dc7\") "
Jul 02 13:05:20.193894 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:20.193735    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5b25188-d9d1-4261-b781-d93cbfcb2dc7-cabundle-cert\") pod \"d5b25188-d9d1-4261-b781-d93cbfcb2dc7\" (UID: \"d5b25188-d9d1-4261-b781-d93cbfcb2dc7\") "
Jul 02 13:05:20.193894 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:20.193759    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5b25188-d9d1-4261-b781-d93cbfcb2dc7-proxy-tls\") pod \"d5b25188-d9d1-4261-b781-d93cbfcb2dc7\" (UID: \"d5b25188-d9d1-4261-b781-d93cbfcb2dc7\") "
Jul 02 13:05:20.194136 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:20.193984    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5b25188-d9d1-4261-b781-d93cbfcb2dc7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d5b25188-d9d1-4261-b781-d93cbfcb2dc7" (UID: "d5b25188-d9d1-4261-b781-d93cbfcb2dc7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:05:20.194136 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:20.194016    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5b25188-d9d1-4261-b781-d93cbfcb2dc7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d5b25188-d9d1-4261-b781-d93cbfcb2dc7" (UID: "d5b25188-d9d1-4261-b781-d93cbfcb2dc7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:05:20.194214 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:20.194133    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5b25188-d9d1-4261-b781-d93cbfcb2dc7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d5b25188-d9d1-4261-b781-d93cbfcb2dc7" (UID: "d5b25188-d9d1-4261-b781-d93cbfcb2dc7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:05:20.195809 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:20.195781    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5b25188-d9d1-4261-b781-d93cbfcb2dc7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d5b25188-d9d1-4261-b781-d93cbfcb2dc7" (UID: "d5b25188-d9d1-4261-b781-d93cbfcb2dc7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:05:20.195958 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:20.195832    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5b25188-d9d1-4261-b781-d93cbfcb2dc7-kube-api-access-sgkpn" (OuterVolumeSpecName: "kube-api-access-sgkpn") pod "d5b25188-d9d1-4261-b781-d93cbfcb2dc7" (UID: "d5b25188-d9d1-4261-b781-d93cbfcb2dc7"). InnerVolumeSpecName "kube-api-access-sgkpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:05:20.294804 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:20.294760    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5b25188-d9d1-4261-b781-d93cbfcb2dc7-cabundle-cert\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 13:05:20.294804 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:20.294802    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5b25188-d9d1-4261-b781-d93cbfcb2dc7-proxy-tls\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 13:05:20.295050 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:20.294820    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5b25188-d9d1-4261-b781-d93cbfcb2dc7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 13:05:20.295050 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:20.294836    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgkpn\" (UniqueName: \"kubernetes.io/projected/d5b25188-d9d1-4261-b781-d93cbfcb2dc7-kube-api-access-sgkpn\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 13:05:20.295050 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:20.294852    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5b25188-d9d1-4261-b781-d93cbfcb2dc7-kserve-provision-location\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\""
Jul 02 13:05:20.721754 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:20.721723    2576 generic.go:356] "Generic (PLEG): container finished" podID="d5b25188-d9d1-4261-b781-d93cbfcb2dc7" containerID="ee74b6167acad1fd631fc89a1ce5ee31e58be3e30cf24ecf9383a116f730dacf" exitCode=0
Jul 02 13:05:20.721952 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:20.721784    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4"
Jul 02 13:05:20.721952 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:20.721800    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4" event={"ID":"d5b25188-d9d1-4261-b781-d93cbfcb2dc7","Type":"ContainerDied","Data":"ee74b6167acad1fd631fc89a1ce5ee31e58be3e30cf24ecf9383a116f730dacf"}
Jul 02 13:05:20.721952 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:20.721890    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4" event={"ID":"d5b25188-d9d1-4261-b781-d93cbfcb2dc7","Type":"ContainerDied","Data":"7b5ef40421b9ea42000750925a863cf6680a1b7f230b0bb2733128ad9c921337"}
Jul 02 13:05:20.721952 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:20.721940    2576 scope.go:117] "RemoveContainer" containerID="6ec96028a0cb43e8e5af7fc49f4d6feb2323e7e5f40b644a28d3673fa4fea112"
Jul 02 13:05:20.731010 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:20.730994    2576 scope.go:117] "RemoveContainer" containerID="ee74b6167acad1fd631fc89a1ce5ee31e58be3e30cf24ecf9383a116f730dacf"
Jul 02 13:05:20.738302 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:20.738284    2576 scope.go:117] "RemoveContainer" containerID="5d0a80ba7b0eb8e8efb11da24aed5bbeeefca8c9590815c7bc9d7fed0dad249c"
Jul 02 13:05:20.746226 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:20.745662    2576 scope.go:117] "RemoveContainer" containerID="6ec96028a0cb43e8e5af7fc49f4d6feb2323e7e5f40b644a28d3673fa4fea112"
Jul 02 13:05:20.746226 ip-10-0-143-88 kubenswrapper[2576]: E0702 13:05:20.746066    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ec96028a0cb43e8e5af7fc49f4d6feb2323e7e5f40b644a28d3673fa4fea112\": container with ID starting with 6ec96028a0cb43e8e5af7fc49f4d6feb2323e7e5f40b644a28d3673fa4fea112 not found: ID does not exist" containerID="6ec96028a0cb43e8e5af7fc49f4d6feb2323e7e5f40b644a28d3673fa4fea112"
Jul 02 13:05:20.746226 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:20.746103    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ec96028a0cb43e8e5af7fc49f4d6feb2323e7e5f40b644a28d3673fa4fea112"} err="failed to get container status \"6ec96028a0cb43e8e5af7fc49f4d6feb2323e7e5f40b644a28d3673fa4fea112\": rpc error: code = NotFound desc = could not find container \"6ec96028a0cb43e8e5af7fc49f4d6feb2323e7e5f40b644a28d3673fa4fea112\": container with ID starting with 6ec96028a0cb43e8e5af7fc49f4d6feb2323e7e5f40b644a28d3673fa4fea112 not found: ID does not exist"
Jul 02 13:05:20.746226 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:20.746126    2576 scope.go:117] "RemoveContainer" containerID="ee74b6167acad1fd631fc89a1ce5ee31e58be3e30cf24ecf9383a116f730dacf"
Jul 02 13:05:20.746536 ip-10-0-143-88 kubenswrapper[2576]: E0702 13:05:20.746395    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee74b6167acad1fd631fc89a1ce5ee31e58be3e30cf24ecf9383a116f730dacf\": container with ID starting with ee74b6167acad1fd631fc89a1ce5ee31e58be3e30cf24ecf9383a116f730dacf not found: ID does not exist" containerID="ee74b6167acad1fd631fc89a1ce5ee31e58be3e30cf24ecf9383a116f730dacf"
Jul 02 13:05:20.746536 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:20.746419    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee74b6167acad1fd631fc89a1ce5ee31e58be3e30cf24ecf9383a116f730dacf"} err="failed to get container status \"ee74b6167acad1fd631fc89a1ce5ee31e58be3e30cf24ecf9383a116f730dacf\": rpc error: code = NotFound desc = could not find container \"ee74b6167acad1fd631fc89a1ce5ee31e58be3e30cf24ecf9383a116f730dacf\": container with ID starting with ee74b6167acad1fd631fc89a1ce5ee31e58be3e30cf24ecf9383a116f730dacf not found: ID does not exist"
Jul 02 13:05:20.746536 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:20.746435    2576 scope.go:117] "RemoveContainer" containerID="5d0a80ba7b0eb8e8efb11da24aed5bbeeefca8c9590815c7bc9d7fed0dad249c"
Jul 02 13:05:20.746725 ip-10-0-143-88 kubenswrapper[2576]: E0702 13:05:20.746652    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d0a80ba7b0eb8e8efb11da24aed5bbeeefca8c9590815c7bc9d7fed0dad249c\": container with ID starting with 5d0a80ba7b0eb8e8efb11da24aed5bbeeefca8c9590815c7bc9d7fed0dad249c not found: ID does not exist" containerID="5d0a80ba7b0eb8e8efb11da24aed5bbeeefca8c9590815c7bc9d7fed0dad249c"
Jul 02 13:05:20.746725 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:20.746673    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d0a80ba7b0eb8e8efb11da24aed5bbeeefca8c9590815c7bc9d7fed0dad249c"} err="failed to get container status \"5d0a80ba7b0eb8e8efb11da24aed5bbeeefca8c9590815c7bc9d7fed0dad249c\": rpc error: code = NotFound desc = could not find container \"5d0a80ba7b0eb8e8efb11da24aed5bbeeefca8c9590815c7bc9d7fed0dad249c\": container with ID starting with 5d0a80ba7b0eb8e8efb11da24aed5bbeeefca8c9590815c7bc9d7fed0dad249c not found: ID does not exist"
Jul 02 13:05:20.748144 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:20.748124    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4"]
Jul 02 13:05:20.750430 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:20.750410    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-4a543-predictor-74489d986-lhhs4"]
Jul 02 13:05:20.863667 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:20.863635    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5b25188-d9d1-4261-b781-d93cbfcb2dc7" path="/var/lib/kubelet/pods/d5b25188-d9d1-4261-b781-d93cbfcb2dc7/volumes"
Jul 02 13:05:50.823373 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:50.823337    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5khgm/must-gather-gh8q9"]
Jul 02 13:05:50.823846 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:50.823699    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5b25188-d9d1-4261-b781-d93cbfcb2dc7" containerName="kserve-container"
Jul 02 13:05:50.823846 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:50.823709    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5b25188-d9d1-4261-b781-d93cbfcb2dc7" containerName="kserve-container"
Jul 02 13:05:50.823846 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:50.823727    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5b25188-d9d1-4261-b781-d93cbfcb2dc7" containerName="kube-rbac-proxy"
Jul 02 13:05:50.823846 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:50.823733    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5b25188-d9d1-4261-b781-d93cbfcb2dc7" containerName="kube-rbac-proxy"
Jul 02 13:05:50.823846 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:50.823743    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5b25188-d9d1-4261-b781-d93cbfcb2dc7" containerName="storage-initializer"
Jul 02 13:05:50.823846 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:50.823748    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5b25188-d9d1-4261-b781-d93cbfcb2dc7" containerName="storage-initializer"
Jul 02 13:05:50.823846 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:50.823803    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5b25188-d9d1-4261-b781-d93cbfcb2dc7" containerName="kserve-container"
Jul 02 13:05:50.823846 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:50.823812    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5b25188-d9d1-4261-b781-d93cbfcb2dc7" containerName="kube-rbac-proxy"
Jul 02 13:05:50.826981 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:50.826961    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5khgm/must-gather-gh8q9"
Jul 02 13:05:50.829942 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:50.829898    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5khgm\"/\"openshift-service-ca.crt\""
Jul 02 13:05:50.830077 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:50.829898    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5khgm\"/\"kube-root-ca.crt\""
Jul 02 13:05:50.830077 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:50.830010    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5khgm\"/\"default-dockercfg-crx8v\""
Jul 02 13:05:50.836863 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:50.836841    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5khgm/must-gather-gh8q9"]
Jul 02 13:05:50.863197 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:50.863163    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqrfq\" (UniqueName: \"kubernetes.io/projected/6a7948f5-8d0f-4e14-a461-623819b3f988-kube-api-access-dqrfq\") pod \"must-gather-gh8q9\" (UID: \"6a7948f5-8d0f-4e14-a461-623819b3f988\") " pod="openshift-must-gather-5khgm/must-gather-gh8q9"
Jul 02 13:05:50.863661 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:50.863474    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6a7948f5-8d0f-4e14-a461-623819b3f988-must-gather-output\") pod \"must-gather-gh8q9\" (UID: \"6a7948f5-8d0f-4e14-a461-623819b3f988\") " pod="openshift-must-gather-5khgm/must-gather-gh8q9"
Jul 02 13:05:50.964255 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:50.964219    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqrfq\" (UniqueName: \"kubernetes.io/projected/6a7948f5-8d0f-4e14-a461-623819b3f988-kube-api-access-dqrfq\") pod \"must-gather-gh8q9\" (UID: \"6a7948f5-8d0f-4e14-a461-623819b3f988\") " pod="openshift-must-gather-5khgm/must-gather-gh8q9"
Jul 02 13:05:50.964460 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:50.964300    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6a7948f5-8d0f-4e14-a461-623819b3f988-must-gather-output\") pod \"must-gather-gh8q9\" (UID: \"6a7948f5-8d0f-4e14-a461-623819b3f988\") " pod="openshift-must-gather-5khgm/must-gather-gh8q9"
Jul 02 13:05:50.964721 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:50.964700    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6a7948f5-8d0f-4e14-a461-623819b3f988-must-gather-output\") pod \"must-gather-gh8q9\" (UID: \"6a7948f5-8d0f-4e14-a461-623819b3f988\") " pod="openshift-must-gather-5khgm/must-gather-gh8q9"
Jul 02 13:05:50.975319 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:50.975292    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqrfq\" (UniqueName: \"kubernetes.io/projected/6a7948f5-8d0f-4e14-a461-623819b3f988-kube-api-access-dqrfq\") pod \"must-gather-gh8q9\" (UID: \"6a7948f5-8d0f-4e14-a461-623819b3f988\") " pod="openshift-must-gather-5khgm/must-gather-gh8q9"
Jul 02 13:05:51.143660 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:51.143628    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5khgm/must-gather-gh8q9"
Jul 02 13:05:51.280563 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:51.280539    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5khgm/must-gather-gh8q9"]
Jul 02 13:05:51.282437 ip-10-0-143-88 kubenswrapper[2576]: W0702 13:05:51.282403    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a7948f5_8d0f_4e14_a461_623819b3f988.slice/crio-fe08d0727ee4e9c3a758145ce741716472281a7d1cb7c91322241f2422e56cc1 WatchSource:0}: Error finding container fe08d0727ee4e9c3a758145ce741716472281a7d1cb7c91322241f2422e56cc1: Status 404 returned error can't find the container with id fe08d0727ee4e9c3a758145ce741716472281a7d1cb7c91322241f2422e56cc1
Jul 02 13:05:51.284408 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:51.284389    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 13:05:51.870738 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:51.870705    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5khgm/must-gather-gh8q9" event={"ID":"6a7948f5-8d0f-4e14-a461-623819b3f988","Type":"ContainerStarted","Data":"fe08d0727ee4e9c3a758145ce741716472281a7d1cb7c91322241f2422e56cc1"}
Jul 02 13:05:52.879206 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:52.879170    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5khgm/must-gather-gh8q9" event={"ID":"6a7948f5-8d0f-4e14-a461-623819b3f988","Type":"ContainerStarted","Data":"77a2eb8863158259f384201bef208659ee90d4da8aa1d7d48b7108768f62f989"}
Jul 02 13:05:52.879646 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:52.879220    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5khgm/must-gather-gh8q9" event={"ID":"6a7948f5-8d0f-4e14-a461-623819b3f988","Type":"ContainerStarted","Data":"ff66dbe120785386d9760fe2fb485921e5082934e67603981d90abb75f0aee16"}
Jul 02 13:05:52.897985 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:52.897907    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5khgm/must-gather-gh8q9" podStartSLOduration=1.945527311 podStartE2EDuration="2.897882388s" totalImagesPullingTime="952.355077ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:05:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:05:52.89505912 +0000 UTC m=+1641.502522145" watchObservedRunningTime="2026-07-02 13:05:52.897882388 +0000 UTC m=+1641.505345412"
Jul 02 13:05:53.741705 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:53.741668    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-72n2l_116aa48e-717b-4d49-897d-5538398f4a2a/global-pull-secret-syncer/0.log"
Jul 02 13:05:54.012270 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:54.012187    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-t6qwm_7f9139f3-1b94-4ddf-854a-53db4e0693f1/konnectivity-agent/0.log"
Jul 02 13:05:54.112100 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:54.112067    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-88.ec2.internal_6f05da72d6558d5d49cf0691bccf02a9/haproxy/0.log"
Jul 02 13:05:57.697698 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:57.697664    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e61794ef-6594-4970-b7ff-3897895050fb/alertmanager/0.log"
Jul 02 13:05:57.755311 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:57.755283    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e61794ef-6594-4970-b7ff-3897895050fb/config-reloader/0.log"
Jul 02 13:05:57.837198 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:57.837171    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e61794ef-6594-4970-b7ff-3897895050fb/kube-rbac-proxy-web/0.log"
Jul 02 13:05:57.942284 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:57.942250    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e61794ef-6594-4970-b7ff-3897895050fb/kube-rbac-proxy/0.log"
Jul 02 13:05:58.034699 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:58.034617    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e61794ef-6594-4970-b7ff-3897895050fb/kube-rbac-proxy-metric/0.log"
Jul 02 13:05:58.170436 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:58.170402    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e61794ef-6594-4970-b7ff-3897895050fb/prom-label-proxy/0.log"
Jul 02 13:05:58.224161 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:58.224127    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e61794ef-6594-4970-b7ff-3897895050fb/init-config-reloader/0.log"
Jul 02 13:05:58.288757 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:58.288678    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-54cb84584-dz8tm_ac58c65c-c830-4ba0-aef5-c5745ed26d93/cluster-monitoring-operator/0.log"
Jul 02 13:05:58.319022 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:58.318992    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-5kfkq_b9cd847c-daf5-40f5-9a23-2718fa67f25c/kube-state-metrics/0.log"
Jul 02 13:05:58.348492 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:58.348465    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-5kfkq_b9cd847c-daf5-40f5-9a23-2718fa67f25c/kube-rbac-proxy-main/0.log"
Jul 02 13:05:58.376965 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:58.376895    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-5kfkq_b9cd847c-daf5-40f5-9a23-2718fa67f25c/kube-rbac-proxy-self/0.log"
Jul 02 13:05:58.413608 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:58.413582    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5c86dcd54c-vfkvr_eacf016b-61ce-4c87-8963-a3c365506452/metrics-server/0.log"
Jul 02 13:05:58.672800 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:58.672740    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w5qjd_eedaddc4-a131-49bd-b077-4a61b9b8f04e/node-exporter/0.log"
Jul 02 13:05:58.704283 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:58.704244    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w5qjd_eedaddc4-a131-49bd-b077-4a61b9b8f04e/kube-rbac-proxy/0.log"
Jul 02 13:05:58.738115 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:58.738094    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w5qjd_eedaddc4-a131-49bd-b077-4a61b9b8f04e/init-textfile/0.log"
Jul 02 13:05:59.136781 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:59.136705    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7b7d896497-vzhw4_d253c0c2-9179-4e35-9522-1c512e869900/prometheus-operator/0.log"
Jul 02 13:05:59.171383 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:59.171350    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7b7d896497-vzhw4_d253c0c2-9179-4e35-9522-1c512e869900/kube-rbac-proxy/0.log"
Jul 02 13:05:59.207447 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:59.207419    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7ddc8784d4-lrwvv_f185005f-3257-447b-9488-d15974db3138/prometheus-operator-admission-webhook/0.log"
Jul 02 13:05:59.247545 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:59.247502    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56bb4d87d4-wt4r7_f0e9931d-45ab-4c8e-8175-3da0da60ba21/telemeter-client/0.log"
Jul 02 13:05:59.283585 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:59.283557    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56bb4d87d4-wt4r7_f0e9931d-45ab-4c8e-8175-3da0da60ba21/reload/0.log"
Jul 02 13:05:59.311374 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:59.311341    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56bb4d87d4-wt4r7_f0e9931d-45ab-4c8e-8175-3da0da60ba21/kube-rbac-proxy/0.log"
Jul 02 13:05:59.350488 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:59.350457    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68df4fdd69-qp4qs_3a8ddc64-6cb7-4593-9e82-07428d018488/thanos-query/0.log"
Jul 02 13:05:59.379639 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:59.379606    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68df4fdd69-qp4qs_3a8ddc64-6cb7-4593-9e82-07428d018488/kube-rbac-proxy-web/0.log"
Jul 02 13:05:59.410557 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:59.410515    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68df4fdd69-qp4qs_3a8ddc64-6cb7-4593-9e82-07428d018488/kube-rbac-proxy/0.log"
Jul 02 13:05:59.440560 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:59.440534    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68df4fdd69-qp4qs_3a8ddc64-6cb7-4593-9e82-07428d018488/prom-label-proxy/0.log"
Jul 02 13:05:59.476309 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:59.476275    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68df4fdd69-qp4qs_3a8ddc64-6cb7-4593-9e82-07428d018488/kube-rbac-proxy-rules/0.log"
Jul 02 13:05:59.509731 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:05:59.509693    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68df4fdd69-qp4qs_3a8ddc64-6cb7-4593-9e82-07428d018488/kube-rbac-proxy-metrics/0.log"
Jul 02 13:06:00.826785 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:00.826750    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cdb5968b9-nfq4q_ffc27c9b-735f-444c-8b69-433ef52cf365/networking-console-plugin/0.log"
Jul 02 13:06:01.737785 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:01.737750    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5khgm/perf-node-gather-daemonset-qz4vp"]
Jul 02 13:06:01.743860 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:01.743832    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5khgm/perf-node-gather-daemonset-qz4vp"
Jul 02 13:06:01.753603 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:01.753570    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5khgm/perf-node-gather-daemonset-qz4vp"]
Jul 02 13:06:01.774367 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:01.774335    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2048ba6f-e090-4d2d-a4ed-1ffff46db3af-podres\") pod \"perf-node-gather-daemonset-qz4vp\" (UID: \"2048ba6f-e090-4d2d-a4ed-1ffff46db3af\") " pod="openshift-must-gather-5khgm/perf-node-gather-daemonset-qz4vp"
Jul 02 13:06:01.774543 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:01.774383    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2048ba6f-e090-4d2d-a4ed-1ffff46db3af-lib-modules\") pod \"perf-node-gather-daemonset-qz4vp\" (UID: \"2048ba6f-e090-4d2d-a4ed-1ffff46db3af\") " pod="openshift-must-gather-5khgm/perf-node-gather-daemonset-qz4vp"
Jul 02 13:06:01.774543 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:01.774430    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2048ba6f-e090-4d2d-a4ed-1ffff46db3af-sys\") pod \"perf-node-gather-daemonset-qz4vp\" (UID: \"2048ba6f-e090-4d2d-a4ed-1ffff46db3af\") " pod="openshift-must-gather-5khgm/perf-node-gather-daemonset-qz4vp"
Jul 02 13:06:01.774543 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:01.774478    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2048ba6f-e090-4d2d-a4ed-1ffff46db3af-proc\") pod \"perf-node-gather-daemonset-qz4vp\" (UID: \"2048ba6f-e090-4d2d-a4ed-1ffff46db3af\") " pod="openshift-must-gather-5khgm/perf-node-gather-daemonset-qz4vp"
Jul 02 13:06:01.774726 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:01.774649    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pksn5\" (UniqueName: \"kubernetes.io/projected/2048ba6f-e090-4d2d-a4ed-1ffff46db3af-kube-api-access-pksn5\") pod \"perf-node-gather-daemonset-qz4vp\" (UID: \"2048ba6f-e090-4d2d-a4ed-1ffff46db3af\") " pod="openshift-must-gather-5khgm/perf-node-gather-daemonset-qz4vp"
Jul 02 13:06:01.817402 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:01.817372    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-74684955bf-fx9k9_3049d03b-9514-4f31-b63d-8821c15d7998/console/0.log"
Jul 02 13:06:01.866991 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:01.866959    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5499b6fb8f-2c2lq_5afef10c-ce2e-4a9d-8e49-e385b6ed71a2/download-server/0.log"
Jul 02 13:06:01.875692 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:01.875657    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2048ba6f-e090-4d2d-a4ed-1ffff46db3af-podres\") pod \"perf-node-gather-daemonset-qz4vp\" (UID: \"2048ba6f-e090-4d2d-a4ed-1ffff46db3af\") " pod="openshift-must-gather-5khgm/perf-node-gather-daemonset-qz4vp"
Jul 02 13:06:01.875692 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:01.875691    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2048ba6f-e090-4d2d-a4ed-1ffff46db3af-lib-modules\") pod \"perf-node-gather-daemonset-qz4vp\" (UID: \"2048ba6f-e090-4d2d-a4ed-1ffff46db3af\") " pod="openshift-must-gather-5khgm/perf-node-gather-daemonset-qz4vp"
Jul 02 13:06:01.875942 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:01.875717    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2048ba6f-e090-4d2d-a4ed-1ffff46db3af-sys\") pod \"perf-node-gather-daemonset-qz4vp\" (UID: \"2048ba6f-e090-4d2d-a4ed-1ffff46db3af\") " pod="openshift-must-gather-5khgm/perf-node-gather-daemonset-qz4vp"
Jul 02 13:06:01.875942 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:01.875753    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2048ba6f-e090-4d2d-a4ed-1ffff46db3af-proc\") pod \"perf-node-gather-daemonset-qz4vp\" (UID: \"2048ba6f-e090-4d2d-a4ed-1ffff46db3af\") " pod="openshift-must-gather-5khgm/perf-node-gather-daemonset-qz4vp"
Jul 02 13:06:01.875942 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:01.875807    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pksn5\" (UniqueName: \"kubernetes.io/projected/2048ba6f-e090-4d2d-a4ed-1ffff46db3af-kube-api-access-pksn5\") pod \"perf-node-gather-daemonset-qz4vp\" (UID: \"2048ba6f-e090-4d2d-a4ed-1ffff46db3af\") " pod="openshift-must-gather-5khgm/perf-node-gather-daemonset-qz4vp"
Jul 02 13:06:01.875942 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:01.875831    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2048ba6f-e090-4d2d-a4ed-1ffff46db3af-podres\") pod \"perf-node-gather-daemonset-qz4vp\" (UID: \"2048ba6f-e090-4d2d-a4ed-1ffff46db3af\") " pod="openshift-must-gather-5khgm/perf-node-gather-daemonset-qz4vp"
Jul 02 13:06:01.875942 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:01.875838    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2048ba6f-e090-4d2d-a4ed-1ffff46db3af-lib-modules\") pod \"perf-node-gather-daemonset-qz4vp\" (UID: \"2048ba6f-e090-4d2d-a4ed-1ffff46db3af\") " pod="openshift-must-gather-5khgm/perf-node-gather-daemonset-qz4vp"
Jul 02 13:06:01.875942 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:01.875854    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2048ba6f-e090-4d2d-a4ed-1ffff46db3af-sys\") pod \"perf-node-gather-daemonset-qz4vp\" (UID: \"2048ba6f-e090-4d2d-a4ed-1ffff46db3af\") " pod="openshift-must-gather-5khgm/perf-node-gather-daemonset-qz4vp"
Jul 02 13:06:01.875942 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:01.875859    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2048ba6f-e090-4d2d-a4ed-1ffff46db3af-proc\") pod \"perf-node-gather-daemonset-qz4vp\" (UID: \"2048ba6f-e090-4d2d-a4ed-1ffff46db3af\") " pod="openshift-must-gather-5khgm/perf-node-gather-daemonset-qz4vp"
Jul 02 13:06:01.887067 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:01.887044    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pksn5\" (UniqueName: \"kubernetes.io/projected/2048ba6f-e090-4d2d-a4ed-1ffff46db3af-kube-api-access-pksn5\") pod \"perf-node-gather-daemonset-qz4vp\" (UID: \"2048ba6f-e090-4d2d-a4ed-1ffff46db3af\") " pod="openshift-must-gather-5khgm/perf-node-gather-daemonset-qz4vp"
Jul 02 13:06:02.061691 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:02.061602    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5khgm/perf-node-gather-daemonset-qz4vp"
Jul 02 13:06:02.238961 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:02.238890    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5khgm/perf-node-gather-daemonset-qz4vp"]
Jul 02 13:06:02.241267 ip-10-0-143-88 kubenswrapper[2576]: W0702 13:06:02.241228    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2048ba6f_e090_4d2d_a4ed_1ffff46db3af.slice/crio-b699738b19503adb5c615207b1d9ff989c372308ef1ee16272633c8b4d92373b WatchSource:0}: Error finding container b699738b19503adb5c615207b1d9ff989c372308ef1ee16272633c8b4d92373b: Status 404 returned error can't find the container with id b699738b19503adb5c615207b1d9ff989c372308ef1ee16272633c8b4d92373b
Jul 02 13:06:02.946706 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:02.946662    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5khgm/perf-node-gather-daemonset-qz4vp" event={"ID":"2048ba6f-e090-4d2d-a4ed-1ffff46db3af","Type":"ContainerStarted","Data":"4b1663262ff39e69de2caa33768e9d92ee58702118c0e497001855f3970fe753"}
Jul 02 13:06:02.947215 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:02.946732    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5khgm/perf-node-gather-daemonset-qz4vp" event={"ID":"2048ba6f-e090-4d2d-a4ed-1ffff46db3af","Type":"ContainerStarted","Data":"b699738b19503adb5c615207b1d9ff989c372308ef1ee16272633c8b4d92373b"}
Jul 02 13:06:02.947606 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:02.947583    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5khgm/perf-node-gather-daemonset-qz4vp"
Jul 02 13:06:02.967397 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:02.967329    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5khgm/perf-node-gather-daemonset-qz4vp" podStartSLOduration=1.967307125 podStartE2EDuration="1.967307125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:06:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:06:02.964347174 +0000 UTC m=+1651.571810212" watchObservedRunningTime="2026-07-02 13:06:02.967307125 +0000 UTC m=+1651.574770149"
Jul 02 13:06:03.579926 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:03.579884    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-s82bz_0c646b36-0375-43c3-a0eb-8e08a2821414/dns/0.log"
Jul 02 13:06:03.609610 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:03.609573    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-s82bz_0c646b36-0375-43c3-a0eb-8e08a2821414/kube-rbac-proxy/0.log"
Jul 02 13:06:03.679105 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:03.679075    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hmjl6_fa04af49-8c6c-46fa-bfcc-500d89b526c9/dns-node-resolver/0.log"
Jul 02 13:06:04.303322 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:04.303284    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mztvs_5f219618-c523-45c1-9042-7e46c4994e49/node-ca/0.log"
Jul 02 13:06:05.310078 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:05.310046    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-656bc596cc-m2fg7_02f1bfff-40d6-4805-8579-9e3f5a390e25/router/0.log"
Jul 02 13:06:05.821479 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:05.821450    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-shm9z_8d276bfd-8b47-48ae-a99c-6e4f069e0fd5/serve-healthcheck-canary/0.log"
Jul 02 13:06:06.233584 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:06.233536    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5f67b55555-r4pwp_dabc8a5d-7710-42b1-93e2-bf1a0f5c94e5/insights-operator/0.log"
Jul 02 13:06:06.233842 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:06.233824    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5f67b55555-r4pwp_dabc8a5d-7710-42b1-93e2-bf1a0f5c94e5/insights-operator/1.log"
Jul 02 13:06:06.420253 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:06.420225    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xn552_880228b8-bf07-4d91-802b-f05146179c2a/kube-rbac-proxy/0.log"
Jul 02 13:06:06.445861 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:06.445815    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xn552_880228b8-bf07-4d91-802b-f05146179c2a/exporter/0.log"
Jul 02 13:06:06.470529 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:06.470501    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xn552_880228b8-bf07-4d91-802b-f05146179c2a/extractor/0.log"
Jul 02 13:06:08.728847 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:08.728812    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-sfx7w_ae6d0696-29b3-4c83-985a-627e145834fb/server/0.log"
Jul 02 13:06:08.879937 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:08.879885    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-v6hf6_a15ff7a3-b6fa-4198-bd40-9a88d1605187/manager/0.log"
Jul 02 13:06:08.927663 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:08.927625    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-lvxwr_08faaa52-ffd6-433c-bddd-bbe21ea259ba/s3-init/0.log"
Jul 02 13:06:08.947198 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:08.947169    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-lvxwr_08faaa52-ffd6-433c-bddd-bbe21ea259ba/download-hf-model/0.log"
Jul 02 13:06:08.979295 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:08.979229    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-ctcr5_d50dab12-740d-4d20-a229-b42c7f5e2f8b/seaweedfs/0.log"
Jul 02 13:06:09.970040 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:09.970015    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5khgm/perf-node-gather-daemonset-qz4vp"
Jul 02 13:06:15.212338 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:15.212259    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hh6v9_64db6250-f644-4f6d-856d-f3186f40db09/kube-multus-additional-cni-plugins/0.log"
Jul 02 13:06:15.258888 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:15.258858    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hh6v9_64db6250-f644-4f6d-856d-f3186f40db09/egress-router-binary-copy/0.log"
Jul 02 13:06:15.291826 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:15.291799    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hh6v9_64db6250-f644-4f6d-856d-f3186f40db09/cni-plugins/0.log"
Jul 02 13:06:15.323385 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:15.323356    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hh6v9_64db6250-f644-4f6d-856d-f3186f40db09/bond-cni-plugin/0.log"
Jul 02 13:06:15.362724 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:15.362694    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hh6v9_64db6250-f644-4f6d-856d-f3186f40db09/routeoverride-cni/0.log"
Jul 02 13:06:15.391502 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:15.391467    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hh6v9_64db6250-f644-4f6d-856d-f3186f40db09/whereabouts-cni-bincopy/0.log"
Jul 02 13:06:15.416390 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:15.416364    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hh6v9_64db6250-f644-4f6d-856d-f3186f40db09/whereabouts-cni/0.log"
Jul 02 13:06:15.677243 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:15.677211    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-fzrpv_6652ccdf-a3c9-42f8-88fc-39690889dd42/kube-multus/0.log"
Jul 02 13:06:15.843749 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:15.843717    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-sbnql_86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7/network-metrics-daemon/0.log"
Jul 02 13:06:15.867999 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:15.867974    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-sbnql_86f0cfb8-8ff4-4bfe-9b61-42b97b9c76d7/kube-rbac-proxy/0.log"
Jul 02 13:06:16.802550 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:16.802517    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2c7ks_116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a/ovn-controller/0.log"
Jul 02 13:06:16.843789 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:16.843760    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2c7ks_116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a/ovn-acl-logging/0.log"
Jul 02 13:06:16.843987 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:16.843764    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2c7ks_116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a/ovn-acl-logging/1.log"
Jul 02 13:06:16.871473 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:16.871447    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2c7ks_116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a/kube-rbac-proxy-node/0.log"
Jul 02 13:06:16.912970 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:16.912933    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2c7ks_116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a/kube-rbac-proxy-ovn-metrics/0.log"
Jul 02 13:06:16.947551 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:16.947519    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2c7ks_116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a/northd/0.log"
Jul 02 13:06:16.997148 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:16.997120    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2c7ks_116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a/nbdb/0.log"
Jul 02 13:06:17.048903 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:17.048868    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2c7ks_116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a/sbdb/0.log"
Jul 02 13:06:17.149940 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:17.149902    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2c7ks_116c84eb-0c3e-4bfc-a1ea-bec6c7fb9a9a/ovnkube-controller/0.log"
Jul 02 13:06:19.132803 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:19.132772    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-56499946bc-9fqqb_afc2b803-2e88-4ea8-ac6c-05ac215e33e9/check-endpoints/0.log"
Jul 02 13:06:19.187961 ip-10-0-143-88 kubenswrapper[2576]: I0702 13:06:19.187925    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-tczr2_ebc69c1d-a909-44d0-bcb1-691e39c5e754/network-check-target-container/0.log"