Jul 01 19:19:19.507987 ip-10-0-137-164 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 01 19:19:19.507997 ip-10-0-137-164 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 01 19:19:19.508004 ip-10-0-137-164 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 01 19:19:19.508248 ip-10-0-137-164 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 01 19:19:29.603384 ip-10-0-137-164 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 01 19:19:29.603400 ip-10-0-137-164 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 3b4f619704564223b3b942aa5337d1f1 --
Jul 01 19:21:32.273030 ip-10-0-137-164 systemd[1]: Starting Kubernetes Kubelet...
Jul 01 19:21:32.712299 ip-10-0-137-164 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 19:21:32.712299 ip-10-0-137-164 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 01 19:21:32.712299 ip-10-0-137-164 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 19:21:32.712299 ip-10-0-137-164 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 01 19:21:32.712299 ip-10-0-137-164 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 19:21:32.712299 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.669180    2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 01 19:21:32.712299 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673401    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 19:21:32.712299 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673411    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 19:21:32.712299 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673414    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 19:21:32.712299 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673417    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 19:21:32.712299 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673420    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 19:21:32.712299 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673423    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 19:21:32.712299 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673426    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 19:21:32.712299 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673429    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 19:21:32.713248 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673432    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 19:21:32.713248 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673434    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 19:21:32.713248 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673437    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 19:21:32.713248 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673439    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 19:21:32.713248 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673442    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 19:21:32.713248 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673444    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 19:21:32.713248 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673447    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 19:21:32.713248 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673449    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 19:21:32.713248 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673452    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 19:21:32.713248 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673454    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 19:21:32.713248 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673457    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 19:21:32.713248 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673460    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 19:21:32.713248 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673463    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 19:21:32.713248 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673465    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 19:21:32.713248 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673468    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 19:21:32.713248 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673472    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 19:21:32.713248 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673476    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 19:21:32.713248 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673479    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 19:21:32.713248 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673481    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 19:21:32.713800 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673484    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 19:21:32.713800 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673487    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 19:21:32.713800 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673490    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 19:21:32.713800 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673493    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 19:21:32.713800 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673496    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 19:21:32.713800 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673498    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 19:21:32.713800 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673501    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 19:21:32.713800 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673503    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 19:21:32.713800 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673506    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 19:21:32.713800 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673509    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 19:21:32.713800 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673511    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 19:21:32.713800 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673514    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 19:21:32.713800 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673516    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 19:21:32.713800 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673519    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 19:21:32.713800 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673522    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 19:21:32.713800 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673525    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 19:21:32.713800 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673527    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 19:21:32.713800 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673530    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 19:21:32.713800 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673533    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 19:21:32.713800 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673535    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 19:21:32.714461 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673538    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 19:21:32.714461 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673540    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 19:21:32.714461 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673543    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 19:21:32.714461 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673545    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 19:21:32.714461 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673548    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 19:21:32.714461 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673551    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 19:21:32.714461 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673553    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 19:21:32.714461 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673555    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 19:21:32.714461 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673558    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 19:21:32.714461 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673560    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 19:21:32.714461 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673563    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 19:21:32.714461 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673565    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 19:21:32.714461 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673568    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 19:21:32.714461 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673570    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 19:21:32.714461 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673573    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 19:21:32.714461 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673575    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 19:21:32.714461 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673578    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 19:21:32.714461 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673580    2574 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 19:21:32.714461 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673582    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 19:21:32.714461 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673585    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 19:21:32.715113 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673587    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 19:21:32.715113 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673590    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 19:21:32.715113 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673592    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 19:21:32.715113 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673595    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 19:21:32.715113 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673598    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 19:21:32.715113 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673601    2574 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 19:21:32.715113 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673603    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 19:21:32.715113 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673606    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 19:21:32.715113 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673608    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 19:21:32.715113 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673611    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 19:21:32.715113 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673613    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 19:21:32.715113 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673616    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 19:21:32.715113 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673619    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 19:21:32.715113 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673621    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 19:21:32.715113 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673624    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 19:21:32.715113 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673626    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 19:21:32.715113 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673632    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 19:21:32.715113 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673636    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 19:21:32.715113 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673639    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 19:21:32.715651 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673643    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 19:21:32.715651 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673646    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 19:21:32.715651 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673649    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 19:21:32.715651 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673652    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 19:21:32.715651 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673655    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 19:21:32.715651 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673658    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 19:21:32.715651 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673661    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 19:21:32.715651 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673664    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 19:21:32.715651 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673667    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 19:21:32.715651 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673669    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 19:21:32.715651 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673672    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 19:21:32.715651 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673674    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 19:21:32.715651 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.673677    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 19:21:32.715651 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675187    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 19:21:32.715651 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675194    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 19:21:32.715651 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675198    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 19:21:32.715651 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675201    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 19:21:32.715651 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675203    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 19:21:32.715651 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675207    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 19:21:32.716177 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675210    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 19:21:32.716177 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675213    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 19:21:32.716177 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675216    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 19:21:32.716177 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675219    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 19:21:32.716177 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675221    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 19:21:32.716177 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675225    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 19:21:32.716177 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675229    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 19:21:32.716177 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675232    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 19:21:32.716177 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675234    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 19:21:32.716177 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675237    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 19:21:32.716177 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675239    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 19:21:32.716177 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675242    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 19:21:32.716177 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675246    2574 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 19:21:32.716177 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675248    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 19:21:32.716177 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675251    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 19:21:32.716177 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675254    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 19:21:32.716177 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675257    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 19:21:32.716177 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675259    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 19:21:32.716177 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675262    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 19:21:32.716794 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675265    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 19:21:32.716794 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675269    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 19:21:32.716794 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675272    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 19:21:32.716794 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675275    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 19:21:32.716794 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675278    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 19:21:32.716794 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675281    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 19:21:32.716794 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675283    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 19:21:32.716794 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675286    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 19:21:32.716794 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675289    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 19:21:32.716794 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675291    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 19:21:32.716794 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675294    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 19:21:32.716794 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675296    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 19:21:32.716794 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675299    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 19:21:32.716794 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675302    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 19:21:32.716794 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675304    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 19:21:32.716794 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675306    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 19:21:32.716794 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675310    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 19:21:32.716794 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675312    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 19:21:32.716794 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675315    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 19:21:32.717534 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675318    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 19:21:32.717534 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675320    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 19:21:32.717534 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675322    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 19:21:32.717534 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675325    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 19:21:32.717534 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675327    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 19:21:32.717534 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675330    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 19:21:32.717534 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675333    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 19:21:32.717534 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675335    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 19:21:32.717534 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675338    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 19:21:32.717534 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675340    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 19:21:32.717534 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675343    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 19:21:32.717534 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675346    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 19:21:32.717534 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675349    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 19:21:32.717534 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675351    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 19:21:32.717534 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675353    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 19:21:32.717534 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675356    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 19:21:32.717534 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675358    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 19:21:32.717534 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675361    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 19:21:32.717534 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675363    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 19:21:32.717534 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675366    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 19:21:32.718561 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675369    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 19:21:32.718561 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675371    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 19:21:32.718561 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675374    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 19:21:32.718561 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675376    2574 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 19:21:32.718561 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675378    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 19:21:32.718561 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675381    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 19:21:32.718561 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675384    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 19:21:32.718561 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675386    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 19:21:32.718561 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675389    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 19:21:32.718561 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675391    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 19:21:32.718561 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675394    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 19:21:32.718561 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675397    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 19:21:32.718561 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675399    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 19:21:32.718561 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675402    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 19:21:32.718561 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675404    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 19:21:32.718561 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675407    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 19:21:32.718561 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675410    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 19:21:32.718561 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675413    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 19:21:32.718561 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675415    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 19:21:32.718561 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675421    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 19:21:32.719628 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675424    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 19:21:32.719628 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675427    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 19:21:32.719628 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675430    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 19:21:32.719628 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675432    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 19:21:32.719628 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675435    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 19:21:32.719628 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675438    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 19:21:32.719628 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675441    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 19:21:32.719628 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675443    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 19:21:32.719628 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675446    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 19:21:32.719628 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675449    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 19:21:32.719628 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675451    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 19:21:32.719628 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675455    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 19:21:32.719628 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675459    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 19:21:32.719628 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675461    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 19:21:32.719628 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675464    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 19:21:32.719628 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675837    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 19:21:32.719628 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675843    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 19:21:32.719628 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675846    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 19:21:32.719628 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675848    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 19:21:32.719628 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675851    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 19:21:32.720654 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675853    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 19:21:32.720654 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675857    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 19:21:32.720654 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675859    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 19:21:32.720654 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675862    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 19:21:32.720654 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675864    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 19:21:32.720654 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675867    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 19:21:32.720654 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675869    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 19:21:32.720654 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675872    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 19:21:32.720654 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675875    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 19:21:32.720654 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675880    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 19:21:32.720654 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675883    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 19:21:32.720654 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675886    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 19:21:32.720654 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675888    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 19:21:32.720654 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675891    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 19:21:32.720654 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675894    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 19:21:32.720654 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675896    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 19:21:32.720654 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675899    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 19:21:32.720654 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675902    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 19:21:32.720654 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675906    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 19:21:32.721729 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675909    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 19:21:32.721729 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675911    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 19:21:32.721729 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675914    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 19:21:32.721729 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675916    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 19:21:32.721729 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675919    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 19:21:32.721729 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675922    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 19:21:32.721729 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675924    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 19:21:32.721729 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675926    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 19:21:32.721729 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675929    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 19:21:32.721729 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675931    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 19:21:32.721729 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675934    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 19:21:32.721729 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675936    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 19:21:32.721729 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675939    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 19:21:32.721729 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675941    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 19:21:32.721729 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675944    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 19:21:32.721729 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675946    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 19:21:32.721729 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675949    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 19:21:32.721729 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675951    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 19:21:32.721729 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675953    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 19:21:32.721729 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675956    2574 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 19:21:32.722756 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675958    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 19:21:32.722756 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675961    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 19:21:32.722756 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675965    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 19:21:32.722756 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675968    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 19:21:32.722756 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675974    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 19:21:32.722756 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675978    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 19:21:32.722756 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675981    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 19:21:32.722756 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675984    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 19:21:32.722756 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675987    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 19:21:32.722756 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675990    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 19:21:32.722756 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675992    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 19:21:32.722756 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675995    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 19:21:32.722756 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.675998    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 19:21:32.722756 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676000    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 19:21:32.722756 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676004    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 19:21:32.722756 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676007    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 19:21:32.722756 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676010    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 19:21:32.722756 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676013    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 19:21:32.722756 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676016    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 19:21:32.723789 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676018    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 19:21:32.723789 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676021    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 19:21:32.723789 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676024    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 19:21:32.723789 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676027    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 19:21:32.723789 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676034    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 19:21:32.723789 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676037    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 19:21:32.723789 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676039    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 19:21:32.723789 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676042    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 19:21:32.723789 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676044    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 19:21:32.723789 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676047    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 19:21:32.723789 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676049    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 19:21:32.723789 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676052    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 19:21:32.723789 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676054    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 19:21:32.723789 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676057    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 19:21:32.723789 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676059    2574 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 19:21:32.723789 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676063    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 19:21:32.723789 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676066    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 19:21:32.723789 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676069    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 19:21:32.723789 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676072    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 19:21:32.723789 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676076    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 19:21:32.724827 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676078    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 19:21:32.724827 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676080    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 19:21:32.724827 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676083    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 19:21:32.724827 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676086    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 19:21:32.724827 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676088    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 19:21:32.724827 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676091    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 19:21:32.724827 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676093    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 19:21:32.724827 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676096    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 19:21:32.724827 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676099    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 19:21:32.724827 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676101    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 19:21:32.724827 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676104    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 19:21:32.724827 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676106    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 19:21:32.724827 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676109    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 19:21:32.724827 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676112    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 19:21:32.724827 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676114    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 19:21:32.724827 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676117    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 19:21:32.724827 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676181    2574 flags.go:64] FLAG: --address="0.0.0.0"
Jul 01 19:21:32.724827 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676188    2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 01 19:21:32.724827 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676199    2574 flags.go:64] FLAG: --anonymous-auth="true"
Jul 01 19:21:32.724827 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676204    2574 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 01 19:21:32.724827 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676207    2574 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 01 19:21:32.726383 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676210    2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 01 19:21:32.726383 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676214    2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 01 19:21:32.726383 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676218    2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 01 19:21:32.726383 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676221    2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 01 19:21:32.726383 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676224    2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 01 19:21:32.726383 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676228    2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 01 19:21:32.726383 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676232    2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 01 19:21:32.726383 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676235    2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 01 19:21:32.726383 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676239    2574 flags.go:64] FLAG: --cgroup-root=""
Jul 01 19:21:32.726383 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676242    2574 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 01 19:21:32.726383 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676245    2574 flags.go:64] FLAG: --client-ca-file=""
Jul 01 19:21:32.726383 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676254    2574 flags.go:64] FLAG: --cloud-provider="external"
Jul 01 19:21:32.726383 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676257    2574 flags.go:64] FLAG: --cluster-dns="[]"
Jul 01 19:21:32.726383 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676264    2574 flags.go:64] FLAG: --cluster-domain=""
Jul 01 19:21:32.726383 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676267    2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 01 19:21:32.726383 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676270    2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 01 19:21:32.726383 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676273    2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 01 19:21:32.726383 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676276    2574 flags.go:64] FLAG: --container-log-max-files="5"
Jul 01 19:21:32.726383 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676280    2574 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 01 19:21:32.726383 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676282    2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 01 19:21:32.726383 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676285    2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 01 19:21:32.726383 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676289    2574 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 01 19:21:32.726383 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676291    2574 flags.go:64] FLAG: --contention-profiling="false"
Jul 01 19:21:32.726383 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676294    2574 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 01 19:21:32.727195 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676297    2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 01 19:21:32.727195 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676300    2574 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 01 19:21:32.727195 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676303    2574 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 01 19:21:32.727195 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676307    2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 01 19:21:32.727195 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676310    2574 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 01 19:21:32.727195 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676313    2574 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 01 19:21:32.727195 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676317    2574 flags.go:64] FLAG: --enable-load-reader="false"
Jul 01 19:21:32.727195 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676319    2574 flags.go:64] FLAG: --enable-server="true"
Jul 01 19:21:32.727195 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676322    2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 01 19:21:32.727195 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676327    2574 flags.go:64] FLAG: --event-burst="100"
Jul 01 19:21:32.727195 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676329    2574 flags.go:64] FLAG: --event-qps="50"
Jul 01 19:21:32.727195 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676332    2574 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 01 19:21:32.727195 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676335    2574 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 01 19:21:32.727195 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676338    2574 flags.go:64] FLAG: --eviction-hard=""
Jul 01 19:21:32.727195 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676342    2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 01 19:21:32.727195 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676345    2574 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 01 19:21:32.727195 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676349    2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 01 19:21:32.727195 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676352    2574 flags.go:64] FLAG: --eviction-soft=""
Jul 01 19:21:32.727195 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676355    2574 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 01 19:21:32.727195 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676358    2574 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 01 19:21:32.727195 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676361    2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 01 19:21:32.727195 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676364    2574 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 01 19:21:32.727195 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676367    2574 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 01 19:21:32.727195 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676370    2574 flags.go:64] FLAG: --fail-swap-on="true"
Jul 01 19:21:32.727195 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676373    2574 flags.go:64] FLAG: --feature-gates=""
Jul 01 19:21:32.728300 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676376    2574 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 01 19:21:32.728300 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676379    2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 01 19:21:32.728300 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676382    2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 01 19:21:32.728300 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676385    2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 01 19:21:32.728300 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676388    2574 flags.go:64] FLAG: --healthz-port="10248"
Jul 01 19:21:32.728300 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676391    2574 flags.go:64] FLAG: --help="false"
Jul 01 19:21:32.728300 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676394    2574 flags.go:64] FLAG: --hostname-override="ip-10-0-137-164.ec2.internal"
Jul 01 19:21:32.728300 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676397    2574 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 01 19:21:32.728300 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676399    2574 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 01 19:21:32.728300 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676402    2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 01 19:21:32.728300 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676405    2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 01 19:21:32.728300 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676409    2574 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 01 19:21:32.728300 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676412    2574 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 01 19:21:32.728300 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676414    2574 flags.go:64] FLAG: --image-service-endpoint=""
Jul 01 19:21:32.728300 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676417    2574 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 01 19:21:32.728300 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676420    2574 flags.go:64] FLAG: --kube-api-burst="100"
Jul 01 19:21:32.728300 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676423    2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 01 19:21:32.728300 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676426    2574 flags.go:64] FLAG: --kube-api-qps="50"
Jul 01 19:21:32.728300 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676428    2574 flags.go:64] FLAG: --kube-reserved=""
Jul 01 19:21:32.728300 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676431    2574 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 01 19:21:32.728300 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676434    2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 01 19:21:32.728300 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676441    2574 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 01 19:21:32.728300 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676444    2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 01 19:21:32.728300 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676448    2574 flags.go:64] FLAG: --lock-file=""
Jul 01 19:21:32.729075 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676451    2574 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 01 19:21:32.729075 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676454    2574 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 01 19:21:32.729075 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676457    2574 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 01 19:21:32.729075 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676462    2574 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 01 19:21:32.729075 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676465    2574 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 01 19:21:32.729075 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676468    2574 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 01 19:21:32.729075 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676471    2574 flags.go:64] FLAG: --logging-format="text"
Jul 01 19:21:32.729075 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676474    2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 01 19:21:32.729075 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676477    2574 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 01 19:21:32.729075 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676480    2574 flags.go:64] FLAG: --manifest-url=""
Jul 01 19:21:32.729075 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676483    2574 flags.go:64] FLAG: --manifest-url-header=""
Jul 01 19:21:32.729075 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676487    2574 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 01 19:21:32.729075 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676490    2574 flags.go:64] FLAG: --max-open-files="1000000"
Jul 01 19:21:32.729075 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676494    2574 flags.go:64] FLAG: --max-pods="110"
Jul 01 19:21:32.729075 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676497    2574 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 01 19:21:32.729075 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676500    2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 01 19:21:32.729075 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676503    2574 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 01 19:21:32.729075 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676505    2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 01 19:21:32.729075 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676509    2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 01 19:21:32.729075 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676512    2574 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 01 19:21:32.729075 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676514    2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 01 19:21:32.729075 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676521    2574 flags.go:64] FLAG: --node-status-max-images="50"
Jul 01 19:21:32.729075 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676524    2574 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 01 19:21:32.729075 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676527    2574 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 01 19:21:32.729075 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676530    2574 flags.go:64] FLAG: --pod-cidr=""
Jul 01 19:21:32.933016 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676533    2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a2632cd31d78133be7b5fb6999469b2acf9b98d077c7f6ae5d0d94083d76b04"
Jul 01 19:21:32.933016 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676538    2574 flags.go:64] FLAG: --pod-manifest-path=""
Jul 01 19:21:32.933016 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676541    2574 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 01 19:21:32.933016 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676544    2574 flags.go:64] FLAG: --pods-per-core="0"
Jul 01 19:21:32.933016 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676548    2574 flags.go:64] FLAG: --port="10250"
Jul 01 19:21:32.933016 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676551    2574 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 01 19:21:32.933016 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676554    2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c4e6653d5d70d2c8"
Jul 01 19:21:32.933016 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676558    2574 flags.go:64] FLAG: --qos-reserved=""
Jul 01 19:21:32.933016 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676561    2574 flags.go:64] FLAG: --read-only-port="10255"
Jul 01 19:21:32.933016 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676564    2574 flags.go:64] FLAG: --register-node="true"
Jul 01 19:21:32.933016 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676567    2574 flags.go:64] FLAG: --register-with-taints=""
Jul 01 19:21:32.933016 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676571    2574 flags.go:64] FLAG: --registry-burst="10"
Jul 01 19:21:32.933016 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676574    2574 flags.go:64] FLAG: --registry-qps="5"
Jul 01 19:21:32.933016 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676577    2574 flags.go:64] FLAG: --reserved-cpus=""
Jul 01 19:21:32.933016 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676580    2574 flags.go:64] FLAG: --reserved-memory=""
Jul 01 19:21:32.933016 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676584    2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 01 19:21:32.933016 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676587    2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 01 19:21:32.933016 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676590    2574 flags.go:64] FLAG: --rotate-certificates="false"
Jul 01 19:21:32.933016 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676592    2574 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 01 19:21:32.933016 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676596    2574 flags.go:64] FLAG: --runonce="false"
Jul 01 19:21:32.933016 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676599    2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 01 19:21:32.933016 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676602    2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 01 19:21:32.933016 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676605    2574 flags.go:64] FLAG: --seccomp-default="false"
Jul 01 19:21:32.933016 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676607    2574 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 01 19:21:32.933016 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676610    2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 01 19:21:32.747751 ip-10-0-137-164 systemd[1]: Started Kubernetes Kubelet.
Jul 01 19:21:32.946232 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676613    2574 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 01 19:21:32.946232 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676616    2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 01 19:21:32.946232 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676619    2574 flags.go:64] FLAG: --storage-driver-password="root"
Jul 01 19:21:32.946232 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676622    2574 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 01 19:21:32.946232 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676625    2574 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 01 19:21:32.946232 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676628    2574 flags.go:64] FLAG: --storage-driver-user="root"
Jul 01 19:21:32.946232 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676630    2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 01 19:21:32.946232 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676634    2574 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 01 19:21:32.946232 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676636    2574 flags.go:64] FLAG: --system-cgroups=""
Jul 01 19:21:32.946232 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676639    2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 01 19:21:32.946232 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676644    2574 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 01 19:21:32.946232 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676648    2574 flags.go:64] FLAG: --tls-cert-file=""
Jul 01 19:21:32.946232 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676651    2574 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 01 19:21:32.946232 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676654    2574 flags.go:64] FLAG: --tls-min-version=""
Jul 01 19:21:32.946232 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676658    2574 flags.go:64] FLAG: --tls-private-key-file=""
Jul 01 19:21:32.946232 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676661    2574 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 01 19:21:32.946232 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676664    2574 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 01 19:21:32.946232 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676667    2574 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 01 19:21:32.946232 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676670    2574 flags.go:64] FLAG: --v="2"
Jul 01 19:21:32.946232 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676673    2574 flags.go:64] FLAG: --version="false"
Jul 01 19:21:32.946232 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676678    2574 flags.go:64] FLAG: --vmodule=""
Jul 01 19:21:32.946232 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676681    2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 01 19:21:32.946232 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.676684    2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 01 19:21:32.946232 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676836    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 19:21:32.946232 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676842    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 19:21:32.948178 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676845    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 19:21:32.948178 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676848    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 19:21:32.948178 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676851    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 19:21:32.948178 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676854    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 19:21:32.948178 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676858    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 19:21:32.948178 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676860    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 19:21:32.948178 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676863    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 19:21:32.948178 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676872    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 19:21:32.948178 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676875    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 19:21:32.948178 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676878    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 19:21:32.948178 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676881    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 19:21:32.948178 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676883    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 19:21:32.948178 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676886    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 19:21:32.948178 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676888    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 19:21:32.948178 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676891    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 19:21:32.948178 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676893    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 19:21:32.948178 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676896    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 19:21:32.948178 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676900    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 19:21:32.948178 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676905    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 19:21:32.949011 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676908    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 19:21:32.949011 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676911    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 19:21:32.949011 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676915    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 19:21:32.949011 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676918    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 19:21:32.949011 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676921    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 19:21:32.949011 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676924    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 19:21:32.949011 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676928    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 19:21:32.949011 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676931    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 19:21:32.949011 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676935    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 19:21:32.949011 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676938    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 19:21:32.949011 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676942    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 19:21:32.949011 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676944    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 19:21:32.949011 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676947    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 19:21:32.949011 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676950    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 19:21:32.949011 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676953    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 19:21:32.949011 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676955    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 19:21:32.949011 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676958    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 19:21:32.949011 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676960    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 19:21:32.949011 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676963    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 19:21:32.950619 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676966    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 19:21:32.950619 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676968    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 19:21:32.950619 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676971    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 19:21:32.950619 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676973    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 19:21:32.950619 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676976    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 19:21:32.950619 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676978    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 19:21:32.950619 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676981    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 19:21:32.950619 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676983    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 19:21:32.950619 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676986    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 19:21:32.950619 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676988    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 19:21:32.950619 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676991    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 19:21:32.950619 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676993    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 19:21:32.950619 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.676997    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 19:21:32.950619 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677000    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 19:21:32.950619 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677002    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 19:21:32.950619 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677007    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 19:21:32.950619 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677009    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 19:21:32.950619 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677012    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 19:21:32.950619 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677014    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 19:21:32.950619 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677017    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 19:21:32.951424 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677019    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 19:21:32.951424 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677022    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 19:21:32.951424 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677025    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 19:21:32.951424 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677028    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 19:21:32.951424 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677030    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 19:21:32.951424 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677033    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 19:21:32.951424 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677036    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 19:21:32.951424 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677038    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 19:21:32.951424 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677041    2574 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 19:21:32.951424 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677043    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 19:21:32.951424 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677046    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 19:21:32.951424 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677048    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 19:21:32.951424 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677050    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 19:21:32.951424 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677053    2574 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 19:21:32.951424 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677056    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 19:21:32.951424 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677058    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 19:21:32.951424 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677061    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 19:21:32.951424 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677063    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 19:21:32.951424 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677066    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 19:21:32.951424 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677068    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 19:21:32.953095 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677071    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 19:21:32.953095 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677073    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 19:21:32.953095 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677076    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 19:21:32.953095 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677078    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 19:21:32.953095 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677082    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 19:21:32.953095 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677085    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 19:21:32.953095 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677087    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 19:21:32.953095 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677091    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 19:21:32.953095 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677093    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 19:21:32.953095 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677095    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 19:21:32.953095 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677098    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 19:21:32.953095 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677101    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 19:21:32.953095 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677103    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 19:21:32.953095 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677106    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 19:21:32.953095 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677108    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 19:21:32.953095 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677112    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 19:21:32.953095 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677116    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 19:21:32.953095 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677119    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 19:21:32.953095 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.677121    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 19:21:32.953906 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.677683    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 19:21:32.953906 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.688579    2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 01 19:21:32.953906 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.688594    2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 01 19:21:32.953906 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.688613    2574 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 01 19:21:32.953906 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.688619    2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 01 19:21:32.953906 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688675    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 19:21:32.953906 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688680    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 19:21:32.953906 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688684    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 19:21:32.953906 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688687    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 19:21:32.953906 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688690    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 19:21:32.953906 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688693    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 19:21:32.953906 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688696    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 19:21:32.953906 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688699    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 19:21:32.953906 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688701    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 19:21:32.953906 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688704    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 19:21:32.955504 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688706    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 19:21:32.955504 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688726    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 19:21:32.955504 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688730    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 19:21:32.955504 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688735    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 19:21:32.955504 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688738    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 19:21:32.955504 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688742    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 19:21:32.955504 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688744    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 19:21:32.955504 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688747    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 19:21:32.955504 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688750    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 19:21:32.955504 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688753    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 19:21:32.955504 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688756    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 19:21:32.955504 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688758    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 19:21:32.955504 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688761    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 19:21:32.955504 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688763    2574 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 19:21:32.955504 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688767    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 19:21:32.955504 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688769    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 19:21:32.955504 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688772    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 19:21:32.955504 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688775    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 19:21:32.955504 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688777    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 19:21:32.955504 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688780    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 19:21:32.956357 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688782    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 19:21:32.956357 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688785    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 19:21:32.956357 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688788    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 19:21:32.956357 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688790    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 19:21:32.956357 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688793    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 19:21:32.956357 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688795    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 19:21:32.956357 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688798    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 19:21:32.956357 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688801    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 19:21:32.956357 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688803    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 19:21:32.956357 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688806    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 19:21:32.956357 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688808    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 19:21:32.956357 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688811    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 19:21:32.956357 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688813    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 19:21:32.956357 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688816    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 19:21:32.956357 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688818    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 19:21:32.956357 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688824    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 19:21:32.956357 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688829    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 19:21:32.956357 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688832    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 19:21:32.956357 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688836    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 19:21:32.958369 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688839    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 19:21:32.958369 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688842    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 19:21:32.958369 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688845    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 19:21:32.958369 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688847    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 19:21:32.958369 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688851    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 19:21:32.958369 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688855    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 19:21:32.958369 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688858    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 19:21:32.958369 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688862    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 19:21:32.958369 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688872    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 19:21:32.958369 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688877    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 19:21:32.958369 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688881    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 19:21:32.958369 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688884    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 19:21:32.958369 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688886    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 19:21:32.958369 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688889    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 19:21:32.958369 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688892    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 19:21:32.958369 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688894    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 19:21:32.958369 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688897    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 19:21:32.958369 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688900    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 19:21:32.958369 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688902    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 19:21:32.959172 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688905    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 19:21:32.959172 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688907    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 19:21:32.959172 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688911    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 19:21:32.959172 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688915    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 19:21:32.959172 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688918    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 19:21:32.959172 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688921    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 19:21:32.959172 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688924    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 19:21:32.959172 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688926    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 19:21:32.959172 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688929    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 19:21:32.959172 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688933    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 19:21:32.959172 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688936    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 19:21:32.959172 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688940    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 19:21:32.959172 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688942    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 19:21:32.959172 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688945    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 19:21:32.959172 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688947    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 19:21:32.959172 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688950    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 19:21:32.959172 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688953    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 19:21:32.959172 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688955    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 19:21:32.959172 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688958    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 19:21:32.960380 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688960    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 19:21:32.960380 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688964    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 19:21:32.960380 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688966    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 19:21:32.960380 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688969    2574 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 19:21:32.960380 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688971    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 19:21:32.960380 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688974    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 19:21:32.960380 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688976    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 19:21:32.960380 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688979    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 19:21:32.960380 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688981    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 19:21:32.960380 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688984    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 19:21:32.960380 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688987    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 19:21:32.960380 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.688989    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 19:21:32.960380 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.688994    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 19:21:32.960380 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689120    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 19:21:32.960380 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689126    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 19:21:32.961251 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689129    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 19:21:32.961251 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689132    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 19:21:32.961251 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689134    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 19:21:32.961251 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689137    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 19:21:32.961251 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689140    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 19:21:32.961251 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689142    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 19:21:32.961251 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689145    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 19:21:32.961251 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689148    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 19:21:32.961251 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689152    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 19:21:32.961251 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689154    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 19:21:32.961251 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689158    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 19:21:32.961251 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689160    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 19:21:32.961251 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689163    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 19:21:32.961251 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689165    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 19:21:32.961251 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689168    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 19:21:32.961251 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689170    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 19:21:32.961251 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689173    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 19:21:32.961251 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689175    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 19:21:32.961251 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689180    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 19:21:32.961251 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689182    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 19:21:32.961975 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689185    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 19:21:32.961975 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689188    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 19:21:32.961975 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689191    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 19:21:32.961975 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689196    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 19:21:32.961975 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689199    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 19:21:32.961975 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689203    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 19:21:32.961975 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689207    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 19:21:32.961975 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689211    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 19:21:32.961975 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689215    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 19:21:32.961975 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689218    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 19:21:32.961975 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689220    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 19:21:32.961975 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689223    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 19:21:32.961975 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689226    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 19:21:32.961975 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689230    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 19:21:32.961975 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689234    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 19:21:32.961975 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689236    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 19:21:32.961975 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689239    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 19:21:32.961975 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689241    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 19:21:32.961975 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689244    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 19:21:32.962818 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689246    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 19:21:32.962818 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689250    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 19:21:32.962818 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689252    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 19:21:32.962818 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689255    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 19:21:32.962818 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689259    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 19:21:32.962818 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689262    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 19:21:32.962818 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689264    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 19:21:32.962818 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689268    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 19:21:32.962818 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689271    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 19:21:32.962818 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689274    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 19:21:32.962818 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689277    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 19:21:32.962818 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689283    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 19:21:32.962818 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689286    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 19:21:32.962818 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689288    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 19:21:32.962818 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689291    2574 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 19:21:32.962818 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689294    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 19:21:32.962818 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689296    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 19:21:32.962818 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689299    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 19:21:32.962818 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689301    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 19:21:32.962818 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689304    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 19:21:32.963614 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689307    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 19:21:32.963614 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689309    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 19:21:32.963614 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689312    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 19:21:32.963614 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689314    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 19:21:32.963614 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689317    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 19:21:32.963614 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689320    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 19:21:32.963614 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689322    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 19:21:32.963614 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689325    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 19:21:32.963614 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689328    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 19:21:32.963614 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689332    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 19:21:32.963614 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689335    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 19:21:32.963614 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689338    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 19:21:32.963614 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689341    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 19:21:32.963614 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689345    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 19:21:32.963614 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689347    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 19:21:32.963614 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689350    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 19:21:32.963614 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689353    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 19:21:32.963614 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689356    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 19:21:32.963614 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689359    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 19:21:32.964450 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689361    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 19:21:32.964450 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689364    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 19:21:32.964450 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689367    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 19:21:32.964450 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689369    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 19:21:32.964450 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689373    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 19:21:32.964450 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689375    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 19:21:32.964450 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689378    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 19:21:32.964450 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689380    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 19:21:32.964450 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689383    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 19:21:32.964450 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689385    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 19:21:32.964450 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689388    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 19:21:32.964450 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689390    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 19:21:32.964450 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689393    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 19:21:32.964450 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689396    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 19:21:32.964450 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689398    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 19:21:32.964450 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689401    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 19:21:32.964450 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689403    2574 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 19:21:32.964450 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689406    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 19:21:32.964450 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:32.689409    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 19:21:32.965231 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.689413    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 19:21:32.965231 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.689509    2574 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 01 19:21:32.965231 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.692793    2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 01 19:21:32.965231 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.694023    2574 server.go:1019] "Starting client certificate rotation"
Jul 01 19:21:32.965231 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.694121    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 01 19:21:32.965231 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.694156    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 01 19:21:32.965231 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.696360    2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 01 19:21:32.965231 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.698391    2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 01 19:21:32.965231 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.710735    2574 log.go:25] "Validated CRI v1 runtime API"
Jul 01 19:21:32.965231 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.712756    2574 log.go:25] "Validated CRI v1 image API"
Jul 01 19:21:32.965231 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.714382    2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 01 19:21:32.965231 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.717734    2574 fs.go:135] Filesystem UUIDs: map[053b7c03-0cd5-4ced-a692-52c2e645201f:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 a1e601f3-f587-40c3-aef2-0814b1ba745a:/dev/nvme0n1p4]
Jul 01 19:21:32.965679 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.717755    2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 01 19:21:32.969307 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.724742    2574 manager.go:216] Machine: {Timestamp:2026-07-01 19:21:32.723895057 +0000 UTC m=+0.333081338 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102783 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28d8a9991298a48769306aeefd34f8 SystemUUID:ec28d8a9-9912-98a4-8769-306aeefd34f8 BootID:3b4f6197-0456-4223-b3b9-42aa5337d1f1 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4d:61:de:7b:03 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4d:61:de:7b:03 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:da:69:be:f4:e2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 01 19:21:32.969307 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.724894    2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 01 19:21:32.969307 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.725083    2574 manager.go:232] Version: {KernelVersion:5.14.0-570.124.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260623-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 01 19:21:32.969307 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.727529    2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 01 19:21:32.969307 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.727567    2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-164.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 01 19:21:32.969307 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.728350    2574 topology_manager.go:138] "Creating topology manager with none policy"
Jul 01 19:21:32.969307 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.728367    2574 container_manager_linux.go:309] "Creating device plugin manager"
Jul 01 19:21:32.969307 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.728378    2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 01 19:21:32.969730 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.728783    2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 01 19:21:32.969730 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.729521    2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 01 19:21:32.969730 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.732007    2574 state_mem.go:36] "Initialized new in-memory state store"
Jul 01 19:21:32.969730 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.732429    2574 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 01 19:21:32.969730 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.733088    2574 kubelet.go:486] "Attempting to sync node with API server"
Jul 01 19:21:32.969730 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.733117    2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 01 19:21:32.969730 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.733139    2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 01 19:21:32.969730 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.733221    2574 kubelet.go:398] "Adding apiserver pod source"
Jul 01 19:21:32.969730 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.733234    2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 01 19:21:32.969730 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.735337    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 01 19:21:32.969730 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.735366    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 01 19:21:32.969730 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.737521    2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 01 19:21:32.969730 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.737810    2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 01 19:21:32.969730 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.738912    2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 01 19:21:32.969730 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.738939    2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 01 19:21:32.969730 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.739462    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 01 19:21:32.969730 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.739480    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 01 19:21:32.969730 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.739490    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 01 19:21:32.969730 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.739495    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 01 19:21:32.969730 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.739504    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 01 19:21:32.969730 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.739509    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 01 19:21:32.969730 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.739518    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 01 19:21:32.969730 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.739523    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 01 19:21:32.969730 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.739529    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 01 19:21:32.969730 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.739534    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 01 19:21:32.969730 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.739542    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 01 19:21:32.969730 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.739548    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 01 19:21:32.969730 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.740887    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 01 19:21:32.969730 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.743529    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 01 19:21:32.969730 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.746900    2574 server.go:1268] "Started kubelet"
Jul 01 19:21:32.969730 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:32.747465    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-164.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 01 19:21:32.969730 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:32.747549    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 01 19:21:32.969730 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.747549    2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 01 19:21:32.969730 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.747618    2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-164.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 01 19:21:32.969730 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.747626    2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 01 19:21:32.969730 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.747690    2574 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 01 19:21:32.969730 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.749417    2574 server.go:310] "Adding debug handlers to kubelet server"
Jul 01 19:21:32.969730 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.751064    2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 01 19:21:32.969730 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:32.754799    2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 01 19:21:32.970928 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:32.755244    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-164.ec2.internal.18be40bf4640d7d2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-164.ec2.internal,UID:ip-10-0-137-164.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-164.ec2.internal,},FirstTimestamp:2026-07-01 19:21:32.746880978 +0000 UTC m=+0.356067233,LastTimestamp:2026-07-01 19:21:32.746880978 +0000 UTC m=+0.356067233,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-164.ec2.internal,}"
Jul 01 19:21:32.970928 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.758039    2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 01 19:21:32.970928 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.758051    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 01 19:21:32.970928 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:32.758136    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-164.ec2.internal\" not found"
Jul 01 19:21:32.970928 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.758510    2574 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 01 19:21:32.970928 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.758525    2574 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 01 19:21:32.970928 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.758513    2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 01 19:21:32.970928 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.758613    2574 reconstruct.go:97] "Volume reconstruction finished"
Jul 01 19:21:32.970928 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.758624    2574 reconciler.go:29] "Reconciler: start to sync state"
Jul 01 19:21:32.970928 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.759073    2574 factory.go:55] Registering systemd factory
Jul 01 19:21:32.970928 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.759084    2574 factory.go:223] Registration of the systemd container factory successfully
Jul 01 19:21:32.970928 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.759295    2574 factory.go:153] Registering CRI-O factory
Jul 01 19:21:32.970928 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.759311    2574 factory.go:223] Registration of the crio container factory successfully
Jul 01 19:21:32.970928 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.759360    2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 01 19:21:32.970928 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.759385    2574 factory.go:103] Registering Raw factory
Jul 01 19:21:32.970928 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.759399    2574 manager.go:1196] Started watching for new ooms in manager
Jul 01 19:21:32.970928 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.759864    2574 manager.go:349] Starting recovery of all containers
Jul 01 19:21:32.970928 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:32.763994    2574 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-164.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 01 19:21:32.970928 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:32.764112    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 01 19:21:32.970928 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.775514    2574 manager.go:354] Recovery completed
Jul 01 19:21:32.970928 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.779404    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 19:21:32.970928 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.781215    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-164.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 19:21:32.970928 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.781248    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-164.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 19:21:32.970928 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.781258    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-164.ec2.internal" event="NodeHasSufficientPID"
Jul 01 19:21:32.970928 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.781795    2574 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 01 19:21:32.970928 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.781804    2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 01 19:21:32.970928 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.781818    2574 state_mem.go:36] "Initialized new in-memory state store"
Jul 01 19:21:32.970928 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.781851    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 19:21:32.970928 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.782465    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-164.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 19:21:32.970928 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.782487    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-164.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 19:21:32.970928 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.782497    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-164.ec2.internal" event="NodeHasSufficientPID"
Jul 01 19:21:32.970928 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.782817    2574 policy_none.go:49] "None policy: Start"
Jul 01 19:21:32.970928 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.782828    2574 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 01 19:21:32.970928 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.782838    2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 01 19:21:32.972990 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:32.783477    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-164.ec2.internal.18be40bf484d0b43  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-164.ec2.internal,UID:ip-10-0-137-164.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-164.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-164.ec2.internal,},FirstTimestamp:2026-07-01 19:21:32.781235011 +0000 UTC m=+0.390421269,LastTimestamp:2026-07-01 19:21:32.781235011 +0000 UTC m=+0.390421269,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-164.ec2.internal,}"
Jul 01 19:21:32.972990 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.783665    2574 policy_none.go:47] "Start"
Jul 01 19:21:32.972990 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:32.795328    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-164.ec2.internal.18be40bf484d50cf  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-164.ec2.internal,UID:ip-10-0-137-164.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-164.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-164.ec2.internal,},FirstTimestamp:2026-07-01 19:21:32.781252815 +0000 UTC m=+0.390439069,LastTimestamp:2026-07-01 19:21:32.781252815 +0000 UTC m=+0.390439069,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-164.ec2.internal,}"
Jul 01 19:21:32.972990 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:32.815651    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-164.ec2.internal.18be40bf484d759b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-164.ec2.internal,UID:ip-10-0-137-164.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-137-164.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-137-164.ec2.internal,},FirstTimestamp:2026-07-01 19:21:32.781262235 +0000 UTC m=+0.390448489,LastTimestamp:2026-07-01 19:21:32.781262235 +0000 UTC m=+0.390448489,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-164.ec2.internal,}"
Jul 01 19:21:32.972990 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:32.827917    2574 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-164.ec2.internal.18be40bf484d0b43\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-164.ec2.internal.18be40bf484d0b43  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-164.ec2.internal,UID:ip-10-0-137-164.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-164.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-164.ec2.internal,},FirstTimestamp:2026-07-01 19:21:32.781235011 +0000 UTC m=+0.390421269,LastTimestamp:2026-07-01 19:21:32.782475861 +0000 UTC m=+0.391662116,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-164.ec2.internal,}"
Jul 01 19:21:32.972990 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.831646    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 01 19:21:32.972990 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.831977    2574 manager.go:337] "Starting Device Plugin manager"
Jul 01 19:21:32.972990 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:32.832008    2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 01 19:21:32.972990 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.832019    2574 server.go:87] "Starting device plugin registration server"
Jul 01 19:21:32.972990 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.832289    2574 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 01 19:21:32.972990 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.832302    2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 01 19:21:32.972990 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.832402    2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 01 19:21:32.972990 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.832479    2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 01 19:21:32.972990 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.832488    2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 01 19:21:32.973786 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.833033    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 01 19:21:32.973786 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.833083    2574 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 01 19:21:32.973786 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.833106    2574 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 01 19:21:32.973786 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:32.833140    2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 01 19:21:32.973786 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:32.833198    2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-164.ec2.internal\" not found"
Jul 01 19:21:32.973786 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:32.833148    2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 01 19:21:32.973786 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:32.840974    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Jul 01 19:21:32.973786 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:32.841277    2574 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-164.ec2.internal.18be40bf484d50cf\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-164.ec2.internal.18be40bf484d50cf  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-164.ec2.internal,UID:ip-10-0-137-164.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-164.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-164.ec2.internal,},FirstTimestamp:2026-07-01 19:21:32.781252815 +0000 UTC m=+0.390439069,LastTimestamp:2026-07-01 19:21:32.782492104 +0000 UTC m=+0.391678359,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-164.ec2.internal,}"
Jul 01 19:21:32.973786 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:32.842791    2574 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-164.ec2.internal.18be40bf484d759b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-164.ec2.internal.18be40bf484d759b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-164.ec2.internal,UID:ip-10-0-137-164.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-137-164.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-137-164.ec2.internal,},FirstTimestamp:2026-07-01 19:21:32.781262235 +0000 UTC m=+0.390448489,LastTimestamp:2026-07-01 19:21:32.782501126 +0000 UTC m=+0.391687381,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-164.ec2.internal,}"
Jul 01 19:21:32.973786 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.850320    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z5jrt"
Jul 01 19:21:32.973786 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:32.853338    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-164.ec2.internal.18be40bf4b80745d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-164.ec2.internal,UID:ip-10-0-137-164.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-137-164.ec2.internal,},FirstTimestamp:2026-07-01 19:21:32.834935901 +0000 UTC m=+0.444122144,LastTimestamp:2026-07-01 19:21:32.834935901 +0000 UTC m=+0.444122144,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-164.ec2.internal,}"
Jul 01 19:21:32.973786 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.858829    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z5jrt"
Jul 01 19:21:32.973786 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.933178    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 19:21:32.973786 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.933377    2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-164.ec2.internal"]
Jul 01 19:21:32.973786 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.933458    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 19:21:32.974584 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.934756    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-164.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 19:21:32.974584 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.934781    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-164.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 19:21:32.974584 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.934798    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-164.ec2.internal" event="NodeHasSufficientPID"
Jul 01 19:21:32.974584 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.934825    2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-164.ec2.internal"
Jul 01 19:21:32.974584 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.935200    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-164.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 19:21:32.974584 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.935225    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-164.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 19:21:32.974584 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.935235    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-164.ec2.internal" event="NodeHasSufficientPID"
Jul 01 19:21:32.974584 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.936460    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 19:21:32.974584 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.936579    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.ec2.internal"
Jul 01 19:21:32.974584 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.936615    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 19:21:32.974584 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.937135    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-164.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 19:21:32.974584 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.937152    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-164.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 19:21:32.974584 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.937164    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-164.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 19:21:32.974584 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.937171    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-164.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 19:21:32.974584 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.937175    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-164.ec2.internal" event="NodeHasSufficientPID"
Jul 01 19:21:32.974584 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.937181    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-164.ec2.internal" event="NodeHasSufficientPID"
Jul 01 19:21:32.974584 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.938709    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-164.ec2.internal"
Jul 01 19:21:32.974584 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.938756    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 19:21:32.974584 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.939388    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-164.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 19:21:32.974584 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.939423    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-164.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 19:21:32.974584 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.939439    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-164.ec2.internal" event="NodeHasSufficientPID"
Jul 01 19:21:32.974584 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:32.953532    2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-164.ec2.internal"
Jul 01 19:21:32.974584 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:32.953549    2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-164.ec2.internal\": node \"ip-10-0-137-164.ec2.internal\" not found"
Jul 01 19:21:33.078881 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:32.987276    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-164.ec2.internal\" not found"
Jul 01 19:21:33.078881 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:33.059874    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9e3a82d86892a337e8c834318f52d910-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-164.ec2.internal\" (UID: \"9e3a82d86892a337e8c834318f52d910\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.ec2.internal"
Jul 01 19:21:33.078881 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:33.059899    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e3a82d86892a337e8c834318f52d910-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-164.ec2.internal\" (UID: \"9e3a82d86892a337e8c834318f52d910\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.ec2.internal"
Jul 01 19:21:33.078881 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:33.059917    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f180425c803edc40a93bf63fc1f451d4-config\") pod \"kube-apiserver-proxy-ip-10-0-137-164.ec2.internal\" (UID: \"f180425c803edc40a93bf63fc1f451d4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-164.ec2.internal"
Jul 01 19:21:33.089805 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:33.087580    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-164.ec2.internal\" not found"
Jul 01 19:21:33.201529 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:33.160860    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9e3a82d86892a337e8c834318f52d910-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-164.ec2.internal\" (UID: \"9e3a82d86892a337e8c834318f52d910\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.ec2.internal"
Jul 01 19:21:33.201529 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:33.160897    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e3a82d86892a337e8c834318f52d910-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-164.ec2.internal\" (UID: \"9e3a82d86892a337e8c834318f52d910\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.ec2.internal"
Jul 01 19:21:33.201529 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:33.160922    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f180425c803edc40a93bf63fc1f451d4-config\") pod \"kube-apiserver-proxy-ip-10-0-137-164.ec2.internal\" (UID: \"f180425c803edc40a93bf63fc1f451d4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-164.ec2.internal"
Jul 01 19:21:33.201529 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:33.160955    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f180425c803edc40a93bf63fc1f451d4-config\") pod \"kube-apiserver-proxy-ip-10-0-137-164.ec2.internal\" (UID: \"f180425c803edc40a93bf63fc1f451d4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-164.ec2.internal"
Jul 01 19:21:33.201529 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:33.160956    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e3a82d86892a337e8c834318f52d910-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-164.ec2.internal\" (UID: \"9e3a82d86892a337e8c834318f52d910\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.ec2.internal"
Jul 01 19:21:33.201529 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:33.160956    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9e3a82d86892a337e8c834318f52d910-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-164.ec2.internal\" (UID: \"9e3a82d86892a337e8c834318f52d910\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.ec2.internal"
Jul 01 19:21:33.201529 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:33.188388    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-164.ec2.internal\" not found"
Jul 01 19:21:33.327853 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:33.289005    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-164.ec2.internal\" not found"
Jul 01 19:21:33.449723 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:33.389477    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-164.ec2.internal\" not found"
Jul 01 19:21:33.560944 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:33.489960    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-164.ec2.internal\" not found"
Jul 01 19:21:33.670218 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:33.590415    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-164.ec2.internal\" not found"
Jul 01 19:21:33.670218 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:33.669059    2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 19:21:33.690934 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:33.690911    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-164.ec2.internal\" not found"
Jul 01 19:21:33.694077 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:33.694061    2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 01 19:21:33.694219 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:33.694200    2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 01 19:21:33.700791 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:33.700773    2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-164.ec2.internal\" not found" node="ip-10-0-137-164.ec2.internal"
Jul 01 19:21:33.703106 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:33.703092    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.ec2.internal"
Jul 01 19:21:33.705514 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:33.705498    2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-164.ec2.internal\" not found" node="ip-10-0-137-164.ec2.internal"
Jul 01 19:21:33.707367 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:33.707355    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-164.ec2.internal"
Jul 01 19:21:33.791061 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:33.791033    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-164.ec2.internal\" not found"
Jul 01 19:21:33.862011 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:33.861930    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-06-30 19:16:34 +0000 UTC" deadline="2028-03-30 14:33:40.06833182 +0000 UTC"
Jul 01 19:21:33.862011 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:33.861958    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15307h12m6.206376454s"
Jul 01 19:21:33.891236 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:33.891215    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-164.ec2.internal\" not found"
Jul 01 19:21:33.920390 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:33.920367    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 19:21:33.923012 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:33.922996    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-164.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 19:21:33.923083 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:33.923026    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-164.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 19:21:33.923083 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:33.923037    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-164.ec2.internal" event="NodeHasSufficientPID"
Jul 01 19:21:33.946376 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:33.946355    2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 19:21:33.976727 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:33.976682    2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 19:21:33.991758 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:33.991734    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-164.ec2.internal\" not found"
Jul 01 19:21:34.066484 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:34.066457    2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 19:21:34.158544 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:34.158474    2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.ec2.internal"
Jul 01 19:21:34.169246 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:34.169224    2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 01 19:21:34.170389 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:34.170374    2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-164.ec2.internal"
Jul 01 19:21:34.180296 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:34.180276    2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 01 19:21:35.888355 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:35.888321    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf180425c803edc40a93bf63fc1f451d4.slice/crio-5425474e3cd4e7ff798eaf635d08daa972eca3a4a195832ddd7651db52742a89 WatchSource:0}: Error finding container 5425474e3cd4e7ff798eaf635d08daa972eca3a4a195832ddd7651db52742a89: Status 404 returned error can't find the container with id 5425474e3cd4e7ff798eaf635d08daa972eca3a4a195832ddd7651db52742a89
Jul 01 19:21:35.888911 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:35.888871    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e3a82d86892a337e8c834318f52d910.slice/crio-093678378cba1758eaa8e9c50940c34a18e323751ef9bdea84b72ce77a2c0bd2 WatchSource:0}: Error finding container 093678378cba1758eaa8e9c50940c34a18e323751ef9bdea84b72ce77a2c0bd2: Status 404 returned error can't find the container with id 093678378cba1758eaa8e9c50940c34a18e323751ef9bdea84b72ce77a2c0bd2
Jul 01 19:21:35.892847 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:35.892829    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 19:21:36.253713 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.253688    2574 apiserver.go:52] "Watching apiserver"
Jul 01 19:21:36.259376 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.259356    2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 01 19:21:36.259724 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.259702    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-7g8pr","openshift-network-operator/iptables-alerter-xtknv","openshift-ovn-kubernetes/ovnkube-node-bplsm","kube-system/global-pull-secret-syncer-mq22h","kube-system/konnectivity-agent-mch4p","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5qlm2","openshift-cluster-node-tuning-operator/tuned-nj5ft","openshift-dns/node-resolver-p58v4","openshift-image-registry/node-ca-kjtgf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.ec2.internal","openshift-multus/multus-additional-cni-plugins-hnpkl","kube-system/kube-apiserver-proxy-ip-10-0-137-164.ec2.internal","openshift-multus/multus-txnzp","openshift-multus/network-metrics-daemon-m6lm4"]
Jul 01 19:21:36.261789 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.261770    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.262807 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.262791    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.263924 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.263886    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.264810 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.264790    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 01 19:21:36.264964 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.264950    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:21:36.265074 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:36.265044    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq22h" podUID="f9eaa621-a5fa-49a2-ad92-8dcd5729803a"
Jul 01 19:21:36.265208 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.265192    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fql52\""
Jul 01 19:21:36.265565 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.265552    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 01 19:21:36.266063 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.266040    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mch4p"
Jul 01 19:21:36.266432 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.266411    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 01 19:21:36.266543 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.266462    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 01 19:21:36.266543 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.266462    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 01 19:21:36.266543 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.266509    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 01 19:21:36.266725 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.266641    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5jc2c\""
Jul 01 19:21:36.266884 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.266870    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 01 19:21:36.267246 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.267231    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5qlm2"
Jul 01 19:21:36.267757 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.267739    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 01 19:21:36.269742 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.268042    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 01 19:21:36.269742 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.268242    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 01 19:21:36.269742 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.268417    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pb56k\""
Jul 01 19:21:36.270992 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.270430    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 01 19:21:36.270992 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.270477    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4kfc9\""
Jul 01 19:21:36.270992 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.270575    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 01 19:21:36.270992 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.270668    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 01 19:21:36.270992 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.270727    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 01 19:21:36.270992 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.270777    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 01 19:21:36.270992 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.270780    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 01 19:21:36.270992 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.270922    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-f4lqk\""
Jul 01 19:21:36.271396 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.271173    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 01 19:21:36.271449 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.271393    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kjtgf"
Jul 01 19:21:36.273140 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.273124    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g8pr"
Jul 01 19:21:36.273215 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:36.273178    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7g8pr" podUID="9e551421-5f11-40eb-8a22-e594b6f4eb61"
Jul 01 19:21:36.273817 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.273800    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 01 19:21:36.273895 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.273855    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 01 19:21:36.273895 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.273889    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 01 19:21:36.274222 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.274205    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-99cmd\""
Jul 01 19:21:36.274288 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.274229    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xtknv"
Jul 01 19:21:36.275510 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.275492    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6lm4"
Jul 01 19:21:36.275607 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:36.275548    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6lm4" podUID="98d86579-8161-4185-8cf7-5f427d712b2e"
Jul 01 19:21:36.275746 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.275728    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 01 19:21:36.276758 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.276741    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hnpkl"
Jul 01 19:21:36.278062 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.278040    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-p58v4"
Jul 01 19:21:36.278229 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.278208    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nvhzg\""
Jul 01 19:21:36.278485 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.278462    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 01 19:21:36.278570 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.278517    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 01 19:21:36.278766 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.278753    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 01 19:21:36.279441 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.279422    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 01 19:21:36.279570 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.279557    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 01 19:21:36.279649 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.279636    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wh5lp\""
Jul 01 19:21:36.280697 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.280680    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 01 19:21:36.280776 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.280680    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lc5ws\""
Jul 01 19:21:36.280829 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.280773    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 01 19:21:36.286152 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.286136    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1673e32e-d2ee-445b-95d6-144e138343df-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hnpkl\" (UID: \"1673e32e-d2ee-445b-95d6-144e138343df\") " pod="openshift-multus/multus-additional-cni-plugins-hnpkl"
Jul 01 19:21:36.286229 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.286159    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb05bf30-f19a-4b15-af14-30bf33f28fa9-etc-kubernetes\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.286229 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.286174    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-etc-openvswitch\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.286229 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.286192    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-host-run-ovn-kubernetes\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.286229 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.286210    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/688e64ef-7029-47fe-bd40-369dcd4e2948-socket-dir\") pod \"aws-ebs-csi-driver-node-5qlm2\" (UID: \"688e64ef-7029-47fe-bd40-369dcd4e2948\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5qlm2"
Jul 01 19:21:36.286229 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.286226    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1673e32e-d2ee-445b-95d6-144e138343df-system-cni-dir\") pod \"multus-additional-cni-plugins-hnpkl\" (UID: \"1673e32e-d2ee-445b-95d6-144e138343df\") " pod="openshift-multus/multus-additional-cni-plugins-hnpkl"
Jul 01 19:21:36.286407 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.286251    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb05bf30-f19a-4b15-af14-30bf33f28fa9-host-var-lib-kubelet\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.286407 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.286286    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-run-systemd\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.286407 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.286316    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.286407 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.286334    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bfb845e6-341d-492e-91c9-9c698a0b8d29-ovnkube-script-lib\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.286407 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.286354    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9b5d6712-c210-43d7-ae23-6fe6bb85ef6c-agent-certs\") pod \"konnectivity-agent-mch4p\" (UID: \"9b5d6712-c210-43d7-ae23-6fe6bb85ef6c\") " pod="kube-system/konnectivity-agent-mch4p"
Jul 01 19:21:36.286407 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.286368    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/846ea9a9-80a2-4628-a2da-e3224610bf57-serviceca\") pod \"node-ca-kjtgf\" (UID: \"846ea9a9-80a2-4628-a2da-e3224610bf57\") " pod="openshift-image-registry/node-ca-kjtgf"
Jul 01 19:21:36.286407 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.286400    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e96a778c-46ff-4720-a550-499d2f39ff83-etc-sysctl-d\") pod \"tuned-nj5ft\" (UID: \"e96a778c-46ff-4720-a550-499d2f39ff83\") " pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.286729 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.286458    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-host-cni-bin\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.286729 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.286489    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4vwt\" (UniqueName: \"kubernetes.io/projected/846ea9a9-80a2-4628-a2da-e3224610bf57-kube-api-access-d4vwt\") pod \"node-ca-kjtgf\" (UID: \"846ea9a9-80a2-4628-a2da-e3224610bf57\") " pod="openshift-image-registry/node-ca-kjtgf"
Jul 01 19:21:36.286729 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.286515    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e96a778c-46ff-4720-a550-499d2f39ff83-etc-sysctl-conf\") pod \"tuned-nj5ft\" (UID: \"e96a778c-46ff-4720-a550-499d2f39ff83\") " pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.286729 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.286550    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e96a778c-46ff-4720-a550-499d2f39ff83-lib-modules\") pod \"tuned-nj5ft\" (UID: \"e96a778c-46ff-4720-a550-499d2f39ff83\") " pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.286729 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.286573    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e96a778c-46ff-4720-a550-499d2f39ff83-host\") pod \"tuned-nj5ft\" (UID: \"e96a778c-46ff-4720-a550-499d2f39ff83\") " pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.286729 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.286595    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/688e64ef-7029-47fe-bd40-369dcd4e2948-sys-fs\") pod \"aws-ebs-csi-driver-node-5qlm2\" (UID: \"688e64ef-7029-47fe-bd40-369dcd4e2948\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5qlm2"
Jul 01 19:21:36.286729 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.286618    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bb05bf30-f19a-4b15-af14-30bf33f28fa9-hostroot\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.286729 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.286641    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvhwt\" (UniqueName: \"kubernetes.io/projected/bb05bf30-f19a-4b15-af14-30bf33f28fa9-kube-api-access-kvhwt\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.286729 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.286689    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-host-slash\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.286729 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.286712    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-host-run-netns\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.287131 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.286737    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb05bf30-f19a-4b15-af14-30bf33f28fa9-host-run-netns\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.287131 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.286760    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-host-kubelet\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.287131 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.286783    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-run-ovn\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.287131 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.286821    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/688e64ef-7029-47fe-bd40-369dcd4e2948-registration-dir\") pod \"aws-ebs-csi-driver-node-5qlm2\" (UID: \"688e64ef-7029-47fe-bd40-369dcd4e2948\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5qlm2"
Jul 01 19:21:36.287131 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.286857    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vl29\" (UniqueName: \"kubernetes.io/projected/688e64ef-7029-47fe-bd40-369dcd4e2948-kube-api-access-8vl29\") pod \"aws-ebs-csi-driver-node-5qlm2\" (UID: \"688e64ef-7029-47fe-bd40-369dcd4e2948\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5qlm2"
Jul 01 19:21:36.287131 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.286885    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bb05bf30-f19a-4b15-af14-30bf33f28fa9-host-run-k8s-cni-cncf-io\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.287131 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.286922    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1673e32e-d2ee-445b-95d6-144e138343df-cnibin\") pod \"multus-additional-cni-plugins-hnpkl\" (UID: \"1673e32e-d2ee-445b-95d6-144e138343df\") " pod="openshift-multus/multus-additional-cni-plugins-hnpkl"
Jul 01 19:21:36.287131 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.286940    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e96a778c-46ff-4720-a550-499d2f39ff83-etc-kubernetes\") pod \"tuned-nj5ft\" (UID: \"e96a778c-46ff-4720-a550-499d2f39ff83\") " pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.287131 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.286965    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e96a778c-46ff-4720-a550-499d2f39ff83-run\") pod \"tuned-nj5ft\" (UID: \"e96a778c-46ff-4720-a550-499d2f39ff83\") " pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.287131 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.286989    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9eaa621-a5fa-49a2-ad92-8dcd5729803a-original-pull-secret\") pod \"global-pull-secret-syncer-mq22h\" (UID: \"f9eaa621-a5fa-49a2-ad92-8dcd5729803a\") " pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:21:36.287131 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.287023    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j6qv\" (UniqueName: \"kubernetes.io/projected/98d86579-8161-4185-8cf7-5f427d712b2e-kube-api-access-4j6qv\") pod \"network-metrics-daemon-m6lm4\" (UID: \"98d86579-8161-4185-8cf7-5f427d712b2e\") " pod="openshift-multus/network-metrics-daemon-m6lm4"
Jul 01 19:21:36.287131 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.287057    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/190cde15-2eee-4b40-b4e1-c7cad1331778-iptables-alerter-script\") pod \"iptables-alerter-xtknv\" (UID: \"190cde15-2eee-4b40-b4e1-c7cad1331778\") " pod="openshift-network-operator/iptables-alerter-xtknv"
Jul 01 19:21:36.287131 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.287082    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e96a778c-46ff-4720-a550-499d2f39ff83-etc-sysconfig\") pod \"tuned-nj5ft\" (UID: \"e96a778c-46ff-4720-a550-499d2f39ff83\") " pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.287788 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.287104    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e96a778c-46ff-4720-a550-499d2f39ff83-etc-systemd\") pod \"tuned-nj5ft\" (UID: \"e96a778c-46ff-4720-a550-499d2f39ff83\") " pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.287788 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.287123    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bb05bf30-f19a-4b15-af14-30bf33f28fa9-multus-daemon-config\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.287788 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.287140    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4f8d\" (UniqueName: \"kubernetes.io/projected/bfb845e6-341d-492e-91c9-9c698a0b8d29-kube-api-access-h4f8d\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.287788 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.287160    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e96a778c-46ff-4720-a550-499d2f39ff83-var-lib-kubelet\") pod \"tuned-nj5ft\" (UID: \"e96a778c-46ff-4720-a550-499d2f39ff83\") " pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.287788 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.287184    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e96a778c-46ff-4720-a550-499d2f39ff83-tmp\") pod \"tuned-nj5ft\" (UID: \"e96a778c-46ff-4720-a550-499d2f39ff83\") " pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.287788 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.287205    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/688e64ef-7029-47fe-bd40-369dcd4e2948-device-dir\") pod \"aws-ebs-csi-driver-node-5qlm2\" (UID: \"688e64ef-7029-47fe-bd40-369dcd4e2948\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5qlm2"
Jul 01 19:21:36.287788 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.287222    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb05bf30-f19a-4b15-af14-30bf33f28fa9-multus-cni-dir\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.287788 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.287255    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-host-cni-netd\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.287788 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.287283    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bfb845e6-341d-492e-91c9-9c698a0b8d29-env-overrides\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.287788 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.287308    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bfb845e6-341d-492e-91c9-9c698a0b8d29-ovn-node-metrics-cert\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.287788 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.287333    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1673e32e-d2ee-445b-95d6-144e138343df-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hnpkl\" (UID: \"1673e32e-d2ee-445b-95d6-144e138343df\") " pod="openshift-multus/multus-additional-cni-plugins-hnpkl"
Jul 01 19:21:36.287788 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.287359    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-run-openvswitch\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.287788 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.287395    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-node-log\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.287788 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.287419    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/190cde15-2eee-4b40-b4e1-c7cad1331778-host-slash\") pod \"iptables-alerter-xtknv\" (UID: \"190cde15-2eee-4b40-b4e1-c7cad1331778\") " pod="openshift-network-operator/iptables-alerter-xtknv"
Jul 01 19:21:36.288255 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.287459    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f9eaa621-a5fa-49a2-ad92-8dcd5729803a-dbus\") pod \"global-pull-secret-syncer-mq22h\" (UID: \"f9eaa621-a5fa-49a2-ad92-8dcd5729803a\") " pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:21:36.288255 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.287482    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/688e64ef-7029-47fe-bd40-369dcd4e2948-etc-selinux\") pod \"aws-ebs-csi-driver-node-5qlm2\" (UID: \"688e64ef-7029-47fe-bd40-369dcd4e2948\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5qlm2"
Jul 01 19:21:36.288255 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.287505    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsxdz\" (UniqueName: \"kubernetes.io/projected/190cde15-2eee-4b40-b4e1-c7cad1331778-kube-api-access-vsxdz\") pod \"iptables-alerter-xtknv\" (UID: \"190cde15-2eee-4b40-b4e1-c7cad1331778\") " pod="openshift-network-operator/iptables-alerter-xtknv"
Jul 01 19:21:36.288255 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.287529    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1673e32e-d2ee-445b-95d6-144e138343df-cni-binary-copy\") pod \"multus-additional-cni-plugins-hnpkl\" (UID: \"1673e32e-d2ee-445b-95d6-144e138343df\") " pod="openshift-multus/multus-additional-cni-plugins-hnpkl"
Jul 01 19:21:36.288255 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.287562    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bb05bf30-f19a-4b15-af14-30bf33f28fa9-os-release\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.288255 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.287604    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bb05bf30-f19a-4b15-af14-30bf33f28fa9-host-var-lib-cni-multus\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.288255 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.287627    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb05bf30-f19a-4b15-af14-30bf33f28fa9-multus-conf-dir\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.288255 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.287649    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f9eaa621-a5fa-49a2-ad92-8dcd5729803a-kubelet-config\") pod \"global-pull-secret-syncer-mq22h\" (UID: \"f9eaa621-a5fa-49a2-ad92-8dcd5729803a\") " pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:21:36.288255 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.287670    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-systemd-units\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.288255 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.287687    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1673e32e-d2ee-445b-95d6-144e138343df-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hnpkl\" (UID: \"1673e32e-d2ee-445b-95d6-144e138343df\") " pod="openshift-multus/multus-additional-cni-plugins-hnpkl"
Jul 01 19:21:36.288255 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.287706    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e96a778c-46ff-4720-a550-499d2f39ff83-sys\") pod \"tuned-nj5ft\" (UID: \"e96a778c-46ff-4720-a550-499d2f39ff83\") " pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.288255 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.287746    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9b5d6712-c210-43d7-ae23-6fe6bb85ef6c-konnectivity-ca\") pod \"konnectivity-agent-mch4p\" (UID: \"9b5d6712-c210-43d7-ae23-6fe6bb85ef6c\") " pod="kube-system/konnectivity-agent-mch4p"
Jul 01 19:21:36.288255 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.287777    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9jb8\" (UniqueName: \"kubernetes.io/projected/1673e32e-d2ee-445b-95d6-144e138343df-kube-api-access-x9jb8\") pod \"multus-additional-cni-plugins-hnpkl\" (UID: \"1673e32e-d2ee-445b-95d6-144e138343df\") " pod="openshift-multus/multus-additional-cni-plugins-hnpkl"
Jul 01 19:21:36.288657 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.287802    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e96a778c-46ff-4720-a550-499d2f39ff83-etc-tuned\") pod \"tuned-nj5ft\" (UID: \"e96a778c-46ff-4720-a550-499d2f39ff83\") " pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.288657 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.287830    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/688e64ef-7029-47fe-bd40-369dcd4e2948-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5qlm2\" (UID: \"688e64ef-7029-47fe-bd40-369dcd4e2948\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5qlm2"
Jul 01 19:21:36.288657 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.287851    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb05bf30-f19a-4b15-af14-30bf33f28fa9-system-cni-dir\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.288657 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.287867    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb05bf30-f19a-4b15-af14-30bf33f28fa9-host-var-lib-cni-bin\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.288657 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.287881    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/846ea9a9-80a2-4628-a2da-e3224610bf57-host\") pod \"node-ca-kjtgf\" (UID: \"846ea9a9-80a2-4628-a2da-e3224610bf57\") " pod="openshift-image-registry/node-ca-kjtgf"
Jul 01 19:21:36.288657 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.287934    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e96a778c-46ff-4720-a550-499d2f39ff83-etc-modprobe-d\") pod \"tuned-nj5ft\" (UID: \"e96a778c-46ff-4720-a550-499d2f39ff83\") " pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.288657 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.287951    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brtjq\" (UniqueName: \"kubernetes.io/projected/e96a778c-46ff-4720-a550-499d2f39ff83-kube-api-access-brtjq\") pod \"tuned-nj5ft\" (UID: \"e96a778c-46ff-4720-a550-499d2f39ff83\") " pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.288657 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.287967    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q775\" (UniqueName: \"kubernetes.io/projected/9e551421-5f11-40eb-8a22-e594b6f4eb61-kube-api-access-8q775\") pod \"network-check-target-7g8pr\" (UID: \"9e551421-5f11-40eb-8a22-e594b6f4eb61\") " pod="openshift-network-diagnostics/network-check-target-7g8pr"
Jul 01 19:21:36.288657 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.288003    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-var-lib-openvswitch\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.288657 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.288031    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb05bf30-f19a-4b15-af14-30bf33f28fa9-cnibin\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.288657 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.288060    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bb05bf30-f19a-4b15-af14-30bf33f28fa9-multus-socket-dir-parent\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.288657 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.288085    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bb05bf30-f19a-4b15-af14-30bf33f28fa9-host-run-multus-certs\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.288657 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.288113    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-log-socket\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.288657 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.288136    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bfb845e6-341d-492e-91c9-9c698a0b8d29-ovnkube-config\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.289094 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.288161    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98d86579-8161-4185-8cf7-5f427d712b2e-metrics-certs\") pod \"network-metrics-daemon-m6lm4\" (UID: \"98d86579-8161-4185-8cf7-5f427d712b2e\") " pod="openshift-multus/network-metrics-daemon-m6lm4"
Jul 01 19:21:36.289094 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.288209    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bb05bf30-f19a-4b15-af14-30bf33f28fa9-cni-binary-copy\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.289094 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.288240    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1673e32e-d2ee-445b-95d6-144e138343df-os-release\") pod \"multus-additional-cni-plugins-hnpkl\" (UID: \"1673e32e-d2ee-445b-95d6-144e138343df\") " pod="openshift-multus/multus-additional-cni-plugins-hnpkl"
Jul 01 19:21:36.296450 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.296427    2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 01 19:21:36.321360 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.321337    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-94zcd"
Jul 01 19:21:36.331652 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.331632    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-94zcd"
Jul 01 19:21:36.355688 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.355629    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.ec2.internal" event={"ID":"9e3a82d86892a337e8c834318f52d910","Type":"ContainerStarted","Data":"093678378cba1758eaa8e9c50940c34a18e323751ef9bdea84b72ce77a2c0bd2"}
Jul 01 19:21:36.356983 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.356960    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-164.ec2.internal" event={"ID":"f180425c803edc40a93bf63fc1f451d4","Type":"ContainerStarted","Data":"5425474e3cd4e7ff798eaf635d08daa972eca3a4a195832ddd7651db52742a89"}
Jul 01 19:21:36.376494 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.376476    2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 01 19:21:36.388702 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.388675    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1673e32e-d2ee-445b-95d6-144e138343df-cnibin\") pod \"multus-additional-cni-plugins-hnpkl\" (UID: \"1673e32e-d2ee-445b-95d6-144e138343df\") " pod="openshift-multus/multus-additional-cni-plugins-hnpkl"
Jul 01 19:21:36.388817 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.388718    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e96a778c-46ff-4720-a550-499d2f39ff83-etc-kubernetes\") pod \"tuned-nj5ft\" (UID: \"e96a778c-46ff-4720-a550-499d2f39ff83\") " pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.388817 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.388746    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e96a778c-46ff-4720-a550-499d2f39ff83-run\") pod \"tuned-nj5ft\" (UID: \"e96a778c-46ff-4720-a550-499d2f39ff83\") " pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.388817 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.388773    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9eaa621-a5fa-49a2-ad92-8dcd5729803a-original-pull-secret\") pod \"global-pull-secret-syncer-mq22h\" (UID: \"f9eaa621-a5fa-49a2-ad92-8dcd5729803a\") " pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:21:36.388817 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.388800    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j6qv\" (UniqueName: \"kubernetes.io/projected/98d86579-8161-4185-8cf7-5f427d712b2e-kube-api-access-4j6qv\") pod \"network-metrics-daemon-m6lm4\" (UID: \"98d86579-8161-4185-8cf7-5f427d712b2e\") " pod="openshift-multus/network-metrics-daemon-m6lm4"
Jul 01 19:21:36.389047 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.388825    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/190cde15-2eee-4b40-b4e1-c7cad1331778-iptables-alerter-script\") pod \"iptables-alerter-xtknv\" (UID: \"190cde15-2eee-4b40-b4e1-c7cad1331778\") " pod="openshift-network-operator/iptables-alerter-xtknv"
Jul 01 19:21:36.389047 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.388849    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e96a778c-46ff-4720-a550-499d2f39ff83-etc-sysconfig\") pod \"tuned-nj5ft\" (UID: \"e96a778c-46ff-4720-a550-499d2f39ff83\") " pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.389047 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.388873    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e96a778c-46ff-4720-a550-499d2f39ff83-etc-systemd\") pod \"tuned-nj5ft\" (UID: \"e96a778c-46ff-4720-a550-499d2f39ff83\") " pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.389047 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.388912    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bb05bf30-f19a-4b15-af14-30bf33f28fa9-multus-daemon-config\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.389047 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.388939    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4f8d\" (UniqueName: \"kubernetes.io/projected/bfb845e6-341d-492e-91c9-9c698a0b8d29-kube-api-access-h4f8d\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.389047 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.388964    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e96a778c-46ff-4720-a550-499d2f39ff83-var-lib-kubelet\") pod \"tuned-nj5ft\" (UID: \"e96a778c-46ff-4720-a550-499d2f39ff83\") " pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.389047 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.388985    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e96a778c-46ff-4720-a550-499d2f39ff83-tmp\") pod \"tuned-nj5ft\" (UID: \"e96a778c-46ff-4720-a550-499d2f39ff83\") " pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.389047 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389004    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/688e64ef-7029-47fe-bd40-369dcd4e2948-device-dir\") pod \"aws-ebs-csi-driver-node-5qlm2\" (UID: \"688e64ef-7029-47fe-bd40-369dcd4e2948\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5qlm2"
Jul 01 19:21:36.389047 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389018    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb05bf30-f19a-4b15-af14-30bf33f28fa9-multus-cni-dir\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.389047 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389032    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-host-cni-netd\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.389515 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389055    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bfb845e6-341d-492e-91c9-9c698a0b8d29-env-overrides\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.389515 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389081    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bfb845e6-341d-492e-91c9-9c698a0b8d29-ovn-node-metrics-cert\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.389515 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389104    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1673e32e-d2ee-445b-95d6-144e138343df-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hnpkl\" (UID: \"1673e32e-d2ee-445b-95d6-144e138343df\") " pod="openshift-multus/multus-additional-cni-plugins-hnpkl"
Jul 01 19:21:36.389515 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389128    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-run-openvswitch\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.389515 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389144    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-node-log\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.389515 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389158    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/190cde15-2eee-4b40-b4e1-c7cad1331778-host-slash\") pod \"iptables-alerter-xtknv\" (UID: \"190cde15-2eee-4b40-b4e1-c7cad1331778\") " pod="openshift-network-operator/iptables-alerter-xtknv"
Jul 01 19:21:36.389515 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389171    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f9eaa621-a5fa-49a2-ad92-8dcd5729803a-dbus\") pod \"global-pull-secret-syncer-mq22h\" (UID: \"f9eaa621-a5fa-49a2-ad92-8dcd5729803a\") " pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:21:36.389515 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389199    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/688e64ef-7029-47fe-bd40-369dcd4e2948-etc-selinux\") pod \"aws-ebs-csi-driver-node-5qlm2\" (UID: \"688e64ef-7029-47fe-bd40-369dcd4e2948\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5qlm2"
Jul 01 19:21:36.389515 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389219    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1673e32e-d2ee-445b-95d6-144e138343df-cnibin\") pod \"multus-additional-cni-plugins-hnpkl\" (UID: \"1673e32e-d2ee-445b-95d6-144e138343df\") " pod="openshift-multus/multus-additional-cni-plugins-hnpkl"
Jul 01 19:21:36.389515 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389220    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxk5g\" (UniqueName: \"kubernetes.io/projected/8d985f94-c35c-431b-bd74-89873c00e1e6-kube-api-access-xxk5g\") pod \"node-resolver-p58v4\" (UID: \"8d985f94-c35c-431b-bd74-89873c00e1e6\") " pod="openshift-dns/node-resolver-p58v4"
Jul 01 19:21:36.389515 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389258    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsxdz\" (UniqueName: \"kubernetes.io/projected/190cde15-2eee-4b40-b4e1-c7cad1331778-kube-api-access-vsxdz\") pod \"iptables-alerter-xtknv\" (UID: \"190cde15-2eee-4b40-b4e1-c7cad1331778\") " pod="openshift-network-operator/iptables-alerter-xtknv"
Jul 01 19:21:36.389515 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389277    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1673e32e-d2ee-445b-95d6-144e138343df-cni-binary-copy\") pod \"multus-additional-cni-plugins-hnpkl\" (UID: \"1673e32e-d2ee-445b-95d6-144e138343df\") " pod="openshift-multus/multus-additional-cni-plugins-hnpkl"
Jul 01 19:21:36.389515 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389293    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bb05bf30-f19a-4b15-af14-30bf33f28fa9-os-release\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.389515 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389308    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bb05bf30-f19a-4b15-af14-30bf33f28fa9-host-var-lib-cni-multus\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.390185 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389322    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb05bf30-f19a-4b15-af14-30bf33f28fa9-multus-conf-dir\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.390185 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389357    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f9eaa621-a5fa-49a2-ad92-8dcd5729803a-kubelet-config\") pod \"global-pull-secret-syncer-mq22h\" (UID: \"f9eaa621-a5fa-49a2-ad92-8dcd5729803a\") " pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:21:36.390185 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389373    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-systemd-units\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.390185 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389388    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1673e32e-d2ee-445b-95d6-144e138343df-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hnpkl\" (UID: \"1673e32e-d2ee-445b-95d6-144e138343df\") " pod="openshift-multus/multus-additional-cni-plugins-hnpkl"
Jul 01 19:21:36.390185 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389404    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e96a778c-46ff-4720-a550-499d2f39ff83-sys\") pod \"tuned-nj5ft\" (UID: \"e96a778c-46ff-4720-a550-499d2f39ff83\") " pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.390185 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389419    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9b5d6712-c210-43d7-ae23-6fe6bb85ef6c-konnectivity-ca\") pod \"konnectivity-agent-mch4p\" (UID: \"9b5d6712-c210-43d7-ae23-6fe6bb85ef6c\") " pod="kube-system/konnectivity-agent-mch4p"
Jul 01 19:21:36.390185 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389433    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9jb8\" (UniqueName: \"kubernetes.io/projected/1673e32e-d2ee-445b-95d6-144e138343df-kube-api-access-x9jb8\") pod \"multus-additional-cni-plugins-hnpkl\" (UID: \"1673e32e-d2ee-445b-95d6-144e138343df\") " pod="openshift-multus/multus-additional-cni-plugins-hnpkl"
Jul 01 19:21:36.390185 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389449    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e96a778c-46ff-4720-a550-499d2f39ff83-etc-tuned\") pod \"tuned-nj5ft\" (UID: \"e96a778c-46ff-4720-a550-499d2f39ff83\") " pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.390185 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389467    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/688e64ef-7029-47fe-bd40-369dcd4e2948-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5qlm2\" (UID: \"688e64ef-7029-47fe-bd40-369dcd4e2948\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5qlm2"
Jul 01 19:21:36.390185 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389482    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb05bf30-f19a-4b15-af14-30bf33f28fa9-system-cni-dir\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.390185 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389505    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb05bf30-f19a-4b15-af14-30bf33f28fa9-host-var-lib-cni-bin\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.390185 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389529    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/846ea9a9-80a2-4628-a2da-e3224610bf57-host\") pod \"node-ca-kjtgf\" (UID: \"846ea9a9-80a2-4628-a2da-e3224610bf57\") " pod="openshift-image-registry/node-ca-kjtgf"
Jul 01 19:21:36.390185 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389543    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e96a778c-46ff-4720-a550-499d2f39ff83-etc-modprobe-d\") pod \"tuned-nj5ft\" (UID: \"e96a778c-46ff-4720-a550-499d2f39ff83\") " pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.390185 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389567    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brtjq\" (UniqueName: \"kubernetes.io/projected/e96a778c-46ff-4720-a550-499d2f39ff83-kube-api-access-brtjq\") pod \"tuned-nj5ft\" (UID: \"e96a778c-46ff-4720-a550-499d2f39ff83\") " pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.390866 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389586    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q775\" (UniqueName: \"kubernetes.io/projected/9e551421-5f11-40eb-8a22-e594b6f4eb61-kube-api-access-8q775\") pod \"network-check-target-7g8pr\" (UID: \"9e551421-5f11-40eb-8a22-e594b6f4eb61\") " pod="openshift-network-diagnostics/network-check-target-7g8pr"
Jul 01 19:21:36.390866 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389603    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-var-lib-openvswitch\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.390866 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389616    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb05bf30-f19a-4b15-af14-30bf33f28fa9-cnibin\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.390866 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389647    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bb05bf30-f19a-4b15-af14-30bf33f28fa9-multus-socket-dir-parent\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.390866 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389667    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bb05bf30-f19a-4b15-af14-30bf33f28fa9-host-run-multus-certs\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.390866 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389692    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-log-socket\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.390866 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389717    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bfb845e6-341d-492e-91c9-9c698a0b8d29-ovnkube-config\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.390866 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389745    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8d985f94-c35c-431b-bd74-89873c00e1e6-hosts-file\") pod \"node-resolver-p58v4\" (UID: \"8d985f94-c35c-431b-bd74-89873c00e1e6\") " pod="openshift-dns/node-resolver-p58v4"
Jul 01 19:21:36.390866 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389769    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98d86579-8161-4185-8cf7-5f427d712b2e-metrics-certs\") pod \"network-metrics-daemon-m6lm4\" (UID: \"98d86579-8161-4185-8cf7-5f427d712b2e\") " pod="openshift-multus/network-metrics-daemon-m6lm4"
Jul 01 19:21:36.390866 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389780    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/190cde15-2eee-4b40-b4e1-c7cad1331778-iptables-alerter-script\") pod \"iptables-alerter-xtknv\" (UID: \"190cde15-2eee-4b40-b4e1-c7cad1331778\") " pod="openshift-network-operator/iptables-alerter-xtknv"
Jul 01 19:21:36.390866 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389793    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bb05bf30-f19a-4b15-af14-30bf33f28fa9-cni-binary-copy\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.390866 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389822    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1673e32e-d2ee-445b-95d6-144e138343df-os-release\") pod \"multus-additional-cni-plugins-hnpkl\" (UID: \"1673e32e-d2ee-445b-95d6-144e138343df\") " pod="openshift-multus/multus-additional-cni-plugins-hnpkl"
Jul 01 19:21:36.390866 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389885    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e96a778c-46ff-4720-a550-499d2f39ff83-etc-sysconfig\") pod \"tuned-nj5ft\" (UID: \"e96a778c-46ff-4720-a550-499d2f39ff83\") " pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.390866 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389911    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1673e32e-d2ee-445b-95d6-144e138343df-os-release\") pod \"multus-additional-cni-plugins-hnpkl\" (UID: \"1673e32e-d2ee-445b-95d6-144e138343df\") " pod="openshift-multus/multus-additional-cni-plugins-hnpkl"
Jul 01 19:21:36.391527 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389933    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1673e32e-d2ee-445b-95d6-144e138343df-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hnpkl\" (UID: \"1673e32e-d2ee-445b-95d6-144e138343df\") " pod="openshift-multus/multus-additional-cni-plugins-hnpkl"
Jul 01 19:21:36.391527 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389963    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb05bf30-f19a-4b15-af14-30bf33f28fa9-etc-kubernetes\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.391527 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389965    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e96a778c-46ff-4720-a550-499d2f39ff83-etc-kubernetes\") pod \"tuned-nj5ft\" (UID: \"e96a778c-46ff-4720-a550-499d2f39ff83\") " pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.391527 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.389992    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-etc-openvswitch\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.391527 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.390022    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-host-run-ovn-kubernetes\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.391527 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.390048    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/688e64ef-7029-47fe-bd40-369dcd4e2948-socket-dir\") pod \"aws-ebs-csi-driver-node-5qlm2\" (UID: \"688e64ef-7029-47fe-bd40-369dcd4e2948\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5qlm2"
Jul 01 19:21:36.391527 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.390083    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1673e32e-d2ee-445b-95d6-144e138343df-system-cni-dir\") pod \"multus-additional-cni-plugins-hnpkl\" (UID: \"1673e32e-d2ee-445b-95d6-144e138343df\") " pod="openshift-multus/multus-additional-cni-plugins-hnpkl"
Jul 01 19:21:36.391527 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.390099    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e96a778c-46ff-4720-a550-499d2f39ff83-run\") pod \"tuned-nj5ft\" (UID: \"e96a778c-46ff-4720-a550-499d2f39ff83\") " pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.391527 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.390112    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb05bf30-f19a-4b15-af14-30bf33f28fa9-host-var-lib-kubelet\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.391527 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.390130    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-run-systemd\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.391527 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.390161    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.391527 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:36.390172    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 19:21:36.391527 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.390187    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bfb845e6-341d-492e-91c9-9c698a0b8d29-ovnkube-script-lib\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.391527 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.390204    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9b5d6712-c210-43d7-ae23-6fe6bb85ef6c-agent-certs\") pod \"konnectivity-agent-mch4p\" (UID: \"9b5d6712-c210-43d7-ae23-6fe6bb85ef6c\") " pod="kube-system/konnectivity-agent-mch4p"
Jul 01 19:21:36.391527 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.390225    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/846ea9a9-80a2-4628-a2da-e3224610bf57-serviceca\") pod \"node-ca-kjtgf\" (UID: \"846ea9a9-80a2-4628-a2da-e3224610bf57\") " pod="openshift-image-registry/node-ca-kjtgf"
Jul 01 19:21:36.392287 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:36.390263    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9eaa621-a5fa-49a2-ad92-8dcd5729803a-original-pull-secret podName:f9eaa621-a5fa-49a2-ad92-8dcd5729803a nodeName:}" failed. No retries permitted until 2026-07-01 19:21:36.890228115 +0000 UTC m=+2.982227547 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f9eaa621-a5fa-49a2-ad92-8dcd5729803a-original-pull-secret") pod "global-pull-secret-syncer-mq22h" (UID: "f9eaa621-a5fa-49a2-ad92-8dcd5729803a") : object "kube-system"/"original-pull-secret" not registered
Jul 01 19:21:36.392287 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.390289    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e96a778c-46ff-4720-a550-499d2f39ff83-etc-sysctl-d\") pod \"tuned-nj5ft\" (UID: \"e96a778c-46ff-4720-a550-499d2f39ff83\") " pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.392287 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.390315    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-host-cni-bin\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.392287 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.390340    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4vwt\" (UniqueName: \"kubernetes.io/projected/846ea9a9-80a2-4628-a2da-e3224610bf57-kube-api-access-d4vwt\") pod \"node-ca-kjtgf\" (UID: \"846ea9a9-80a2-4628-a2da-e3224610bf57\") " pod="openshift-image-registry/node-ca-kjtgf"
Jul 01 19:21:36.392287 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.390361    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e96a778c-46ff-4720-a550-499d2f39ff83-etc-sysctl-conf\") pod \"tuned-nj5ft\" (UID: \"e96a778c-46ff-4720-a550-499d2f39ff83\") " pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.392287 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.390381    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e96a778c-46ff-4720-a550-499d2f39ff83-lib-modules\") pod \"tuned-nj5ft\" (UID: \"e96a778c-46ff-4720-a550-499d2f39ff83\") " pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.392287 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.390411    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e96a778c-46ff-4720-a550-499d2f39ff83-host\") pod \"tuned-nj5ft\" (UID: \"e96a778c-46ff-4720-a550-499d2f39ff83\") " pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.392287 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.390434    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/688e64ef-7029-47fe-bd40-369dcd4e2948-sys-fs\") pod \"aws-ebs-csi-driver-node-5qlm2\" (UID: \"688e64ef-7029-47fe-bd40-369dcd4e2948\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5qlm2"
Jul 01 19:21:36.392287 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.390450    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bb05bf30-f19a-4b15-af14-30bf33f28fa9-hostroot\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.392287 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.390465    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvhwt\" (UniqueName: \"kubernetes.io/projected/bb05bf30-f19a-4b15-af14-30bf33f28fa9-kube-api-access-kvhwt\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.392287 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.390481    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-host-slash\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.392287 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.390504    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-host-run-netns\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.392287 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.390534    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb05bf30-f19a-4b15-af14-30bf33f28fa9-host-run-netns\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.392287 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.390553    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-host-kubelet\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.392961 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.390568    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-run-ovn\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.392961 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.390588    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/688e64ef-7029-47fe-bd40-369dcd4e2948-registration-dir\") pod \"aws-ebs-csi-driver-node-5qlm2\" (UID: \"688e64ef-7029-47fe-bd40-369dcd4e2948\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5qlm2"
Jul 01 19:21:36.392961 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.390593    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/846ea9a9-80a2-4628-a2da-e3224610bf57-serviceca\") pod \"node-ca-kjtgf\" (UID: \"846ea9a9-80a2-4628-a2da-e3224610bf57\") " pod="openshift-image-registry/node-ca-kjtgf"
Jul 01 19:21:36.392961 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.390615    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vl29\" (UniqueName: \"kubernetes.io/projected/688e64ef-7029-47fe-bd40-369dcd4e2948-kube-api-access-8vl29\") pod \"aws-ebs-csi-driver-node-5qlm2\" (UID: \"688e64ef-7029-47fe-bd40-369dcd4e2948\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5qlm2"
Jul 01 19:21:36.392961 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.390642    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d985f94-c35c-431b-bd74-89873c00e1e6-tmp-dir\") pod \"node-resolver-p58v4\" (UID: \"8d985f94-c35c-431b-bd74-89873c00e1e6\") " pod="openshift-dns/node-resolver-p58v4"
Jul 01 19:21:36.392961 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.390653    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e96a778c-46ff-4720-a550-499d2f39ff83-etc-systemd\") pod \"tuned-nj5ft\" (UID: \"e96a778c-46ff-4720-a550-499d2f39ff83\") " pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.392961 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.390665    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bb05bf30-f19a-4b15-af14-30bf33f28fa9-host-run-k8s-cni-cncf-io\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.392961 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.390741    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bb05bf30-f19a-4b15-af14-30bf33f28fa9-host-run-k8s-cni-cncf-io\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.392961 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.390828    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f9eaa621-a5fa-49a2-ad92-8dcd5729803a-dbus\") pod \"global-pull-secret-syncer-mq22h\" (UID: \"f9eaa621-a5fa-49a2-ad92-8dcd5729803a\") " pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:21:36.392961 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.391067    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e96a778c-46ff-4720-a550-499d2f39ff83-var-lib-kubelet\") pod \"tuned-nj5ft\" (UID: \"e96a778c-46ff-4720-a550-499d2f39ff83\") " pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.392961 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.391189    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bb05bf30-f19a-4b15-af14-30bf33f28fa9-multus-daemon-config\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.392961 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.391291    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/688e64ef-7029-47fe-bd40-369dcd4e2948-etc-selinux\") pod \"aws-ebs-csi-driver-node-5qlm2\" (UID: \"688e64ef-7029-47fe-bd40-369dcd4e2948\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5qlm2"
Jul 01 19:21:36.392961 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.391335    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1673e32e-d2ee-445b-95d6-144e138343df-cni-binary-copy\") pod \"multus-additional-cni-plugins-hnpkl\" (UID: \"1673e32e-d2ee-445b-95d6-144e138343df\") " pod="openshift-multus/multus-additional-cni-plugins-hnpkl"
Jul 01 19:21:36.392961 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.391344    2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 01 19:21:36.392961 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.391397    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bb05bf30-f19a-4b15-af14-30bf33f28fa9-os-release\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.393662 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.391425    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bb05bf30-f19a-4b15-af14-30bf33f28fa9-host-var-lib-cni-multus\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.393662 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.391446    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb05bf30-f19a-4b15-af14-30bf33f28fa9-multus-conf-dir\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.393662 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.391491    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f9eaa621-a5fa-49a2-ad92-8dcd5729803a-kubelet-config\") pod \"global-pull-secret-syncer-mq22h\" (UID: \"f9eaa621-a5fa-49a2-ad92-8dcd5729803a\") " pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:21:36.393662 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.391514    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/688e64ef-7029-47fe-bd40-369dcd4e2948-device-dir\") pod \"aws-ebs-csi-driver-node-5qlm2\" (UID: \"688e64ef-7029-47fe-bd40-369dcd4e2948\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5qlm2"
Jul 01 19:21:36.393662 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.391532    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-systemd-units\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.393662 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.391584    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e96a778c-46ff-4720-a550-499d2f39ff83-sys\") pod \"tuned-nj5ft\" (UID: \"e96a778c-46ff-4720-a550-499d2f39ff83\") " pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.393662 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.391720    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1673e32e-d2ee-445b-95d6-144e138343df-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hnpkl\" (UID: \"1673e32e-d2ee-445b-95d6-144e138343df\") " pod="openshift-multus/multus-additional-cni-plugins-hnpkl"
Jul 01 19:21:36.393662 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.391776    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb05bf30-f19a-4b15-af14-30bf33f28fa9-etc-kubernetes\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.393662 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.391816    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-etc-openvswitch\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.393662 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.391860    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-host-run-ovn-kubernetes\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.393662 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.391975    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/688e64ef-7029-47fe-bd40-369dcd4e2948-socket-dir\") pod \"aws-ebs-csi-driver-node-5qlm2\" (UID: \"688e64ef-7029-47fe-bd40-369dcd4e2948\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5qlm2"
Jul 01 19:21:36.393662 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.391988    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1673e32e-d2ee-445b-95d6-144e138343df-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hnpkl\" (UID: \"1673e32e-d2ee-445b-95d6-144e138343df\") " pod="openshift-multus/multus-additional-cni-plugins-hnpkl"
Jul 01 19:21:36.393662 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.392021    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1673e32e-d2ee-445b-95d6-144e138343df-system-cni-dir\") pod \"multus-additional-cni-plugins-hnpkl\" (UID: \"1673e32e-d2ee-445b-95d6-144e138343df\") " pod="openshift-multus/multus-additional-cni-plugins-hnpkl"
Jul 01 19:21:36.393662 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.392050    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9b5d6712-c210-43d7-ae23-6fe6bb85ef6c-konnectivity-ca\") pod \"konnectivity-agent-mch4p\" (UID: \"9b5d6712-c210-43d7-ae23-6fe6bb85ef6c\") " pod="kube-system/konnectivity-agent-mch4p"
Jul 01 19:21:36.394365 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.392069    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb05bf30-f19a-4b15-af14-30bf33f28fa9-host-var-lib-kubelet\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.394365 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.392093    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-host-cni-netd\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.394365 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.392127    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-run-systemd\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.394365 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.392148    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-run-openvswitch\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.394365 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.392311    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-node-log\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.394365 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.392358    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/190cde15-2eee-4b40-b4e1-c7cad1331778-host-slash\") pod \"iptables-alerter-xtknv\" (UID: \"190cde15-2eee-4b40-b4e1-c7cad1331778\") " pod="openshift-network-operator/iptables-alerter-xtknv"
Jul 01 19:21:36.394365 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.392409    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb05bf30-f19a-4b15-af14-30bf33f28fa9-multus-cni-dir\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.394365 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.392618    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e96a778c-46ff-4720-a550-499d2f39ff83-etc-sysctl-d\") pod \"tuned-nj5ft\" (UID: \"e96a778c-46ff-4720-a550-499d2f39ff83\") " pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.394365 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.392616    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bfb845e6-341d-492e-91c9-9c698a0b8d29-env-overrides\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.394365 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.392664    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bfb845e6-341d-492e-91c9-9c698a0b8d29-ovnkube-script-lib\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.394365 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.392698    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-host-cni-bin\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.394365 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.392713    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-host-slash\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.394365 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.392746    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1673e32e-d2ee-445b-95d6-144e138343df-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hnpkl\" (UID: \"1673e32e-d2ee-445b-95d6-144e138343df\") " pod="openshift-multus/multus-additional-cni-plugins-hnpkl"
Jul 01 19:21:36.394365 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.392754    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-host-run-netns\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.394365 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.392791    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb05bf30-f19a-4b15-af14-30bf33f28fa9-host-run-netns\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.395093 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.392837    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/688e64ef-7029-47fe-bd40-369dcd4e2948-registration-dir\") pod \"aws-ebs-csi-driver-node-5qlm2\" (UID: \"688e64ef-7029-47fe-bd40-369dcd4e2948\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5qlm2"
Jul 01 19:21:36.395093 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.392838    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-run-ovn\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.395093 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.392886    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e96a778c-46ff-4720-a550-499d2f39ff83-host\") pod \"tuned-nj5ft\" (UID: \"e96a778c-46ff-4720-a550-499d2f39ff83\") " pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.395093 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.392923    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/688e64ef-7029-47fe-bd40-369dcd4e2948-sys-fs\") pod \"aws-ebs-csi-driver-node-5qlm2\" (UID: \"688e64ef-7029-47fe-bd40-369dcd4e2948\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5qlm2"
Jul 01 19:21:36.395093 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.392923    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.395093 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.392950    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-host-kubelet\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.395093 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.392983    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bb05bf30-f19a-4b15-af14-30bf33f28fa9-hostroot\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.395093 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.392995    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/846ea9a9-80a2-4628-a2da-e3224610bf57-host\") pod \"node-ca-kjtgf\" (UID: \"846ea9a9-80a2-4628-a2da-e3224610bf57\") " pod="openshift-image-registry/node-ca-kjtgf"
Jul 01 19:21:36.395093 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.393031    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e96a778c-46ff-4720-a550-499d2f39ff83-etc-sysctl-conf\") pod \"tuned-nj5ft\" (UID: \"e96a778c-46ff-4720-a550-499d2f39ff83\") " pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.395093 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.393039    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/688e64ef-7029-47fe-bd40-369dcd4e2948-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5qlm2\" (UID: \"688e64ef-7029-47fe-bd40-369dcd4e2948\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5qlm2"
Jul 01 19:21:36.395093 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.393078    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e96a778c-46ff-4720-a550-499d2f39ff83-etc-modprobe-d\") pod \"tuned-nj5ft\" (UID: \"e96a778c-46ff-4720-a550-499d2f39ff83\") " pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.395093 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.393087    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb05bf30-f19a-4b15-af14-30bf33f28fa9-system-cni-dir\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.395093 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.393120    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e96a778c-46ff-4720-a550-499d2f39ff83-lib-modules\") pod \"tuned-nj5ft\" (UID: \"e96a778c-46ff-4720-a550-499d2f39ff83\") " pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.395093 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.393132    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb05bf30-f19a-4b15-af14-30bf33f28fa9-host-var-lib-cni-bin\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.395093 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.393138    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb05bf30-f19a-4b15-af14-30bf33f28fa9-cnibin\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.395572 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.393169    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-var-lib-openvswitch\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.395572 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.393202    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bb05bf30-f19a-4b15-af14-30bf33f28fa9-multus-socket-dir-parent\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.395572 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:36.393246    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:21:36.395572 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.393277    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bfb845e6-341d-492e-91c9-9c698a0b8d29-log-socket\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.395572 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:36.393303    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98d86579-8161-4185-8cf7-5f427d712b2e-metrics-certs podName:98d86579-8161-4185-8cf7-5f427d712b2e nodeName:}" failed. No retries permitted until 2026-07-01 19:21:36.893286866 +0000 UTC m=+2.985286310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/98d86579-8161-4185-8cf7-5f427d712b2e-metrics-certs") pod "network-metrics-daemon-m6lm4" (UID: "98d86579-8161-4185-8cf7-5f427d712b2e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:21:36.395572 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.393245    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bb05bf30-f19a-4b15-af14-30bf33f28fa9-host-run-multus-certs\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.395572 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.393664    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bfb845e6-341d-492e-91c9-9c698a0b8d29-ovnkube-config\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.395572 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.393704    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bb05bf30-f19a-4b15-af14-30bf33f28fa9-cni-binary-copy\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.395572 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.394990    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e96a778c-46ff-4720-a550-499d2f39ff83-tmp\") pod \"tuned-nj5ft\" (UID: \"e96a778c-46ff-4720-a550-499d2f39ff83\") " pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.395572 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.395034    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e96a778c-46ff-4720-a550-499d2f39ff83-etc-tuned\") pod \"tuned-nj5ft\" (UID: \"e96a778c-46ff-4720-a550-499d2f39ff83\") " pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.396004 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.395607    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9b5d6712-c210-43d7-ae23-6fe6bb85ef6c-agent-certs\") pod \"konnectivity-agent-mch4p\" (UID: \"9b5d6712-c210-43d7-ae23-6fe6bb85ef6c\") " pod="kube-system/konnectivity-agent-mch4p"
Jul 01 19:21:36.396541 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.396515    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bfb845e6-341d-492e-91c9-9c698a0b8d29-ovn-node-metrics-cert\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.399598 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.399575    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j6qv\" (UniqueName: \"kubernetes.io/projected/98d86579-8161-4185-8cf7-5f427d712b2e-kube-api-access-4j6qv\") pod \"network-metrics-daemon-m6lm4\" (UID: \"98d86579-8161-4185-8cf7-5f427d712b2e\") " pod="openshift-multus/network-metrics-daemon-m6lm4"
Jul 01 19:21:36.400136 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.400099    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4f8d\" (UniqueName: \"kubernetes.io/projected/bfb845e6-341d-492e-91c9-9c698a0b8d29-kube-api-access-h4f8d\") pod \"ovnkube-node-bplsm\" (UID: \"bfb845e6-341d-492e-91c9-9c698a0b8d29\") " pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.401819 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.401789    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsxdz\" (UniqueName: \"kubernetes.io/projected/190cde15-2eee-4b40-b4e1-c7cad1331778-kube-api-access-vsxdz\") pod \"iptables-alerter-xtknv\" (UID: \"190cde15-2eee-4b40-b4e1-c7cad1331778\") " pod="openshift-network-operator/iptables-alerter-xtknv"
Jul 01 19:21:36.402969 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:36.402889    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 19:21:36.402969 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:36.402929    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 19:21:36.402969 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:36.402944    2574 projected.go:196] Error preparing data for projected volume kube-api-access-8q775 for pod openshift-network-diagnostics/network-check-target-7g8pr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:21:36.403202 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:36.403056    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e551421-5f11-40eb-8a22-e594b6f4eb61-kube-api-access-8q775 podName:9e551421-5f11-40eb-8a22-e594b6f4eb61 nodeName:}" failed. No retries permitted until 2026-07-01 19:21:36.903027627 +0000 UTC m=+2.995027058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8q775" (UniqueName: "kubernetes.io/projected/9e551421-5f11-40eb-8a22-e594b6f4eb61-kube-api-access-8q775") pod "network-check-target-7g8pr" (UID: "9e551421-5f11-40eb-8a22-e594b6f4eb61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:21:36.403821 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.403798    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9jb8\" (UniqueName: \"kubernetes.io/projected/1673e32e-d2ee-445b-95d6-144e138343df-kube-api-access-x9jb8\") pod \"multus-additional-cni-plugins-hnpkl\" (UID: \"1673e32e-d2ee-445b-95d6-144e138343df\") " pod="openshift-multus/multus-additional-cni-plugins-hnpkl"
Jul 01 19:21:36.404090 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.404069    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvhwt\" (UniqueName: \"kubernetes.io/projected/bb05bf30-f19a-4b15-af14-30bf33f28fa9-kube-api-access-kvhwt\") pod \"multus-txnzp\" (UID: \"bb05bf30-f19a-4b15-af14-30bf33f28fa9\") " pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.405546 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.405528    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brtjq\" (UniqueName: \"kubernetes.io/projected/e96a778c-46ff-4720-a550-499d2f39ff83-kube-api-access-brtjq\") pod \"tuned-nj5ft\" (UID: \"e96a778c-46ff-4720-a550-499d2f39ff83\") " pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.406671 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.406648    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vl29\" (UniqueName: \"kubernetes.io/projected/688e64ef-7029-47fe-bd40-369dcd4e2948-kube-api-access-8vl29\") pod \"aws-ebs-csi-driver-node-5qlm2\" (UID: \"688e64ef-7029-47fe-bd40-369dcd4e2948\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5qlm2"
Jul 01 19:21:36.408675 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.408654    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4vwt\" (UniqueName: \"kubernetes.io/projected/846ea9a9-80a2-4628-a2da-e3224610bf57-kube-api-access-d4vwt\") pod \"node-ca-kjtgf\" (UID: \"846ea9a9-80a2-4628-a2da-e3224610bf57\") " pod="openshift-image-registry/node-ca-kjtgf"
Jul 01 19:21:36.491927 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.491872    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d985f94-c35c-431b-bd74-89873c00e1e6-tmp-dir\") pod \"node-resolver-p58v4\" (UID: \"8d985f94-c35c-431b-bd74-89873c00e1e6\") " pod="openshift-dns/node-resolver-p58v4"
Jul 01 19:21:36.492107 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.491965    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxk5g\" (UniqueName: \"kubernetes.io/projected/8d985f94-c35c-431b-bd74-89873c00e1e6-kube-api-access-xxk5g\") pod \"node-resolver-p58v4\" (UID: \"8d985f94-c35c-431b-bd74-89873c00e1e6\") " pod="openshift-dns/node-resolver-p58v4"
Jul 01 19:21:36.492107 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.492014    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8d985f94-c35c-431b-bd74-89873c00e1e6-hosts-file\") pod \"node-resolver-p58v4\" (UID: \"8d985f94-c35c-431b-bd74-89873c00e1e6\") " pod="openshift-dns/node-resolver-p58v4"
Jul 01 19:21:36.492220 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.492106    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8d985f94-c35c-431b-bd74-89873c00e1e6-hosts-file\") pod \"node-resolver-p58v4\" (UID: \"8d985f94-c35c-431b-bd74-89873c00e1e6\") " pod="openshift-dns/node-resolver-p58v4"
Jul 01 19:21:36.492526 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.492502    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d985f94-c35c-431b-bd74-89873c00e1e6-tmp-dir\") pod \"node-resolver-p58v4\" (UID: \"8d985f94-c35c-431b-bd74-89873c00e1e6\") " pod="openshift-dns/node-resolver-p58v4"
Jul 01 19:21:36.500982 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.500953    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxk5g\" (UniqueName: \"kubernetes.io/projected/8d985f94-c35c-431b-bd74-89873c00e1e6-kube-api-access-xxk5g\") pod \"node-resolver-p58v4\" (UID: \"8d985f94-c35c-431b-bd74-89873c00e1e6\") " pod="openshift-dns/node-resolver-p58v4"
Jul 01 19:21:36.572959 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.572867    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nj5ft"
Jul 01 19:21:36.579093 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.578955    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-txnzp"
Jul 01 19:21:36.582596 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:36.582558    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode96a778c_46ff_4720_a550_499d2f39ff83.slice/crio-d8612ea0691a9ab764b573bd1ca3a75b07f7b0b1074baf232b1ad4627dde5c4e WatchSource:0}: Error finding container d8612ea0691a9ab764b573bd1ca3a75b07f7b0b1074baf232b1ad4627dde5c4e: Status 404 returned error can't find the container with id d8612ea0691a9ab764b573bd1ca3a75b07f7b0b1074baf232b1ad4627dde5c4e
Jul 01 19:21:36.585820 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.585621    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:21:36.589986 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:36.589111    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb05bf30_f19a_4b15_af14_30bf33f28fa9.slice/crio-4e7c0ba46edf3bb138825cd5bca7c776426e27de387097f33afab6c306472cc1 WatchSource:0}: Error finding container 4e7c0ba46edf3bb138825cd5bca7c776426e27de387097f33afab6c306472cc1: Status 404 returned error can't find the container with id 4e7c0ba46edf3bb138825cd5bca7c776426e27de387097f33afab6c306472cc1
Jul 01 19:21:36.589986 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.589759    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mch4p"
Jul 01 19:21:36.596229 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.596144    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5qlm2"
Jul 01 19:21:36.596663 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:36.596642    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfb845e6_341d_492e_91c9_9c698a0b8d29.slice/crio-8bd98ebbb23b49cd864079b77adeacef519751537809bd4877ab179087d2aebd WatchSource:0}: Error finding container 8bd98ebbb23b49cd864079b77adeacef519751537809bd4877ab179087d2aebd: Status 404 returned error can't find the container with id 8bd98ebbb23b49cd864079b77adeacef519751537809bd4877ab179087d2aebd
Jul 01 19:21:36.603252 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.603184    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kjtgf"
Jul 01 19:21:36.603554 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:36.603528    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b5d6712_c210_43d7_ae23_6fe6bb85ef6c.slice/crio-66c83bbb40fa4ef49417e80204e9a7efa4c4c5270ae743493625bd9205c1ed84 WatchSource:0}: Error finding container 66c83bbb40fa4ef49417e80204e9a7efa4c4c5270ae743493625bd9205c1ed84: Status 404 returned error can't find the container with id 66c83bbb40fa4ef49417e80204e9a7efa4c4c5270ae743493625bd9205c1ed84
Jul 01 19:21:36.606769 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:36.606741    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod688e64ef_7029_47fe_bd40_369dcd4e2948.slice/crio-a63a04efc8915c9cc0357708b9fd0870350afa7eacb9673179e4884a77c98a4a WatchSource:0}: Error finding container a63a04efc8915c9cc0357708b9fd0870350afa7eacb9673179e4884a77c98a4a: Status 404 returned error can't find the container with id a63a04efc8915c9cc0357708b9fd0870350afa7eacb9673179e4884a77c98a4a
Jul 01 19:21:36.609845 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.609823    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xtknv"
Jul 01 19:21:36.620760 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.618624    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hnpkl"
Jul 01 19:21:36.623506 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.623397    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-p58v4"
Jul 01 19:21:36.635322 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:36.635301    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1673e32e_d2ee_445b_95d6_144e138343df.slice/crio-d2dc20c99043b63645426101b870b09d2d87781ee024186c0bec88b70cb780ed WatchSource:0}: Error finding container d2dc20c99043b63645426101b870b09d2d87781ee024186c0bec88b70cb780ed: Status 404 returned error can't find the container with id d2dc20c99043b63645426101b870b09d2d87781ee024186c0bec88b70cb780ed
Jul 01 19:21:36.642222 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:21:36.642180    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d985f94_c35c_431b_bd74_89873c00e1e6.slice/crio-c5165abfb5b9a3af15d4d8129591944c3de4865f1ff0b6845eea23d80826e94b WatchSource:0}: Error finding container c5165abfb5b9a3af15d4d8129591944c3de4865f1ff0b6845eea23d80826e94b: Status 404 returned error can't find the container with id c5165abfb5b9a3af15d4d8129591944c3de4865f1ff0b6845eea23d80826e94b
Jul 01 19:21:36.895268 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.895232    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9eaa621-a5fa-49a2-ad92-8dcd5729803a-original-pull-secret\") pod \"global-pull-secret-syncer-mq22h\" (UID: \"f9eaa621-a5fa-49a2-ad92-8dcd5729803a\") " pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:21:36.895446 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.895313    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98d86579-8161-4185-8cf7-5f427d712b2e-metrics-certs\") pod \"network-metrics-daemon-m6lm4\" (UID: \"98d86579-8161-4185-8cf7-5f427d712b2e\") " pod="openshift-multus/network-metrics-daemon-m6lm4"
Jul 01 19:21:36.895446 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:36.895369    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 19:21:36.895446 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:36.895410    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:21:36.895446 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:36.895444    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9eaa621-a5fa-49a2-ad92-8dcd5729803a-original-pull-secret podName:f9eaa621-a5fa-49a2-ad92-8dcd5729803a nodeName:}" failed. No retries permitted until 2026-07-01 19:21:37.895425836 +0000 UTC m=+3.987425282 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f9eaa621-a5fa-49a2-ad92-8dcd5729803a-original-pull-secret") pod "global-pull-secret-syncer-mq22h" (UID: "f9eaa621-a5fa-49a2-ad92-8dcd5729803a") : object "kube-system"/"original-pull-secret" not registered
Jul 01 19:21:36.895629 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:36.895469    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98d86579-8161-4185-8cf7-5f427d712b2e-metrics-certs podName:98d86579-8161-4185-8cf7-5f427d712b2e nodeName:}" failed. No retries permitted until 2026-07-01 19:21:37.89545846 +0000 UTC m=+3.987457890 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/98d86579-8161-4185-8cf7-5f427d712b2e-metrics-certs") pod "network-metrics-daemon-m6lm4" (UID: "98d86579-8161-4185-8cf7-5f427d712b2e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:21:36.996443 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:36.996400    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q775\" (UniqueName: \"kubernetes.io/projected/9e551421-5f11-40eb-8a22-e594b6f4eb61-kube-api-access-8q775\") pod \"network-check-target-7g8pr\" (UID: \"9e551421-5f11-40eb-8a22-e594b6f4eb61\") " pod="openshift-network-diagnostics/network-check-target-7g8pr"
Jul 01 19:21:36.996615 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:36.996591    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 19:21:36.996671 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:36.996617    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 19:21:36.996671 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:36.996630    2574 projected.go:196] Error preparing data for projected volume kube-api-access-8q775 for pod openshift-network-diagnostics/network-check-target-7g8pr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:21:36.996755 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:36.996693    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e551421-5f11-40eb-8a22-e594b6f4eb61-kube-api-access-8q775 podName:9e551421-5f11-40eb-8a22-e594b6f4eb61 nodeName:}" failed. No retries permitted until 2026-07-01 19:21:37.996673903 +0000 UTC m=+4.088673335 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8q775" (UniqueName: "kubernetes.io/projected/9e551421-5f11-40eb-8a22-e594b6f4eb61-kube-api-access-8q775") pod "network-check-target-7g8pr" (UID: "9e551421-5f11-40eb-8a22-e594b6f4eb61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:21:37.334282 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:37.334161    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 19:16:36 +0000 UTC" deadline="2027-12-10 13:40:48.691513522 +0000 UTC"
Jul 01 19:21:37.334282 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:37.334224    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12642h19m11.357293609s"
Jul 01 19:21:37.350713 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:37.350685    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g8pr"
Jul 01 19:21:37.350847 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:37.350733    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6lm4"
Jul 01 19:21:37.350847 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:37.350813    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7g8pr" podUID="9e551421-5f11-40eb-8a22-e594b6f4eb61"
Jul 01 19:21:37.351012 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:37.350958    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6lm4" podUID="98d86579-8161-4185-8cf7-5f427d712b2e"
Jul 01 19:21:37.363141 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:37.363113    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-p58v4" event={"ID":"8d985f94-c35c-431b-bd74-89873c00e1e6","Type":"ContainerStarted","Data":"c5165abfb5b9a3af15d4d8129591944c3de4865f1ff0b6845eea23d80826e94b"}
Jul 01 19:21:37.365647 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:37.365614    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5qlm2" event={"ID":"688e64ef-7029-47fe-bd40-369dcd4e2948","Type":"ContainerStarted","Data":"a63a04efc8915c9cc0357708b9fd0870350afa7eacb9673179e4884a77c98a4a"}
Jul 01 19:21:37.368720 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:37.368621    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hnpkl" event={"ID":"1673e32e-d2ee-445b-95d6-144e138343df","Type":"ContainerStarted","Data":"d2dc20c99043b63645426101b870b09d2d87781ee024186c0bec88b70cb780ed"}
Jul 01 19:21:37.371736 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:37.371682    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xtknv" event={"ID":"190cde15-2eee-4b40-b4e1-c7cad1331778","Type":"ContainerStarted","Data":"2ad4587487a33a1ff5b9b9059d5c05174d6ddceb8234a938efa92c819e7a5201"}
Jul 01 19:21:37.373251 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:37.373185    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kjtgf" event={"ID":"846ea9a9-80a2-4628-a2da-e3224610bf57","Type":"ContainerStarted","Data":"f52bd0a74f811be544859e178d550355012dcb40dc5a5f358805050fd352369e"}
Jul 01 19:21:37.375515 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:37.375445    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mch4p" event={"ID":"9b5d6712-c210-43d7-ae23-6fe6bb85ef6c","Type":"ContainerStarted","Data":"66c83bbb40fa4ef49417e80204e9a7efa4c4c5270ae743493625bd9205c1ed84"}
Jul 01 19:21:37.377441 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:37.377335    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bplsm" event={"ID":"bfb845e6-341d-492e-91c9-9c698a0b8d29","Type":"ContainerStarted","Data":"8bd98ebbb23b49cd864079b77adeacef519751537809bd4877ab179087d2aebd"}
Jul 01 19:21:37.378960 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:37.378836    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-txnzp" event={"ID":"bb05bf30-f19a-4b15-af14-30bf33f28fa9","Type":"ContainerStarted","Data":"4e7c0ba46edf3bb138825cd5bca7c776426e27de387097f33afab6c306472cc1"}
Jul 01 19:21:37.381726 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:37.381664    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nj5ft" event={"ID":"e96a778c-46ff-4720-a550-499d2f39ff83","Type":"ContainerStarted","Data":"d8612ea0691a9ab764b573bd1ca3a75b07f7b0b1074baf232b1ad4627dde5c4e"}
Jul 01 19:21:37.906777 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:37.906739    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9eaa621-a5fa-49a2-ad92-8dcd5729803a-original-pull-secret\") pod \"global-pull-secret-syncer-mq22h\" (UID: \"f9eaa621-a5fa-49a2-ad92-8dcd5729803a\") " pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:21:37.906971 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:37.906830    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98d86579-8161-4185-8cf7-5f427d712b2e-metrics-certs\") pod \"network-metrics-daemon-m6lm4\" (UID: \"98d86579-8161-4185-8cf7-5f427d712b2e\") " pod="openshift-multus/network-metrics-daemon-m6lm4"
Jul 01 19:21:37.907042 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:37.906983    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:21:37.907097 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:37.907045    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98d86579-8161-4185-8cf7-5f427d712b2e-metrics-certs podName:98d86579-8161-4185-8cf7-5f427d712b2e nodeName:}" failed. No retries permitted until 2026-07-01 19:21:39.907026649 +0000 UTC m=+5.999026092 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/98d86579-8161-4185-8cf7-5f427d712b2e-metrics-certs") pod "network-metrics-daemon-m6lm4" (UID: "98d86579-8161-4185-8cf7-5f427d712b2e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:21:37.907462 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:37.907441    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 19:21:37.907555 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:37.907492    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9eaa621-a5fa-49a2-ad92-8dcd5729803a-original-pull-secret podName:f9eaa621-a5fa-49a2-ad92-8dcd5729803a nodeName:}" failed. No retries permitted until 2026-07-01 19:21:39.907478255 +0000 UTC m=+5.999477686 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f9eaa621-a5fa-49a2-ad92-8dcd5729803a-original-pull-secret") pod "global-pull-secret-syncer-mq22h" (UID: "f9eaa621-a5fa-49a2-ad92-8dcd5729803a") : object "kube-system"/"original-pull-secret" not registered
Jul 01 19:21:38.007772 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:38.007741    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q775\" (UniqueName: \"kubernetes.io/projected/9e551421-5f11-40eb-8a22-e594b6f4eb61-kube-api-access-8q775\") pod \"network-check-target-7g8pr\" (UID: \"9e551421-5f11-40eb-8a22-e594b6f4eb61\") " pod="openshift-network-diagnostics/network-check-target-7g8pr"
Jul 01 19:21:38.007982 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:38.007963    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 19:21:38.008062 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:38.007993    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 19:21:38.008062 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:38.008007    2574 projected.go:196] Error preparing data for projected volume kube-api-access-8q775 for pod openshift-network-diagnostics/network-check-target-7g8pr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:21:38.008180 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:38.008066    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e551421-5f11-40eb-8a22-e594b6f4eb61-kube-api-access-8q775 podName:9e551421-5f11-40eb-8a22-e594b6f4eb61 nodeName:}" failed. No retries permitted until 2026-07-01 19:21:40.008047882 +0000 UTC m=+6.100047327 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8q775" (UniqueName: "kubernetes.io/projected/9e551421-5f11-40eb-8a22-e594b6f4eb61-kube-api-access-8q775") pod "network-check-target-7g8pr" (UID: "9e551421-5f11-40eb-8a22-e594b6f4eb61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:21:38.300769 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:38.300691    2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 19:21:38.334765 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:38.334715    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 19:16:36 +0000 UTC" deadline="2027-12-25 18:38:20.791105156 +0000 UTC"
Jul 01 19:21:38.334765 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:38.334757    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13007h16m42.456352679s"
Jul 01 19:21:38.352294 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:38.351326    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:21:38.352552 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:38.352520    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq22h" podUID="f9eaa621-a5fa-49a2-ad92-8dcd5729803a"
Jul 01 19:21:39.350971 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:39.350468    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6lm4"
Jul 01 19:21:39.350971 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:39.350607    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6lm4" podUID="98d86579-8161-4185-8cf7-5f427d712b2e"
Jul 01 19:21:39.351494 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:39.350994    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g8pr"
Jul 01 19:21:39.351494 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:39.351104    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7g8pr" podUID="9e551421-5f11-40eb-8a22-e594b6f4eb61"
Jul 01 19:21:39.925433 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:39.925395    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9eaa621-a5fa-49a2-ad92-8dcd5729803a-original-pull-secret\") pod \"global-pull-secret-syncer-mq22h\" (UID: \"f9eaa621-a5fa-49a2-ad92-8dcd5729803a\") " pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:21:39.925608 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:39.925486    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98d86579-8161-4185-8cf7-5f427d712b2e-metrics-certs\") pod \"network-metrics-daemon-m6lm4\" (UID: \"98d86579-8161-4185-8cf7-5f427d712b2e\") " pod="openshift-multus/network-metrics-daemon-m6lm4"
Jul 01 19:21:39.925680 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:39.925608    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:21:39.925680 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:39.925671    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98d86579-8161-4185-8cf7-5f427d712b2e-metrics-certs podName:98d86579-8161-4185-8cf7-5f427d712b2e nodeName:}" failed. No retries permitted until 2026-07-01 19:21:43.92565328 +0000 UTC m=+10.017652713 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/98d86579-8161-4185-8cf7-5f427d712b2e-metrics-certs") pod "network-metrics-daemon-m6lm4" (UID: "98d86579-8161-4185-8cf7-5f427d712b2e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:21:39.926093 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:39.926076    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 19:21:39.926170 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:39.926129    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9eaa621-a5fa-49a2-ad92-8dcd5729803a-original-pull-secret podName:f9eaa621-a5fa-49a2-ad92-8dcd5729803a nodeName:}" failed. No retries permitted until 2026-07-01 19:21:43.926113253 +0000 UTC m=+10.018112684 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f9eaa621-a5fa-49a2-ad92-8dcd5729803a-original-pull-secret") pod "global-pull-secret-syncer-mq22h" (UID: "f9eaa621-a5fa-49a2-ad92-8dcd5729803a") : object "kube-system"/"original-pull-secret" not registered
Jul 01 19:21:40.026257 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:40.026218    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q775\" (UniqueName: \"kubernetes.io/projected/9e551421-5f11-40eb-8a22-e594b6f4eb61-kube-api-access-8q775\") pod \"network-check-target-7g8pr\" (UID: \"9e551421-5f11-40eb-8a22-e594b6f4eb61\") " pod="openshift-network-diagnostics/network-check-target-7g8pr"
Jul 01 19:21:40.026439 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:40.026403    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 19:21:40.026439 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:40.026425    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 19:21:40.026439 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:40.026438    2574 projected.go:196] Error preparing data for projected volume kube-api-access-8q775 for pod openshift-network-diagnostics/network-check-target-7g8pr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:21:40.026590 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:40.026501    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e551421-5f11-40eb-8a22-e594b6f4eb61-kube-api-access-8q775 podName:9e551421-5f11-40eb-8a22-e594b6f4eb61 nodeName:}" failed. No retries permitted until 2026-07-01 19:21:44.026475844 +0000 UTC m=+10.118475275 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8q775" (UniqueName: "kubernetes.io/projected/9e551421-5f11-40eb-8a22-e594b6f4eb61-kube-api-access-8q775") pod "network-check-target-7g8pr" (UID: "9e551421-5f11-40eb-8a22-e594b6f4eb61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:21:40.352888 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:40.352854    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:21:40.353333 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:40.352997    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq22h" podUID="f9eaa621-a5fa-49a2-ad92-8dcd5729803a"
Jul 01 19:21:41.350935 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:41.350839    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6lm4"
Jul 01 19:21:41.351114 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:41.351006    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6lm4" podUID="98d86579-8161-4185-8cf7-5f427d712b2e"
Jul 01 19:21:41.351114 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:41.351073    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g8pr"
Jul 01 19:21:41.351246 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:41.351220    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7g8pr" podUID="9e551421-5f11-40eb-8a22-e594b6f4eb61"
Jul 01 19:21:42.351723 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:42.351428    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:21:42.351723 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:42.351587    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq22h" podUID="f9eaa621-a5fa-49a2-ad92-8dcd5729803a"
Jul 01 19:21:43.350645 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:43.350606    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g8pr"
Jul 01 19:21:43.350829 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:43.350738    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7g8pr" podUID="9e551421-5f11-40eb-8a22-e594b6f4eb61"
Jul 01 19:21:43.351165 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:43.351003    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6lm4"
Jul 01 19:21:43.351165 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:43.351132    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6lm4" podUID="98d86579-8161-4185-8cf7-5f427d712b2e"
Jul 01 19:21:43.960679 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:43.960635    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98d86579-8161-4185-8cf7-5f427d712b2e-metrics-certs\") pod \"network-metrics-daemon-m6lm4\" (UID: \"98d86579-8161-4185-8cf7-5f427d712b2e\") " pod="openshift-multus/network-metrics-daemon-m6lm4"
Jul 01 19:21:43.961052 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:43.960706    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9eaa621-a5fa-49a2-ad92-8dcd5729803a-original-pull-secret\") pod \"global-pull-secret-syncer-mq22h\" (UID: \"f9eaa621-a5fa-49a2-ad92-8dcd5729803a\") " pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:21:43.961052 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:43.960817    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:21:43.961052 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:43.960831    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 19:21:43.961052 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:43.960920    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9eaa621-a5fa-49a2-ad92-8dcd5729803a-original-pull-secret podName:f9eaa621-a5fa-49a2-ad92-8dcd5729803a nodeName:}" failed. No retries permitted until 2026-07-01 19:21:51.960883017 +0000 UTC m=+18.052882463 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f9eaa621-a5fa-49a2-ad92-8dcd5729803a-original-pull-secret") pod "global-pull-secret-syncer-mq22h" (UID: "f9eaa621-a5fa-49a2-ad92-8dcd5729803a") : object "kube-system"/"original-pull-secret" not registered
Jul 01 19:21:43.961052 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:43.960946    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98d86579-8161-4185-8cf7-5f427d712b2e-metrics-certs podName:98d86579-8161-4185-8cf7-5f427d712b2e nodeName:}" failed. No retries permitted until 2026-07-01 19:21:51.960933358 +0000 UTC m=+18.052932787 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/98d86579-8161-4185-8cf7-5f427d712b2e-metrics-certs") pod "network-metrics-daemon-m6lm4" (UID: "98d86579-8161-4185-8cf7-5f427d712b2e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:21:44.061071 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:44.061023    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q775\" (UniqueName: \"kubernetes.io/projected/9e551421-5f11-40eb-8a22-e594b6f4eb61-kube-api-access-8q775\") pod \"network-check-target-7g8pr\" (UID: \"9e551421-5f11-40eb-8a22-e594b6f4eb61\") " pod="openshift-network-diagnostics/network-check-target-7g8pr"
Jul 01 19:21:44.061260 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:44.061227    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 19:21:44.061260 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:44.061250    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 19:21:44.061260 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:44.061263    2574 projected.go:196] Error preparing data for projected volume kube-api-access-8q775 for pod openshift-network-diagnostics/network-check-target-7g8pr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:21:44.061421 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:44.061326    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e551421-5f11-40eb-8a22-e594b6f4eb61-kube-api-access-8q775 podName:9e551421-5f11-40eb-8a22-e594b6f4eb61 nodeName:}" failed. No retries permitted until 2026-07-01 19:21:52.061309085 +0000 UTC m=+18.153308540 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8q775" (UniqueName: "kubernetes.io/projected/9e551421-5f11-40eb-8a22-e594b6f4eb61-kube-api-access-8q775") pod "network-check-target-7g8pr" (UID: "9e551421-5f11-40eb-8a22-e594b6f4eb61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:21:44.352813 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:44.352786    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:21:44.352982 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:44.352928    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq22h" podUID="f9eaa621-a5fa-49a2-ad92-8dcd5729803a"
Jul 01 19:21:45.350972 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:45.350936    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6lm4"
Jul 01 19:21:45.351416 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:45.350979    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g8pr"
Jul 01 19:21:45.351416 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:45.351089    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6lm4" podUID="98d86579-8161-4185-8cf7-5f427d712b2e"
Jul 01 19:21:45.351416 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:45.351216    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7g8pr" podUID="9e551421-5f11-40eb-8a22-e594b6f4eb61"
Jul 01 19:21:46.351648 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:46.351607    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:21:46.352087 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:46.351836    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq22h" podUID="f9eaa621-a5fa-49a2-ad92-8dcd5729803a"
Jul 01 19:21:47.350834 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:47.350805    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6lm4"
Jul 01 19:21:47.351013 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:47.350923    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g8pr"
Jul 01 19:21:47.351076 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:47.351010    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7g8pr" podUID="9e551421-5f11-40eb-8a22-e594b6f4eb61"
Jul 01 19:21:47.351076 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:47.350925    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6lm4" podUID="98d86579-8161-4185-8cf7-5f427d712b2e"
Jul 01 19:21:48.351456 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:48.351416    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:21:48.351940 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:48.351542    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq22h" podUID="f9eaa621-a5fa-49a2-ad92-8dcd5729803a"
Jul 01 19:21:49.351252 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:49.351216    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6lm4"
Jul 01 19:21:49.351415 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:49.351231    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g8pr"
Jul 01 19:21:49.351415 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:49.351344    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6lm4" podUID="98d86579-8161-4185-8cf7-5f427d712b2e"
Jul 01 19:21:49.351415 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:49.351377    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7g8pr" podUID="9e551421-5f11-40eb-8a22-e594b6f4eb61"
Jul 01 19:21:50.351028 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:50.350990    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:21:50.351219 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:50.351127    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq22h" podUID="f9eaa621-a5fa-49a2-ad92-8dcd5729803a"
Jul 01 19:21:51.351457 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:51.351419    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6lm4"
Jul 01 19:21:51.351457 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:51.351453    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g8pr"
Jul 01 19:21:51.352038 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:51.351555    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6lm4" podUID="98d86579-8161-4185-8cf7-5f427d712b2e"
Jul 01 19:21:51.352038 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:51.351637    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7g8pr" podUID="9e551421-5f11-40eb-8a22-e594b6f4eb61"
Jul 01 19:21:52.020095 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:52.020005    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98d86579-8161-4185-8cf7-5f427d712b2e-metrics-certs\") pod \"network-metrics-daemon-m6lm4\" (UID: \"98d86579-8161-4185-8cf7-5f427d712b2e\") " pod="openshift-multus/network-metrics-daemon-m6lm4"
Jul 01 19:21:52.020095 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:52.020080    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9eaa621-a5fa-49a2-ad92-8dcd5729803a-original-pull-secret\") pod \"global-pull-secret-syncer-mq22h\" (UID: \"f9eaa621-a5fa-49a2-ad92-8dcd5729803a\") " pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:21:52.020284 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:52.020160    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:21:52.020284 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:52.020193    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 19:21:52.020284 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:52.020230    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98d86579-8161-4185-8cf7-5f427d712b2e-metrics-certs podName:98d86579-8161-4185-8cf7-5f427d712b2e nodeName:}" failed. No retries permitted until 2026-07-01 19:22:08.020215425 +0000 UTC m=+34.112214851 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/98d86579-8161-4185-8cf7-5f427d712b2e-metrics-certs") pod "network-metrics-daemon-m6lm4" (UID: "98d86579-8161-4185-8cf7-5f427d712b2e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:21:52.020284 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:52.020245    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9eaa621-a5fa-49a2-ad92-8dcd5729803a-original-pull-secret podName:f9eaa621-a5fa-49a2-ad92-8dcd5729803a nodeName:}" failed. No retries permitted until 2026-07-01 19:22:08.020238636 +0000 UTC m=+34.112238062 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f9eaa621-a5fa-49a2-ad92-8dcd5729803a-original-pull-secret") pod "global-pull-secret-syncer-mq22h" (UID: "f9eaa621-a5fa-49a2-ad92-8dcd5729803a") : object "kube-system"/"original-pull-secret" not registered
Jul 01 19:21:52.121035 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:52.120992    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q775\" (UniqueName: \"kubernetes.io/projected/9e551421-5f11-40eb-8a22-e594b6f4eb61-kube-api-access-8q775\") pod \"network-check-target-7g8pr\" (UID: \"9e551421-5f11-40eb-8a22-e594b6f4eb61\") " pod="openshift-network-diagnostics/network-check-target-7g8pr"
Jul 01 19:21:52.121206 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:52.121185    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 19:21:52.121248 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:52.121215    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 19:21:52.121248 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:52.121227    2574 projected.go:196] Error preparing data for projected volume kube-api-access-8q775 for pod openshift-network-diagnostics/network-check-target-7g8pr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:21:52.121319 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:52.121291    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e551421-5f11-40eb-8a22-e594b6f4eb61-kube-api-access-8q775 podName:9e551421-5f11-40eb-8a22-e594b6f4eb61 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:08.121276618 +0000 UTC m=+34.213276045 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8q775" (UniqueName: "kubernetes.io/projected/9e551421-5f11-40eb-8a22-e594b6f4eb61-kube-api-access-8q775") pod "network-check-target-7g8pr" (UID: "9e551421-5f11-40eb-8a22-e594b6f4eb61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:21:52.350770 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:52.350696    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:21:52.350980 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:52.350789    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq22h" podUID="f9eaa621-a5fa-49a2-ad92-8dcd5729803a"
Jul 01 19:21:53.351163 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:53.351113    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g8pr"
Jul 01 19:21:53.351627 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:53.351247    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7g8pr" podUID="9e551421-5f11-40eb-8a22-e594b6f4eb61"
Jul 01 19:21:53.351627 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:53.351289    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6lm4"
Jul 01 19:21:53.351627 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:53.351443    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6lm4" podUID="98d86579-8161-4185-8cf7-5f427d712b2e"
Jul 01 19:21:54.351406 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:54.351370    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:21:54.351856 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:54.351457    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq22h" podUID="f9eaa621-a5fa-49a2-ad92-8dcd5729803a"
Jul 01 19:21:55.351190 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:55.351160    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6lm4"
Jul 01 19:21:55.351333 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:55.351273    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6lm4" podUID="98d86579-8161-4185-8cf7-5f427d712b2e"
Jul 01 19:21:55.351333 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:55.351322    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g8pr"
Jul 01 19:21:55.351446 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:55.351428    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7g8pr" podUID="9e551421-5f11-40eb-8a22-e594b6f4eb61"
Jul 01 19:21:56.351259 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:56.351072    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:21:56.351404 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:56.351351    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq22h" podUID="f9eaa621-a5fa-49a2-ad92-8dcd5729803a"
Jul 01 19:21:56.424462 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:56.424440    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bplsm_bfb845e6-341d-492e-91c9-9c698a0b8d29/ovn-acl-logging/0.log"
Jul 01 19:21:56.424913 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:56.424869    2574 generic.go:356] "Generic (PLEG): container finished" podID="bfb845e6-341d-492e-91c9-9c698a0b8d29" containerID="181ab6561e522bbc18ed74c3bc2e091b7fc124ac52c25dd624c24ab8f5937a1d" exitCode=1
Jul 01 19:21:56.424993 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:56.424931    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bplsm" event={"ID":"bfb845e6-341d-492e-91c9-9c698a0b8d29","Type":"ContainerStarted","Data":"246823069b655fc80a0f24a084726b2d466332a1589999c673757b3139e42245"}
Jul 01 19:21:56.425133 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:56.425079    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bplsm" event={"ID":"bfb845e6-341d-492e-91c9-9c698a0b8d29","Type":"ContainerStarted","Data":"d3f7f258b6064f1e288d13cfbceb622efddfceebc3e41f247ad657e748145f2d"}
Jul 01 19:21:56.425194 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:56.425139    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bplsm" event={"ID":"bfb845e6-341d-492e-91c9-9c698a0b8d29","Type":"ContainerDied","Data":"181ab6561e522bbc18ed74c3bc2e091b7fc124ac52c25dd624c24ab8f5937a1d"}
Jul 01 19:21:56.425194 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:56.425190    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bplsm" event={"ID":"bfb845e6-341d-492e-91c9-9c698a0b8d29","Type":"ContainerStarted","Data":"ee6b857dc4d31526c3dc8124c49f8e0381d533c43565f0bc27aca4c0c532eae2"}
Jul 01 19:21:56.426627 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:56.426541    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-txnzp" event={"ID":"bb05bf30-f19a-4b15-af14-30bf33f28fa9","Type":"ContainerStarted","Data":"9a4e3320b38cf4caedc3139fb4640b442ef78b0415a87d49aa5ec4f395cbcf99"}
Jul 01 19:21:56.429554 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:56.429526    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nj5ft" event={"ID":"e96a778c-46ff-4720-a550-499d2f39ff83","Type":"ContainerStarted","Data":"4ccc3c100c2a4256bf55e9740dc0cb70f0a18920bba5f23d7977426ab0127ed4"}
Jul 01 19:21:56.432883 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:56.432454    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-164.ec2.internal" event={"ID":"f180425c803edc40a93bf63fc1f451d4","Type":"ContainerStarted","Data":"9584eabc69d86133bc26a917c695e353a8518c273841950e8f1e0054e4da3123"}
Jul 01 19:21:56.446249 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:56.446194    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-txnzp" podStartSLOduration=2.816392713 podStartE2EDuration="22.44617495s" totalImagesPullingTime="19.629782237s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:21:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:21:56.445696508 +0000 UTC m=+22.537695983" watchObservedRunningTime="2026-07-01 19:21:56.44617495 +0000 UTC m=+22.538174398"
Jul 01 19:21:56.464539 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:56.464484    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nj5ft" podStartSLOduration=3.202453778 podStartE2EDuration="22.464463416s" totalImagesPullingTime="19.262009638s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:21:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:21:56.463448611 +0000 UTC m=+22.555448083" watchObservedRunningTime="2026-07-01 19:21:56.464463416 +0000 UTC m=+22.556462866"
Jul 01 19:21:56.481546 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:56.480936    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-164.ec2.internal" podStartSLOduration=21.480892313 podStartE2EDuration="21.480892313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:21:56.480879652 +0000 UTC m=+22.572879102" watchObservedRunningTime="2026-07-01 19:21:56.480892313 +0000 UTC m=+22.572891765"
Jul 01 19:21:57.350861 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:57.350657    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6lm4"
Jul 01 19:21:57.351046 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:57.350711    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g8pr"
Jul 01 19:21:57.351046 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:57.350976    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6lm4" podUID="98d86579-8161-4185-8cf7-5f427d712b2e"
Jul 01 19:21:57.351046 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:57.351032    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7g8pr" podUID="9e551421-5f11-40eb-8a22-e594b6f4eb61"
Jul 01 19:21:57.436498 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:57.436423    2574 generic.go:356] "Generic (PLEG): container finished" podID="1673e32e-d2ee-445b-95d6-144e138343df" containerID="9deb720e075b836a184cc3e291a1f304fc321cd0ad7f141bce1a5e9b14ebd671" exitCode=0
Jul 01 19:21:57.437272 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:57.436501    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hnpkl" event={"ID":"1673e32e-d2ee-445b-95d6-144e138343df","Type":"ContainerDied","Data":"9deb720e075b836a184cc3e291a1f304fc321cd0ad7f141bce1a5e9b14ebd671"}
Jul 01 19:21:57.438065 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:57.437952    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xtknv" event={"ID":"190cde15-2eee-4b40-b4e1-c7cad1331778","Type":"ContainerStarted","Data":"78b31554671db66df87627af2bce6577e45eb3563737c6c9c1800ac67981ba70"}
Jul 01 19:21:57.439374 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:57.439355    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kjtgf" event={"ID":"846ea9a9-80a2-4628-a2da-e3224610bf57","Type":"ContainerStarted","Data":"5b87cef63a19529804caf0ad735646af0caa91188c84b89ba6a23b2256d47955"}
Jul 01 19:21:57.440639 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:57.440614    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mch4p" event={"ID":"9b5d6712-c210-43d7-ae23-6fe6bb85ef6c","Type":"ContainerStarted","Data":"c96165c14ecc6349501a23c69c7097f6a73ab2318855343b319cafe5950b79ac"}
Jul 01 19:21:57.443198 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:57.443181    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bplsm_bfb845e6-341d-492e-91c9-9c698a0b8d29/ovn-acl-logging/0.log"
Jul 01 19:21:57.443564 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:57.443542    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bplsm" event={"ID":"bfb845e6-341d-492e-91c9-9c698a0b8d29","Type":"ContainerStarted","Data":"37bb95999f17644c02cf3229080a13b4b186bd309e7f92d07cafeb3ed2824e44"}
Jul 01 19:21:57.443653 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:57.443616    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bplsm" event={"ID":"bfb845e6-341d-492e-91c9-9c698a0b8d29","Type":"ContainerStarted","Data":"a224e74e83339235a26456d6ab8a8b46509152c62d03e123f78a099ee7632c80"}
Jul 01 19:21:57.444884 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:57.444865    2574 generic.go:356] "Generic (PLEG): container finished" podID="9e3a82d86892a337e8c834318f52d910" containerID="5678fc14cbe2c4f00579ef4ee5f472e2324e9e800a0354db5257a61d7bd932f8" exitCode=0
Jul 01 19:21:57.444989 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:57.444947    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.ec2.internal" event={"ID":"9e3a82d86892a337e8c834318f52d910","Type":"ContainerDied","Data":"5678fc14cbe2c4f00579ef4ee5f472e2324e9e800a0354db5257a61d7bd932f8"}
Jul 01 19:21:57.446290 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:57.446272    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-p58v4" event={"ID":"8d985f94-c35c-431b-bd74-89873c00e1e6","Type":"ContainerStarted","Data":"6d6878910c95de63873f3eacb436092e7094122be60a09e6613f94e0c06ddf89"}
Jul 01 19:21:57.447685 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:57.447663    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5qlm2" event={"ID":"688e64ef-7029-47fe-bd40-369dcd4e2948","Type":"ContainerStarted","Data":"8142f4be043fb93f9340754218d42a3db30e73ec3a83bc4bad5eaba1edd0d2c5"}
Jul 01 19:21:57.487518 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:57.487470    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mch4p" podStartSLOduration=8.365668836 podStartE2EDuration="23.487453748s" totalImagesPullingTime="15.121784912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:21:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:21:57.487070171 +0000 UTC m=+23.579069619" watchObservedRunningTime="2026-07-01 19:21:57.487453748 +0000 UTC m=+23.579453197"
Jul 01 19:21:57.507180 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:57.507128    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kjtgf" podStartSLOduration=4.277023627 podStartE2EDuration="23.507110953s" totalImagesPullingTime="19.230087326s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:21:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:21:57.506530408 +0000 UTC m=+23.598529881" watchObservedRunningTime="2026-07-01 19:21:57.507110953 +0000 UTC m=+23.599110401"
Jul 01 19:21:57.521671 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:57.521618    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-p58v4" podStartSLOduration=4.318901915 podStartE2EDuration="23.521601089s" totalImagesPullingTime="19.202699174s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:21:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:21:57.521537834 +0000 UTC m=+23.613537282" watchObservedRunningTime="2026-07-01 19:21:57.521601089 +0000 UTC m=+23.613600537"
Jul 01 19:21:58.113028 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:58.113007    2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 01 19:21:58.199686 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:58.199655    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mch4p"
Jul 01 19:21:58.200220 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:58.200198    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mch4p"
Jul 01 19:21:58.226469 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:58.226426    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xtknv" podStartSLOduration=5.458421678 podStartE2EDuration="24.22641077s" totalImagesPullingTime="18.767989092s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:21:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:21:57.54122161 +0000 UTC m=+23.633221057" watchObservedRunningTime="2026-07-01 19:21:58.22641077 +0000 UTC m=+24.318410285"
Jul 01 19:21:58.350859 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:58.350780    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:21:58.351038 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:58.350912    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq22h" podUID="f9eaa621-a5fa-49a2-ad92-8dcd5729803a"
Jul 01 19:21:58.361031 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:58.360959    2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-01T19:21:58.113023782Z","UUID":"031be15f-0f95-4ddc-b860-64e332ae0f57","Handler":null,"Name":"","Endpoint":""}
Jul 01 19:21:58.362518 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:58.362497    2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 01 19:21:58.362626 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:58.362533    2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 01 19:21:58.452115 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:58.452077    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5qlm2" event={"ID":"688e64ef-7029-47fe-bd40-369dcd4e2948","Type":"ContainerStarted","Data":"0ae55d19618d691d7496370130a52bbf883b3fd12bc25c7d99c8b1dbf7324e1a"}
Jul 01 19:21:58.454212 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:58.454180    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.ec2.internal" event={"ID":"9e3a82d86892a337e8c834318f52d910","Type":"ContainerStarted","Data":"a0ed92de0f69e463c798d4af8c8b218f022e67605d506c1a0e4e44e4f68be419"}
Jul 01 19:21:58.454473 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:58.454454    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mch4p"
Jul 01 19:21:58.455069 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:58.455047    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mch4p"
Jul 01 19:21:58.478624 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:58.478578    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.ec2.internal" podStartSLOduration=23.478561822 podStartE2EDuration="23.478561822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:21:58.477876867 +0000 UTC m=+24.569876315" watchObservedRunningTime="2026-07-01 19:21:58.478561822 +0000 UTC m=+24.570561273"
Jul 01 19:21:59.350643 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:59.350620    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6lm4"
Jul 01 19:21:59.350779 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:59.350661    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g8pr"
Jul 01 19:21:59.350779 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:59.350751    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6lm4" podUID="98d86579-8161-4185-8cf7-5f427d712b2e"
Jul 01 19:21:59.350972 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:21:59.350890    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7g8pr" podUID="9e551421-5f11-40eb-8a22-e594b6f4eb61"
Jul 01 19:21:59.460326 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:59.460301    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bplsm_bfb845e6-341d-492e-91c9-9c698a0b8d29/ovn-acl-logging/0.log"
Jul 01 19:21:59.460734 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:59.460715    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bplsm" event={"ID":"bfb845e6-341d-492e-91c9-9c698a0b8d29","Type":"ContainerStarted","Data":"ece072a712d54a52775947a5600df9b83c4af1a390d6ea055341714a41c29092"}
Jul 01 19:21:59.463054 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:59.463029    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5qlm2" event={"ID":"688e64ef-7029-47fe-bd40-369dcd4e2948","Type":"ContainerStarted","Data":"0e8d17f7c8915920a6b1a003643ea7ee89739845af930d3961eb5fc1de87c7fb"}
Jul 01 19:21:59.487478 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:21:59.487421    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5qlm2" podStartSLOduration=3.636240254 podStartE2EDuration="25.487402309s" totalImagesPullingTime="21.851162055s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:21:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:21:59.486744728 +0000 UTC m=+25.578744188" watchObservedRunningTime="2026-07-01 19:21:59.487402309 +0000 UTC m=+25.579401758"
Jul 01 19:22:00.350884 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:00.350852    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:22:00.351109 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:00.350979    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq22h" podUID="f9eaa621-a5fa-49a2-ad92-8dcd5729803a"
Jul 01 19:22:01.350959 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:01.350721    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6lm4"
Jul 01 19:22:01.351370 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:01.350748    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g8pr"
Jul 01 19:22:01.351370 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:01.351028    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6lm4" podUID="98d86579-8161-4185-8cf7-5f427d712b2e"
Jul 01 19:22:01.351370 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:01.351077    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7g8pr" podUID="9e551421-5f11-40eb-8a22-e594b6f4eb61"
Jul 01 19:22:02.351164 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:02.351137    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:22:02.352000 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:02.351263    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq22h" podUID="f9eaa621-a5fa-49a2-ad92-8dcd5729803a"
Jul 01 19:22:02.473951 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:02.473753    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bplsm_bfb845e6-341d-492e-91c9-9c698a0b8d29/ovn-acl-logging/0.log"
Jul 01 19:22:02.474427 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:02.474396    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bplsm" event={"ID":"bfb845e6-341d-492e-91c9-9c698a0b8d29","Type":"ContainerStarted","Data":"70df550c90b909855366430490fd68359f33fb2611651dffca4fa3eb1d8fdc32"}
Jul 01 19:22:02.474708 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:02.474688    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:22:02.474962 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:02.474934    2574 scope.go:117] "RemoveContainer" containerID="181ab6561e522bbc18ed74c3bc2e091b7fc124ac52c25dd624c24ab8f5937a1d"
Jul 01 19:22:02.476486 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:02.476459    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hnpkl" event={"ID":"1673e32e-d2ee-445b-95d6-144e138343df","Type":"ContainerStarted","Data":"a5b21111171f33754a10def1d21c396d2115b122e7ee378f1e291775f0a72ef1"}
Jul 01 19:22:02.491561 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:02.491528    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:22:03.350662 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:03.350586    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6lm4"
Jul 01 19:22:03.350837 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:03.350691    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6lm4" podUID="98d86579-8161-4185-8cf7-5f427d712b2e"
Jul 01 19:22:03.350837 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:03.350744    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g8pr"
Jul 01 19:22:03.350837 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:03.350818    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7g8pr" podUID="9e551421-5f11-40eb-8a22-e594b6f4eb61"
Jul 01 19:22:03.480125 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:03.480095    2574 generic.go:356] "Generic (PLEG): container finished" podID="1673e32e-d2ee-445b-95d6-144e138343df" containerID="a5b21111171f33754a10def1d21c396d2115b122e7ee378f1e291775f0a72ef1" exitCode=0
Jul 01 19:22:03.480542 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:03.480187    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hnpkl" event={"ID":"1673e32e-d2ee-445b-95d6-144e138343df","Type":"ContainerDied","Data":"a5b21111171f33754a10def1d21c396d2115b122e7ee378f1e291775f0a72ef1"}
Jul 01 19:22:03.483891 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:03.483872    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bplsm_bfb845e6-341d-492e-91c9-9c698a0b8d29/ovn-acl-logging/0.log"
Jul 01 19:22:03.484231 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:03.484209    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bplsm" event={"ID":"bfb845e6-341d-492e-91c9-9c698a0b8d29","Type":"ContainerStarted","Data":"c72a2dee73569a0c51c19e321c920183364de442f6dad5931a4848723caa600d"}
Jul 01 19:22:03.484497 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:03.484472    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:22:03.484606 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:03.484551    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:22:03.499517 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:03.499496    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:22:03.536794 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:03.536750    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bplsm" podStartSLOduration=10.275118869 podStartE2EDuration="29.536737247s" totalImagesPullingTime="19.261618378s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:21:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:03.53665028 +0000 UTC m=+29.628649728" watchObservedRunningTime="2026-07-01 19:22:03.536737247 +0000 UTC m=+29.628736695"
Jul 01 19:22:04.352113 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:04.352088    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:22:04.352320 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:04.352181    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq22h" podUID="f9eaa621-a5fa-49a2-ad92-8dcd5729803a"
Jul 01 19:22:05.350755 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:05.350673    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6lm4"
Jul 01 19:22:05.351121 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:05.350770    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6lm4" podUID="98d86579-8161-4185-8cf7-5f427d712b2e"
Jul 01 19:22:05.351121 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:05.350793    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g8pr"
Jul 01 19:22:05.351121 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:05.350837    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7g8pr" podUID="9e551421-5f11-40eb-8a22-e594b6f4eb61"
Jul 01 19:22:05.491213 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:05.491176    2574 generic.go:356] "Generic (PLEG): container finished" podID="1673e32e-d2ee-445b-95d6-144e138343df" containerID="e77f3da624a755d7ab1c42e2b86e73e6dcac2d7ec3b0cf7d54ba43c0b73fdcda" exitCode=0
Jul 01 19:22:05.491365 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:05.491250    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hnpkl" event={"ID":"1673e32e-d2ee-445b-95d6-144e138343df","Type":"ContainerDied","Data":"e77f3da624a755d7ab1c42e2b86e73e6dcac2d7ec3b0cf7d54ba43c0b73fdcda"}
Jul 01 19:22:06.351257 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:06.351209    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:22:06.351591 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:06.351332    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq22h" podUID="f9eaa621-a5fa-49a2-ad92-8dcd5729803a"
Jul 01 19:22:06.496005 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:06.495968    2574 generic.go:356] "Generic (PLEG): container finished" podID="1673e32e-d2ee-445b-95d6-144e138343df" containerID="9996893e5b80e697a6a3422cc7b9e4fb92324a1cf76fbbe41bff5ff818659b6f" exitCode=0
Jul 01 19:22:06.496166 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:06.496023    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hnpkl" event={"ID":"1673e32e-d2ee-445b-95d6-144e138343df","Type":"ContainerDied","Data":"9996893e5b80e697a6a3422cc7b9e4fb92324a1cf76fbbe41bff5ff818659b6f"}
Jul 01 19:22:07.351216 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:07.351188    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g8pr"
Jul 01 19:22:07.351381 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:07.351192    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6lm4"
Jul 01 19:22:07.351381 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:07.351290    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7g8pr" podUID="9e551421-5f11-40eb-8a22-e594b6f4eb61"
Jul 01 19:22:07.351381 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:07.351364    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6lm4" podUID="98d86579-8161-4185-8cf7-5f427d712b2e"
Jul 01 19:22:08.044203 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:08.044166    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9eaa621-a5fa-49a2-ad92-8dcd5729803a-original-pull-secret\") pod \"global-pull-secret-syncer-mq22h\" (UID: \"f9eaa621-a5fa-49a2-ad92-8dcd5729803a\") " pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:22:08.044386 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:08.044316    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98d86579-8161-4185-8cf7-5f427d712b2e-metrics-certs\") pod \"network-metrics-daemon-m6lm4\" (UID: \"98d86579-8161-4185-8cf7-5f427d712b2e\") " pod="openshift-multus/network-metrics-daemon-m6lm4"
Jul 01 19:22:08.044450 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:08.044380    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 19:22:08.044450 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:08.044425    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:22:08.044557 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:08.044452    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9eaa621-a5fa-49a2-ad92-8dcd5729803a-original-pull-secret podName:f9eaa621-a5fa-49a2-ad92-8dcd5729803a nodeName:}" failed. No retries permitted until 2026-07-01 19:22:40.044433126 +0000 UTC m=+66.136432563 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f9eaa621-a5fa-49a2-ad92-8dcd5729803a-original-pull-secret") pod "global-pull-secret-syncer-mq22h" (UID: "f9eaa621-a5fa-49a2-ad92-8dcd5729803a") : object "kube-system"/"original-pull-secret" not registered
Jul 01 19:22:08.044557 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:08.044482    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98d86579-8161-4185-8cf7-5f427d712b2e-metrics-certs podName:98d86579-8161-4185-8cf7-5f427d712b2e nodeName:}" failed. No retries permitted until 2026-07-01 19:22:40.044460483 +0000 UTC m=+66.136459932 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/98d86579-8161-4185-8cf7-5f427d712b2e-metrics-certs") pod "network-metrics-daemon-m6lm4" (UID: "98d86579-8161-4185-8cf7-5f427d712b2e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:22:08.145352 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:08.145315    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q775\" (UniqueName: \"kubernetes.io/projected/9e551421-5f11-40eb-8a22-e594b6f4eb61-kube-api-access-8q775\") pod \"network-check-target-7g8pr\" (UID: \"9e551421-5f11-40eb-8a22-e594b6f4eb61\") " pod="openshift-network-diagnostics/network-check-target-7g8pr"
Jul 01 19:22:08.145510 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:08.145457    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 19:22:08.145510 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:08.145471    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 19:22:08.145510 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:08.145480    2574 projected.go:196] Error preparing data for projected volume kube-api-access-8q775 for pod openshift-network-diagnostics/network-check-target-7g8pr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:22:08.145622 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:08.145529    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e551421-5f11-40eb-8a22-e594b6f4eb61-kube-api-access-8q775 podName:9e551421-5f11-40eb-8a22-e594b6f4eb61 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:40.145514306 +0000 UTC m=+66.237513733 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8q775" (UniqueName: "kubernetes.io/projected/9e551421-5f11-40eb-8a22-e594b6f4eb61-kube-api-access-8q775") pod "network-check-target-7g8pr" (UID: "9e551421-5f11-40eb-8a22-e594b6f4eb61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:22:08.351176 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:08.351149    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:22:08.351442 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:08.351282    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq22h" podUID="f9eaa621-a5fa-49a2-ad92-8dcd5729803a"
Jul 01 19:22:09.351511 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:09.351473    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6lm4"
Jul 01 19:22:09.351991 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:09.351473    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g8pr"
Jul 01 19:22:09.351991 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:09.351588    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6lm4" podUID="98d86579-8161-4185-8cf7-5f427d712b2e"
Jul 01 19:22:09.351991 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:09.351717    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7g8pr" podUID="9e551421-5f11-40eb-8a22-e594b6f4eb61"
Jul 01 19:22:10.351525 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:10.351491    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:22:10.352005 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:10.351634    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq22h" podUID="f9eaa621-a5fa-49a2-ad92-8dcd5729803a"
Jul 01 19:22:11.351175 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:11.351147    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g8pr"
Jul 01 19:22:11.351406 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:11.351271    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7g8pr" podUID="9e551421-5f11-40eb-8a22-e594b6f4eb61"
Jul 01 19:22:11.351406 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:11.351321    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6lm4"
Jul 01 19:22:11.351494 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:11.351402    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6lm4" podUID="98d86579-8161-4185-8cf7-5f427d712b2e"
Jul 01 19:22:12.356195 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:12.356170    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:22:12.356610 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:12.356269    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq22h" podUID="f9eaa621-a5fa-49a2-ad92-8dcd5729803a"
Jul 01 19:22:12.512265 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:12.512234    2574 generic.go:356] "Generic (PLEG): container finished" podID="1673e32e-d2ee-445b-95d6-144e138343df" containerID="9dda5fa4a4dcbe7d58f668e09b102b6f58fc4559899ca05a6dc46782621e9394" exitCode=0
Jul 01 19:22:12.512413 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:12.512297    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hnpkl" event={"ID":"1673e32e-d2ee-445b-95d6-144e138343df","Type":"ContainerDied","Data":"9dda5fa4a4dcbe7d58f668e09b102b6f58fc4559899ca05a6dc46782621e9394"}
Jul 01 19:22:13.350662 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:13.350624    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6lm4"
Jul 01 19:22:13.350662 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:13.350670    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g8pr"
Jul 01 19:22:13.350942 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:13.350777    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6lm4" podUID="98d86579-8161-4185-8cf7-5f427d712b2e"
Jul 01 19:22:13.350942 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:13.350895    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7g8pr" podUID="9e551421-5f11-40eb-8a22-e594b6f4eb61"
Jul 01 19:22:13.517650 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:13.517618    2574 generic.go:356] "Generic (PLEG): container finished" podID="1673e32e-d2ee-445b-95d6-144e138343df" containerID="560b2f018b45b03baa89ee6e87ea1ee164b20726c53eb4f4751674665e1e04a6" exitCode=0
Jul 01 19:22:13.518061 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:13.517684    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hnpkl" event={"ID":"1673e32e-d2ee-445b-95d6-144e138343df","Type":"ContainerDied","Data":"560b2f018b45b03baa89ee6e87ea1ee164b20726c53eb4f4751674665e1e04a6"}
Jul 01 19:22:14.351961 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:14.351930    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:22:14.352130 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:14.352015    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq22h" podUID="f9eaa621-a5fa-49a2-ad92-8dcd5729803a"
Jul 01 19:22:14.522635 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:14.522596    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hnpkl" event={"ID":"1673e32e-d2ee-445b-95d6-144e138343df","Type":"ContainerStarted","Data":"49a5105f0e478eb5a35b263f0f1fddc8aba52120fa8aa7c8e78143c4f0f2dc61"}
Jul 01 19:22:14.550134 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:14.550090    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hnpkl" podStartSLOduration=9.266218706 podStartE2EDuration="40.550076595s" totalImagesPullingTime="31.283857889s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:21:34 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:14.548724191 +0000 UTC m=+40.640723640" watchObservedRunningTime="2026-07-01 19:22:14.550076595 +0000 UTC m=+40.642076042"
Jul 01 19:22:15.350483 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:15.350457    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6lm4"
Jul 01 19:22:15.350483 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:15.350479    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g8pr"
Jul 01 19:22:15.350689 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:15.350555    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6lm4" podUID="98d86579-8161-4185-8cf7-5f427d712b2e"
Jul 01 19:22:15.350689 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:15.350659    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7g8pr" podUID="9e551421-5f11-40eb-8a22-e594b6f4eb61"
Jul 01 19:22:16.351106 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:16.351072    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:22:16.351497 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:16.351178    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq22h" podUID="f9eaa621-a5fa-49a2-ad92-8dcd5729803a"
Jul 01 19:22:17.350796 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:17.350765    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6lm4"
Jul 01 19:22:17.350988 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:17.350765    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g8pr"
Jul 01 19:22:17.350988 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:17.350888    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6lm4" podUID="98d86579-8161-4185-8cf7-5f427d712b2e"
Jul 01 19:22:17.350988 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:17.350976    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7g8pr" podUID="9e551421-5f11-40eb-8a22-e594b6f4eb61"
Jul 01 19:22:18.351398 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:18.351366    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:22:18.351782 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:18.351497    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq22h" podUID="f9eaa621-a5fa-49a2-ad92-8dcd5729803a"
Jul 01 19:22:19.351404 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:19.351367    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6lm4"
Jul 01 19:22:19.351838 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:19.351474    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6lm4" podUID="98d86579-8161-4185-8cf7-5f427d712b2e"
Jul 01 19:22:19.351838 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:19.351538    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g8pr"
Jul 01 19:22:19.351838 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:19.351646    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7g8pr" podUID="9e551421-5f11-40eb-8a22-e594b6f4eb61"
Jul 01 19:22:20.351235 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:20.351195    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:22:20.351390 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:20.351313    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq22h" podUID="f9eaa621-a5fa-49a2-ad92-8dcd5729803a"
Jul 01 19:22:21.005546 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:21.005355    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mq22h"]
Jul 01 19:22:21.006092 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:21.005635    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:22:21.006092 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:21.005724    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq22h" podUID="f9eaa621-a5fa-49a2-ad92-8dcd5729803a"
Jul 01 19:22:21.007875 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:21.007848    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7g8pr"]
Jul 01 19:22:21.008016 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:21.007971    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g8pr"
Jul 01 19:22:21.008080 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:21.008056    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7g8pr" podUID="9e551421-5f11-40eb-8a22-e594b6f4eb61"
Jul 01 19:22:21.021541 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:21.021509    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m6lm4"]
Jul 01 19:22:21.021661 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:21.021603    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6lm4"
Jul 01 19:22:21.021711 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:21.021693    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6lm4" podUID="98d86579-8161-4185-8cf7-5f427d712b2e"
Jul 01 19:22:22.354148 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:22.354119    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:22:22.354655 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:22.354221    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq22h" podUID="f9eaa621-a5fa-49a2-ad92-8dcd5729803a"
Jul 01 19:22:22.354655 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:22.354269    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6lm4"
Jul 01 19:22:22.354655 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:22.354359    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6lm4" podUID="98d86579-8161-4185-8cf7-5f427d712b2e"
Jul 01 19:22:23.351352 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:23.351330    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g8pr"
Jul 01 19:22:23.351478 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:23.351427    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7g8pr" podUID="9e551421-5f11-40eb-8a22-e594b6f4eb61"
Jul 01 19:22:24.351786 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:24.351758    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6lm4"
Jul 01 19:22:24.352296 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:24.351859    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6lm4" podUID="98d86579-8161-4185-8cf7-5f427d712b2e"
Jul 01 19:22:24.352296 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:24.351916    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:22:24.352296 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:24.352026    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq22h" podUID="f9eaa621-a5fa-49a2-ad92-8dcd5729803a"
Jul 01 19:22:25.140866 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.140838    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-164.ec2.internal" event="NodeReady"
Jul 01 19:22:25.141088 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.140973    2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 01 19:22:25.226504 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.226475    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-n29hh"]
Jul 01 19:22:25.239391 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.239366    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ndjvj"]
Jul 01 19:22:25.239524 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.239507    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n29hh"
Jul 01 19:22:25.242014 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.241995    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 01 19:22:25.242126 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.242068    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 01 19:22:25.242184 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.242167    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-99qlb\""
Jul 01 19:22:25.251732 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.251711    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ndjvj"
Jul 01 19:22:25.257784 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.257758    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 01 19:22:25.258069 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.258053    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-45tq7\""
Jul 01 19:22:25.258804 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.258784    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n29hh"]
Jul 01 19:22:25.259446 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.259424    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ndjvj"]
Jul 01 19:22:25.261549 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.261531    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 01 19:22:25.264770 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.264753    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 01 19:22:25.267972 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.267952    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bvvz\" (UniqueName: \"kubernetes.io/projected/58b6a949-f2a2-41a0-9f47-79c420dd33ef-kube-api-access-7bvvz\") pod \"dns-default-n29hh\" (UID: \"58b6a949-f2a2-41a0-9f47-79c420dd33ef\") " pod="openshift-dns/dns-default-n29hh"
Jul 01 19:22:25.268079 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.267988    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58b6a949-f2a2-41a0-9f47-79c420dd33ef-metrics-tls\") pod \"dns-default-n29hh\" (UID: \"58b6a949-f2a2-41a0-9f47-79c420dd33ef\") " pod="openshift-dns/dns-default-n29hh"
Jul 01 19:22:25.268079 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.268014    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0e57b1d-40fc-4eef-86f4-51970c68b93b-cert\") pod \"ingress-canary-ndjvj\" (UID: \"a0e57b1d-40fc-4eef-86f4-51970c68b93b\") " pod="openshift-ingress-canary/ingress-canary-ndjvj"
Jul 01 19:22:25.268079 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.268049    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58b6a949-f2a2-41a0-9f47-79c420dd33ef-tmp-dir\") pod \"dns-default-n29hh\" (UID: \"58b6a949-f2a2-41a0-9f47-79c420dd33ef\") " pod="openshift-dns/dns-default-n29hh"
Jul 01 19:22:25.268227 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.268104    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4l62\" (UniqueName: \"kubernetes.io/projected/a0e57b1d-40fc-4eef-86f4-51970c68b93b-kube-api-access-v4l62\") pod \"ingress-canary-ndjvj\" (UID: \"a0e57b1d-40fc-4eef-86f4-51970c68b93b\") " pod="openshift-ingress-canary/ingress-canary-ndjvj"
Jul 01 19:22:25.268227 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.268167    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/58b6a949-f2a2-41a0-9f47-79c420dd33ef-config-volume\") pod \"dns-default-n29hh\" (UID: \"58b6a949-f2a2-41a0-9f47-79c420dd33ef\") " pod="openshift-dns/dns-default-n29hh"
Jul 01 19:22:25.316487 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.316463    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-n8l46"]
Jul 01 19:22:25.328108 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.328086    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-n8l46"
Jul 01 19:22:25.330516 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.330495    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-h6qs4\""
Jul 01 19:22:25.330957 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.330937    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 01 19:22:25.331053 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.330939    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 01 19:22:25.331053 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.330942    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 01 19:22:25.331053 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.331009    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 01 19:22:25.332781 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.332759    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-n8l46"]
Jul 01 19:22:25.350705 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.350682    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g8pr"
Jul 01 19:22:25.356522 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.356504    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 01 19:22:25.356853 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.356780    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 01 19:22:25.356853 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.356833    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-j4bvv\""
Jul 01 19:22:25.369254 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.369236    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bvvz\" (UniqueName: \"kubernetes.io/projected/58b6a949-f2a2-41a0-9f47-79c420dd33ef-kube-api-access-7bvvz\") pod \"dns-default-n29hh\" (UID: \"58b6a949-f2a2-41a0-9f47-79c420dd33ef\") " pod="openshift-dns/dns-default-n29hh"
Jul 01 19:22:25.369401 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.369265    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3359c7fa-a51a-4dc7-b46e-c99a59154ba9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n8l46\" (UID: \"3359c7fa-a51a-4dc7-b46e-c99a59154ba9\") " pod="openshift-insights/insights-runtime-extractor-n8l46"
Jul 01 19:22:25.369401 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.369287    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58b6a949-f2a2-41a0-9f47-79c420dd33ef-metrics-tls\") pod \"dns-default-n29hh\" (UID: \"58b6a949-f2a2-41a0-9f47-79c420dd33ef\") " pod="openshift-dns/dns-default-n29hh"
Jul 01 19:22:25.369401 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.369314    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3359c7fa-a51a-4dc7-b46e-c99a59154ba9-data-volume\") pod \"insights-runtime-extractor-n8l46\" (UID: \"3359c7fa-a51a-4dc7-b46e-c99a59154ba9\") " pod="openshift-insights/insights-runtime-extractor-n8l46"
Jul 01 19:22:25.369401 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.369340    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0e57b1d-40fc-4eef-86f4-51970c68b93b-cert\") pod \"ingress-canary-ndjvj\" (UID: \"a0e57b1d-40fc-4eef-86f4-51970c68b93b\") " pod="openshift-ingress-canary/ingress-canary-ndjvj"
Jul 01 19:22:25.369401 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.369372    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58b6a949-f2a2-41a0-9f47-79c420dd33ef-tmp-dir\") pod \"dns-default-n29hh\" (UID: \"58b6a949-f2a2-41a0-9f47-79c420dd33ef\") " pod="openshift-dns/dns-default-n29hh"
Jul 01 19:22:25.369401 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.369396    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4l62\" (UniqueName: \"kubernetes.io/projected/a0e57b1d-40fc-4eef-86f4-51970c68b93b-kube-api-access-v4l62\") pod \"ingress-canary-ndjvj\" (UID: \"a0e57b1d-40fc-4eef-86f4-51970c68b93b\") " pod="openshift-ingress-canary/ingress-canary-ndjvj"
Jul 01 19:22:25.369702 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.369447    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/58b6a949-f2a2-41a0-9f47-79c420dd33ef-config-volume\") pod \"dns-default-n29hh\" (UID: \"58b6a949-f2a2-41a0-9f47-79c420dd33ef\") " pod="openshift-dns/dns-default-n29hh"
Jul 01 19:22:25.369702 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.369471    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3359c7fa-a51a-4dc7-b46e-c99a59154ba9-crio-socket\") pod \"insights-runtime-extractor-n8l46\" (UID: \"3359c7fa-a51a-4dc7-b46e-c99a59154ba9\") " pod="openshift-insights/insights-runtime-extractor-n8l46"
Jul 01 19:22:25.369702 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.369499    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3359c7fa-a51a-4dc7-b46e-c99a59154ba9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n8l46\" (UID: \"3359c7fa-a51a-4dc7-b46e-c99a59154ba9\") " pod="openshift-insights/insights-runtime-extractor-n8l46"
Jul 01 19:22:25.369702 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.369553    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crhrc\" (UniqueName: \"kubernetes.io/projected/3359c7fa-a51a-4dc7-b46e-c99a59154ba9-kube-api-access-crhrc\") pod \"insights-runtime-extractor-n8l46\" (UID: \"3359c7fa-a51a-4dc7-b46e-c99a59154ba9\") " pod="openshift-insights/insights-runtime-extractor-n8l46"
Jul 01 19:22:25.369834 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.369758    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58b6a949-f2a2-41a0-9f47-79c420dd33ef-tmp-dir\") pod \"dns-default-n29hh\" (UID: \"58b6a949-f2a2-41a0-9f47-79c420dd33ef\") " pod="openshift-dns/dns-default-n29hh"
Jul 01 19:22:25.369987 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.369969    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/58b6a949-f2a2-41a0-9f47-79c420dd33ef-config-volume\") pod \"dns-default-n29hh\" (UID: \"58b6a949-f2a2-41a0-9f47-79c420dd33ef\") " pod="openshift-dns/dns-default-n29hh"
Jul 01 19:22:25.373210 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.373193    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58b6a949-f2a2-41a0-9f47-79c420dd33ef-metrics-tls\") pod \"dns-default-n29hh\" (UID: \"58b6a949-f2a2-41a0-9f47-79c420dd33ef\") " pod="openshift-dns/dns-default-n29hh"
Jul 01 19:22:25.373210 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.373204    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0e57b1d-40fc-4eef-86f4-51970c68b93b-cert\") pod \"ingress-canary-ndjvj\" (UID: \"a0e57b1d-40fc-4eef-86f4-51970c68b93b\") " pod="openshift-ingress-canary/ingress-canary-ndjvj"
Jul 01 19:22:25.378700 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.378680    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bvvz\" (UniqueName: \"kubernetes.io/projected/58b6a949-f2a2-41a0-9f47-79c420dd33ef-kube-api-access-7bvvz\") pod \"dns-default-n29hh\" (UID: \"58b6a949-f2a2-41a0-9f47-79c420dd33ef\") " pod="openshift-dns/dns-default-n29hh"
Jul 01 19:22:25.378780 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.378693    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4l62\" (UniqueName: \"kubernetes.io/projected/a0e57b1d-40fc-4eef-86f4-51970c68b93b-kube-api-access-v4l62\") pod \"ingress-canary-ndjvj\" (UID: \"a0e57b1d-40fc-4eef-86f4-51970c68b93b\") " pod="openshift-ingress-canary/ingress-canary-ndjvj"
Jul 01 19:22:25.469892 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.469808    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3359c7fa-a51a-4dc7-b46e-c99a59154ba9-crio-socket\") pod \"insights-runtime-extractor-n8l46\" (UID: \"3359c7fa-a51a-4dc7-b46e-c99a59154ba9\") " pod="openshift-insights/insights-runtime-extractor-n8l46"
Jul 01 19:22:25.469892 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.469843    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3359c7fa-a51a-4dc7-b46e-c99a59154ba9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n8l46\" (UID: \"3359c7fa-a51a-4dc7-b46e-c99a59154ba9\") " pod="openshift-insights/insights-runtime-extractor-n8l46"
Jul 01 19:22:25.469892 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.469865    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crhrc\" (UniqueName: \"kubernetes.io/projected/3359c7fa-a51a-4dc7-b46e-c99a59154ba9-kube-api-access-crhrc\") pod \"insights-runtime-extractor-n8l46\" (UID: \"3359c7fa-a51a-4dc7-b46e-c99a59154ba9\") " pod="openshift-insights/insights-runtime-extractor-n8l46"
Jul 01 19:22:25.469892 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.469895    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3359c7fa-a51a-4dc7-b46e-c99a59154ba9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n8l46\" (UID: \"3359c7fa-a51a-4dc7-b46e-c99a59154ba9\") " pod="openshift-insights/insights-runtime-extractor-n8l46"
Jul 01 19:22:25.470236 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.469927    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3359c7fa-a51a-4dc7-b46e-c99a59154ba9-data-volume\") pod \"insights-runtime-extractor-n8l46\" (UID: \"3359c7fa-a51a-4dc7-b46e-c99a59154ba9\") " pod="openshift-insights/insights-runtime-extractor-n8l46"
Jul 01 19:22:25.470236 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.470037    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3359c7fa-a51a-4dc7-b46e-c99a59154ba9-crio-socket\") pod \"insights-runtime-extractor-n8l46\" (UID: \"3359c7fa-a51a-4dc7-b46e-c99a59154ba9\") " pod="openshift-insights/insights-runtime-extractor-n8l46"
Jul 01 19:22:25.470336 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.470265    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3359c7fa-a51a-4dc7-b46e-c99a59154ba9-data-volume\") pod \"insights-runtime-extractor-n8l46\" (UID: \"3359c7fa-a51a-4dc7-b46e-c99a59154ba9\") " pod="openshift-insights/insights-runtime-extractor-n8l46"
Jul 01 19:22:25.470574 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.470550    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3359c7fa-a51a-4dc7-b46e-c99a59154ba9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n8l46\" (UID: \"3359c7fa-a51a-4dc7-b46e-c99a59154ba9\") " pod="openshift-insights/insights-runtime-extractor-n8l46"
Jul 01 19:22:25.472284 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.472267    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3359c7fa-a51a-4dc7-b46e-c99a59154ba9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n8l46\" (UID: \"3359c7fa-a51a-4dc7-b46e-c99a59154ba9\") " pod="openshift-insights/insights-runtime-extractor-n8l46"
Jul 01 19:22:25.481126 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.481107    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crhrc\" (UniqueName: \"kubernetes.io/projected/3359c7fa-a51a-4dc7-b46e-c99a59154ba9-kube-api-access-crhrc\") pod \"insights-runtime-extractor-n8l46\" (UID: \"3359c7fa-a51a-4dc7-b46e-c99a59154ba9\") " pod="openshift-insights/insights-runtime-extractor-n8l46"
Jul 01 19:22:25.548102 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.548077    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n29hh"
Jul 01 19:22:25.560335 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.560312    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ndjvj"
Jul 01 19:22:25.637388 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.637208    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-n8l46"
Jul 01 19:22:25.766283 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.766252    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ndjvj"]
Jul 01 19:22:25.774261 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.774193    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n29hh"]
Jul 01 19:22:25.776596 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:22:25.776574    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58b6a949_f2a2_41a0_9f47_79c420dd33ef.slice/crio-52d7c238ede4298e15a2cd200a0eab40d8e8bbf8e06c71797686718637c76080 WatchSource:0}: Error finding container 52d7c238ede4298e15a2cd200a0eab40d8e8bbf8e06c71797686718637c76080: Status 404 returned error can't find the container with id 52d7c238ede4298e15a2cd200a0eab40d8e8bbf8e06c71797686718637c76080
Jul 01 19:22:25.797609 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:25.797586    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-n8l46"]
Jul 01 19:22:25.800449 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:22:25.800425    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3359c7fa_a51a_4dc7_b46e_c99a59154ba9.slice/crio-cb1b638afbcc1fe87f17c6f5695736b5594aa3e90de5f0a59537d88868a145de WatchSource:0}: Error finding container cb1b638afbcc1fe87f17c6f5695736b5594aa3e90de5f0a59537d88868a145de: Status 404 returned error can't find the container with id cb1b638afbcc1fe87f17c6f5695736b5594aa3e90de5f0a59537d88868a145de
Jul 01 19:22:26.354754 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:26.354728    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6lm4"
Jul 01 19:22:26.354968 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:26.354792    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:22:26.358440 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:26.358269    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k4h7l\""
Jul 01 19:22:26.358440 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:26.358303    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 01 19:22:26.358440 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:26.358342    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 01 19:22:26.550792 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:26.550757    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ndjvj" event={"ID":"a0e57b1d-40fc-4eef-86f4-51970c68b93b","Type":"ContainerStarted","Data":"f2f3343b8621566ca61334ed9e16787c698e657efbe95f38e44107e99c6a4d76"}
Jul 01 19:22:26.552197 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:26.552166    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n29hh" event={"ID":"58b6a949-f2a2-41a0-9f47-79c420dd33ef","Type":"ContainerStarted","Data":"52d7c238ede4298e15a2cd200a0eab40d8e8bbf8e06c71797686718637c76080"}
Jul 01 19:22:26.553408 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:26.553380    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n8l46" event={"ID":"3359c7fa-a51a-4dc7-b46e-c99a59154ba9","Type":"ContainerStarted","Data":"cb1b638afbcc1fe87f17c6f5695736b5594aa3e90de5f0a59537d88868a145de"}
Jul 01 19:22:26.703100 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:26.702967    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7b7d896497-8wxbh"]
Jul 01 19:22:26.717156 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:26.717128    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7b7d896497-8wxbh"
Jul 01 19:22:26.718993 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:26.718948    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7b7d896497-8wxbh"]
Jul 01 19:22:26.720510 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:26.720475    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 01 19:22:26.720736 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:26.720719    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 01 19:22:26.720969 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:26.720953    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 01 19:22:26.721096 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:26.721057    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-cv4s5\""
Jul 01 19:22:26.721210 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:26.721196    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 01 19:22:26.721449 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:26.721419    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 01 19:22:26.781713 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:26.781540    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2063d45-dbf3-402d-8a70-ea43861e5380-metrics-client-ca\") pod \"prometheus-operator-7b7d896497-8wxbh\" (UID: \"c2063d45-dbf3-402d-8a70-ea43861e5380\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-8wxbh"
Jul 01 19:22:26.781713 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:26.781598    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2063d45-dbf3-402d-8a70-ea43861e5380-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-8wxbh\" (UID: \"c2063d45-dbf3-402d-8a70-ea43861e5380\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-8wxbh"
Jul 01 19:22:26.781713 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:26.781670    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c2063d45-dbf3-402d-8a70-ea43861e5380-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7b7d896497-8wxbh\" (UID: \"c2063d45-dbf3-402d-8a70-ea43861e5380\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-8wxbh"
Jul 01 19:22:26.781713 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:26.781702    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pklsc\" (UniqueName: \"kubernetes.io/projected/c2063d45-dbf3-402d-8a70-ea43861e5380-kube-api-access-pklsc\") pod \"prometheus-operator-7b7d896497-8wxbh\" (UID: \"c2063d45-dbf3-402d-8a70-ea43861e5380\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-8wxbh"
Jul 01 19:22:26.882583 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:26.882534    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2063d45-dbf3-402d-8a70-ea43861e5380-metrics-client-ca\") pod \"prometheus-operator-7b7d896497-8wxbh\" (UID: \"c2063d45-dbf3-402d-8a70-ea43861e5380\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-8wxbh"
Jul 01 19:22:26.883294 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:26.883270    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2063d45-dbf3-402d-8a70-ea43861e5380-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-8wxbh\" (UID: \"c2063d45-dbf3-402d-8a70-ea43861e5380\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-8wxbh"
Jul 01 19:22:26.883406 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:26.883349    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c2063d45-dbf3-402d-8a70-ea43861e5380-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7b7d896497-8wxbh\" (UID: \"c2063d45-dbf3-402d-8a70-ea43861e5380\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-8wxbh"
Jul 01 19:22:26.883406 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:26.883387    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pklsc\" (UniqueName: \"kubernetes.io/projected/c2063d45-dbf3-402d-8a70-ea43861e5380-kube-api-access-pklsc\") pod \"prometheus-operator-7b7d896497-8wxbh\" (UID: \"c2063d45-dbf3-402d-8a70-ea43861e5380\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-8wxbh"
Jul 01 19:22:26.884759 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:26.884734    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2063d45-dbf3-402d-8a70-ea43861e5380-metrics-client-ca\") pod \"prometheus-operator-7b7d896497-8wxbh\" (UID: \"c2063d45-dbf3-402d-8a70-ea43861e5380\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-8wxbh"
Jul 01 19:22:26.889827 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:26.889802    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2063d45-dbf3-402d-8a70-ea43861e5380-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-8wxbh\" (UID: \"c2063d45-dbf3-402d-8a70-ea43861e5380\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-8wxbh"
Jul 01 19:22:26.892952 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:26.892890    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pklsc\" (UniqueName: \"kubernetes.io/projected/c2063d45-dbf3-402d-8a70-ea43861e5380-kube-api-access-pklsc\") pod \"prometheus-operator-7b7d896497-8wxbh\" (UID: \"c2063d45-dbf3-402d-8a70-ea43861e5380\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-8wxbh"
Jul 01 19:22:26.893051 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:26.892972    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c2063d45-dbf3-402d-8a70-ea43861e5380-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7b7d896497-8wxbh\" (UID: \"c2063d45-dbf3-402d-8a70-ea43861e5380\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-8wxbh"
Jul 01 19:22:27.030308 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:27.030172    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7b7d896497-8wxbh"
Jul 01 19:22:27.238076 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:27.236844    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7b7d896497-8wxbh"]
Jul 01 19:22:27.965321 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:22:27.965274    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2063d45_dbf3_402d_8a70_ea43861e5380.slice/crio-b61eb9b278a3ff0333aef5f6e74b8b84d286f321f8f072a183743afe835569a2 WatchSource:0}: Error finding container b61eb9b278a3ff0333aef5f6e74b8b84d286f321f8f072a183743afe835569a2: Status 404 returned error can't find the container with id b61eb9b278a3ff0333aef5f6e74b8b84d286f321f8f072a183743afe835569a2
Jul 01 19:22:28.561519 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:28.561479    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7b7d896497-8wxbh" event={"ID":"c2063d45-dbf3-402d-8a70-ea43861e5380","Type":"ContainerStarted","Data":"b61eb9b278a3ff0333aef5f6e74b8b84d286f321f8f072a183743afe835569a2"}
Jul 01 19:22:30.522129 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:30.522050    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5499b6fb8f-g2fv2"]
Jul 01 19:22:30.529765 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:30.529725    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5499b6fb8f-g2fv2"
Jul 01 19:22:30.532666 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:30.532607    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 01 19:22:30.532666 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:30.532634    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 01 19:22:30.532666 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:30.532636    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-mmr57\""
Jul 01 19:22:30.539484 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:30.539443    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5499b6fb8f-g2fv2"]
Jul 01 19:22:30.619564 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:30.619530    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pphw2\" (UniqueName: \"kubernetes.io/projected/d20bf32b-d17e-47a2-a6d0-809d3e465a08-kube-api-access-pphw2\") pod \"downloads-5499b6fb8f-g2fv2\" (UID: \"d20bf32b-d17e-47a2-a6d0-809d3e465a08\") " pod="openshift-console/downloads-5499b6fb8f-g2fv2"
Jul 01 19:22:30.720914 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:30.720870    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pphw2\" (UniqueName: \"kubernetes.io/projected/d20bf32b-d17e-47a2-a6d0-809d3e465a08-kube-api-access-pphw2\") pod \"downloads-5499b6fb8f-g2fv2\" (UID: \"d20bf32b-d17e-47a2-a6d0-809d3e465a08\") " pod="openshift-console/downloads-5499b6fb8f-g2fv2"
Jul 01 19:22:30.737760 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:30.737718    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pphw2\" (UniqueName: \"kubernetes.io/projected/d20bf32b-d17e-47a2-a6d0-809d3e465a08-kube-api-access-pphw2\") pod \"downloads-5499b6fb8f-g2fv2\" (UID: \"d20bf32b-d17e-47a2-a6d0-809d3e465a08\") " pod="openshift-console/downloads-5499b6fb8f-g2fv2"
Jul 01 19:22:30.840406 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:30.840307    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5499b6fb8f-g2fv2"
Jul 01 19:22:31.546180 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:31.546040    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5499b6fb8f-g2fv2"]
Jul 01 19:22:31.557755 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:22:31.557380    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd20bf32b_d17e_47a2_a6d0_809d3e465a08.slice/crio-e925903674318075ee9b7f8fc8abf85364465ab1e05e011bc653e5095ae4c33f WatchSource:0}: Error finding container e925903674318075ee9b7f8fc8abf85364465ab1e05e011bc653e5095ae4c33f: Status 404 returned error can't find the container with id e925903674318075ee9b7f8fc8abf85364465ab1e05e011bc653e5095ae4c33f
Jul 01 19:22:31.576836 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:31.576803    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n8l46" event={"ID":"3359c7fa-a51a-4dc7-b46e-c99a59154ba9","Type":"ContainerStarted","Data":"d848b291486981068677548f38b6f19a7df51b1a2c31d8d4c6774cd4bd433055"}
Jul 01 19:22:31.579870 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:31.579812    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7b7d896497-8wxbh" event={"ID":"c2063d45-dbf3-402d-8a70-ea43861e5380","Type":"ContainerStarted","Data":"262ade9cd3907ff3f891b534105d6bb4f269b32a698c304ea567dfb6d9a90e19"}
Jul 01 19:22:31.582696 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:31.582628    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n29hh" event={"ID":"58b6a949-f2a2-41a0-9f47-79c420dd33ef","Type":"ContainerStarted","Data":"0ceb83b3b21bd3d6c9a8e1d07eb9216db45c078cec41e71a4b44fa815ec889b7"}
Jul 01 19:22:31.586690 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:31.586605    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ndjvj" event={"ID":"a0e57b1d-40fc-4eef-86f4-51970c68b93b","Type":"ContainerStarted","Data":"4b522f278131a5f0bd8535a6cbf606987c7b3658804d84244bc2b3c8bcb4c937"}
Jul 01 19:22:31.588543 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:31.588509    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5499b6fb8f-g2fv2" event={"ID":"d20bf32b-d17e-47a2-a6d0-809d3e465a08","Type":"ContainerStarted","Data":"e925903674318075ee9b7f8fc8abf85364465ab1e05e011bc653e5095ae4c33f"}
Jul 01 19:22:31.629787 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:31.629659    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ndjvj" podStartSLOduration=1.028659934 podStartE2EDuration="6.629639325s" totalImagesPullingTime="5.600979391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:22:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:31.628757628 +0000 UTC m=+57.720757076" watchObservedRunningTime="2026-07-01 19:22:31.629639325 +0000 UTC m=+57.721638775"
Jul 01 19:22:31.629956 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:31.629781    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7b7d896497-8wxbh" podStartSLOduration=2.21853698 podStartE2EDuration="5.629773255s" totalImagesPullingTime="3.411236275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:22:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:31.603150004 +0000 UTC m=+57.695149452" watchObservedRunningTime="2026-07-01 19:22:31.629773255 +0000 UTC m=+57.721772704"
Jul 01 19:22:32.593665 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:32.593577    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n29hh" event={"ID":"58b6a949-f2a2-41a0-9f47-79c420dd33ef","Type":"ContainerStarted","Data":"ca0c8f3962ce411196297d20f790ac047d733534b207931491fdebfd90e42141"}
Jul 01 19:22:32.594101 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:32.593675    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-n29hh"
Jul 01 19:22:32.595337 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:32.595318    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n8l46" event={"ID":"3359c7fa-a51a-4dc7-b46e-c99a59154ba9","Type":"ContainerStarted","Data":"34ab9f3a8cf1ccfa10467b78a6a815469cacce510e08694484609daf2af69b8d"}
Jul 01 19:22:32.596947 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:32.596918    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7b7d896497-8wxbh" event={"ID":"c2063d45-dbf3-402d-8a70-ea43861e5380","Type":"ContainerStarted","Data":"b69da86f9d87d89db0faa90eb1d346fab75a7dc13e5dda04292aa8b35e8edc3e"}
Jul 01 19:22:32.612720 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:32.612677    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-n29hh" podStartSLOduration=2.663282696 podStartE2EDuration="7.612664929s" totalImagesPullingTime="4.949382233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:22:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:32.611378181 +0000 UTC m=+58.703377651" watchObservedRunningTime="2026-07-01 19:22:32.612664929 +0000 UTC m=+58.704664368"
Jul 01 19:22:33.173087 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.173052    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-cxsgw"]
Jul 01 19:22:33.175410 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.175383    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-cxsgw"
Jul 01 19:22:33.179196 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.178941    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-s65n5\""
Jul 01 19:22:33.179196 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.179168    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 01 19:22:33.179383 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.179174    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 01 19:22:33.186001 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.184824    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-gjdxq"]
Jul 01 19:22:33.187222 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.187198    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-845b954455-gjdxq"
Jul 01 19:22:33.189753 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.189573    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 01 19:22:33.190274 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.189878    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 01 19:22:33.190274 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.189935    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 01 19:22:33.190274 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.190235    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-j8zd6\""
Jul 01 19:22:33.199782 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.199757    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-cxsgw"]
Jul 01 19:22:33.206742 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.206723    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-gjdxq"]
Jul 01 19:22:33.232802 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.230843    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ntch4"]
Jul 01 19:22:33.243489 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.243453    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ntch4"
Jul 01 19:22:33.243606 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.243581    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ceea4b5-bcc4-405c-9364-c2dc815b0c97-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-cxsgw\" (UID: \"2ceea4b5-bcc4-405c-9364-c2dc815b0c97\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-cxsgw"
Jul 01 19:22:33.243667 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.243623    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ceea4b5-bcc4-405c-9364-c2dc815b0c97-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-cxsgw\" (UID: \"2ceea4b5-bcc4-405c-9364-c2dc815b0c97\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-cxsgw"
Jul 01 19:22:33.243727 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.243664    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/074f32fe-7406-4e01-9868-da1ec706d589-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-gjdxq\" (UID: \"074f32fe-7406-4e01-9868-da1ec706d589\") " pod="openshift-monitoring/kube-state-metrics-845b954455-gjdxq"
Jul 01 19:22:33.243727 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.243694    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/074f32fe-7406-4e01-9868-da1ec706d589-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-gjdxq\" (UID: \"074f32fe-7406-4e01-9868-da1ec706d589\") " pod="openshift-monitoring/kube-state-metrics-845b954455-gjdxq"
Jul 01 19:22:33.243829 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.243753    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ceea4b5-bcc4-405c-9364-c2dc815b0c97-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-cxsgw\" (UID: \"2ceea4b5-bcc4-405c-9364-c2dc815b0c97\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-cxsgw"
Jul 01 19:22:33.243829 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.243793    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n4p5\" (UniqueName: \"kubernetes.io/projected/2ceea4b5-bcc4-405c-9364-c2dc815b0c97-kube-api-access-8n4p5\") pod \"openshift-state-metrics-57f99c8b49-cxsgw\" (UID: \"2ceea4b5-bcc4-405c-9364-c2dc815b0c97\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-cxsgw"
Jul 01 19:22:33.243965 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.243830    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/074f32fe-7406-4e01-9868-da1ec706d589-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-gjdxq\" (UID: \"074f32fe-7406-4e01-9868-da1ec706d589\") " pod="openshift-monitoring/kube-state-metrics-845b954455-gjdxq"
Jul 01 19:22:33.243965 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.243854    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/074f32fe-7406-4e01-9868-da1ec706d589-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-gjdxq\" (UID: \"074f32fe-7406-4e01-9868-da1ec706d589\") " pod="openshift-monitoring/kube-state-metrics-845b954455-gjdxq"
Jul 01 19:22:33.243965 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.243880    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/074f32fe-7406-4e01-9868-da1ec706d589-metrics-client-ca\") pod \"kube-state-metrics-845b954455-gjdxq\" (UID: \"074f32fe-7406-4e01-9868-da1ec706d589\") " pod="openshift-monitoring/kube-state-metrics-845b954455-gjdxq"
Jul 01 19:22:33.244103 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.243964    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq489\" (UniqueName: \"kubernetes.io/projected/074f32fe-7406-4e01-9868-da1ec706d589-kube-api-access-wq489\") pod \"kube-state-metrics-845b954455-gjdxq\" (UID: \"074f32fe-7406-4e01-9868-da1ec706d589\") " pod="openshift-monitoring/kube-state-metrics-845b954455-gjdxq"
Jul 01 19:22:33.252810 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.252777    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 01 19:22:33.253132 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.253111    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 01 19:22:33.253234 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.253114    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jrgdj\""
Jul 01 19:22:33.254637 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.254613    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 01 19:22:33.344594 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.344557    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ceea4b5-bcc4-405c-9364-c2dc815b0c97-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-cxsgw\" (UID: \"2ceea4b5-bcc4-405c-9364-c2dc815b0c97\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-cxsgw"
Jul 01 19:22:33.344768 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.344612    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8638dc64-df92-4461-b287-314a1fe5944e-node-exporter-accelerators-collector-config\") pod \"node-exporter-ntch4\" (UID: \"8638dc64-df92-4461-b287-314a1fe5944e\") " pod="openshift-monitoring/node-exporter-ntch4"
Jul 01 19:22:33.344768 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.344646    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8638dc64-df92-4461-b287-314a1fe5944e-sys\") pod \"node-exporter-ntch4\" (UID: \"8638dc64-df92-4461-b287-314a1fe5944e\") " pod="openshift-monitoring/node-exporter-ntch4"
Jul 01 19:22:33.344768 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.344673    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8638dc64-df92-4461-b287-314a1fe5944e-metrics-client-ca\") pod \"node-exporter-ntch4\" (UID: \"8638dc64-df92-4461-b287-314a1fe5944e\") " pod="openshift-monitoring/node-exporter-ntch4"
Jul 01 19:22:33.344768 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.344703    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq489\" (UniqueName: \"kubernetes.io/projected/074f32fe-7406-4e01-9868-da1ec706d589-kube-api-access-wq489\") pod \"kube-state-metrics-845b954455-gjdxq\" (UID: \"074f32fe-7406-4e01-9868-da1ec706d589\") " pod="openshift-monitoring/kube-state-metrics-845b954455-gjdxq"
Jul 01 19:22:33.344768 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.344738    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ceea4b5-bcc4-405c-9364-c2dc815b0c97-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-cxsgw\" (UID: \"2ceea4b5-bcc4-405c-9364-c2dc815b0c97\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-cxsgw"
Jul 01 19:22:33.345026 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.344838    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8638dc64-df92-4461-b287-314a1fe5944e-node-exporter-tls\") pod \"node-exporter-ntch4\" (UID: \"8638dc64-df92-4461-b287-314a1fe5944e\") " pod="openshift-monitoring/node-exporter-ntch4"
Jul 01 19:22:33.345026 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.344885    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8638dc64-df92-4461-b287-314a1fe5944e-root\") pod \"node-exporter-ntch4\" (UID: \"8638dc64-df92-4461-b287-314a1fe5944e\") " pod="openshift-monitoring/node-exporter-ntch4"
Jul 01 19:22:33.345026 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.344965    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n4p5\" (UniqueName: \"kubernetes.io/projected/2ceea4b5-bcc4-405c-9364-c2dc815b0c97-kube-api-access-8n4p5\") pod \"openshift-state-metrics-57f99c8b49-cxsgw\" (UID: \"2ceea4b5-bcc4-405c-9364-c2dc815b0c97\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-cxsgw"
Jul 01 19:22:33.346116 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.345206    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/074f32fe-7406-4e01-9868-da1ec706d589-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-gjdxq\" (UID: \"074f32fe-7406-4e01-9868-da1ec706d589\") " pod="openshift-monitoring/kube-state-metrics-845b954455-gjdxq"
Jul 01 19:22:33.346116 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.345240    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/074f32fe-7406-4e01-9868-da1ec706d589-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-gjdxq\" (UID: \"074f32fe-7406-4e01-9868-da1ec706d589\") " pod="openshift-monitoring/kube-state-metrics-845b954455-gjdxq"
Jul 01 19:22:33.346116 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.345265    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/074f32fe-7406-4e01-9868-da1ec706d589-metrics-client-ca\") pod \"kube-state-metrics-845b954455-gjdxq\" (UID: \"074f32fe-7406-4e01-9868-da1ec706d589\") " pod="openshift-monitoring/kube-state-metrics-845b954455-gjdxq"
Jul 01 19:22:33.346116 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.345290    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8638dc64-df92-4461-b287-314a1fe5944e-node-exporter-wtmp\") pod \"node-exporter-ntch4\" (UID: \"8638dc64-df92-4461-b287-314a1fe5944e\") " pod="openshift-monitoring/node-exporter-ntch4"
Jul 01 19:22:33.346116 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.345315    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8638dc64-df92-4461-b287-314a1fe5944e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ntch4\" (UID: \"8638dc64-df92-4461-b287-314a1fe5944e\") " pod="openshift-monitoring/node-exporter-ntch4"
Jul 01 19:22:33.346116 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:33.345443    2574 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 01 19:22:33.346116 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:22:33.345502    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/074f32fe-7406-4e01-9868-da1ec706d589-kube-state-metrics-tls podName:074f32fe-7406-4e01-9868-da1ec706d589 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:33.845478777 +0000 UTC m=+59.937478208 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/074f32fe-7406-4e01-9868-da1ec706d589-kube-state-metrics-tls") pod "kube-state-metrics-845b954455-gjdxq" (UID: "074f32fe-7406-4e01-9868-da1ec706d589") : secret "kube-state-metrics-tls" not found
Jul 01 19:22:33.346116 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.345561    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ceea4b5-bcc4-405c-9364-c2dc815b0c97-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-cxsgw\" (UID: \"2ceea4b5-bcc4-405c-9364-c2dc815b0c97\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-cxsgw"
Jul 01 19:22:33.346116 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.345594    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/074f32fe-7406-4e01-9868-da1ec706d589-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-gjdxq\" (UID: \"074f32fe-7406-4e01-9868-da1ec706d589\") " pod="openshift-monitoring/kube-state-metrics-845b954455-gjdxq"
Jul 01 19:22:33.346116 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.345632    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6zkk\" (UniqueName: \"kubernetes.io/projected/8638dc64-df92-4461-b287-314a1fe5944e-kube-api-access-s6zkk\") pod \"node-exporter-ntch4\" (UID: \"8638dc64-df92-4461-b287-314a1fe5944e\") " pod="openshift-monitoring/node-exporter-ntch4"
Jul 01 19:22:33.346116 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.345664    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/074f32fe-7406-4e01-9868-da1ec706d589-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-gjdxq\" (UID: \"074f32fe-7406-4e01-9868-da1ec706d589\") " pod="openshift-monitoring/kube-state-metrics-845b954455-gjdxq"
Jul 01 19:22:33.346116 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.345592    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/074f32fe-7406-4e01-9868-da1ec706d589-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-gjdxq\" (UID: \"074f32fe-7406-4e01-9868-da1ec706d589\") " pod="openshift-monitoring/kube-state-metrics-845b954455-gjdxq"
Jul 01 19:22:33.346116 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.345692    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8638dc64-df92-4461-b287-314a1fe5944e-node-exporter-textfile\") pod \"node-exporter-ntch4\" (UID: \"8638dc64-df92-4461-b287-314a1fe5944e\") " pod="openshift-monitoring/node-exporter-ntch4"
Jul 01 19:22:33.346826 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.346057    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/074f32fe-7406-4e01-9868-da1ec706d589-metrics-client-ca\") pod \"kube-state-metrics-845b954455-gjdxq\" (UID: \"074f32fe-7406-4e01-9868-da1ec706d589\") " pod="openshift-monitoring/kube-state-metrics-845b954455-gjdxq"
Jul 01 19:22:33.346826 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.346279    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ceea4b5-bcc4-405c-9364-c2dc815b0c97-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-cxsgw\" (UID: \"2ceea4b5-bcc4-405c-9364-c2dc815b0c97\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-cxsgw"
Jul 01 19:22:33.346826 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.346523    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/074f32fe-7406-4e01-9868-da1ec706d589-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-gjdxq\" (UID: \"074f32fe-7406-4e01-9868-da1ec706d589\") " pod="openshift-monitoring/kube-state-metrics-845b954455-gjdxq"
Jul 01 19:22:33.347698 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.347673    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ceea4b5-bcc4-405c-9364-c2dc815b0c97-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-cxsgw\" (UID: \"2ceea4b5-bcc4-405c-9364-c2dc815b0c97\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-cxsgw"
Jul 01 19:22:33.347803 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.347719    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ceea4b5-bcc4-405c-9364-c2dc815b0c97-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-cxsgw\" (UID: \"2ceea4b5-bcc4-405c-9364-c2dc815b0c97\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-cxsgw"
Jul 01 19:22:33.348363 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.348339    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/074f32fe-7406-4e01-9868-da1ec706d589-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-gjdxq\" (UID: \"074f32fe-7406-4e01-9868-da1ec706d589\") " pod="openshift-monitoring/kube-state-metrics-845b954455-gjdxq"
Jul 01 19:22:33.365551 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.365525    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n4p5\" (UniqueName: \"kubernetes.io/projected/2ceea4b5-bcc4-405c-9364-c2dc815b0c97-kube-api-access-8n4p5\") pod \"openshift-state-metrics-57f99c8b49-cxsgw\" (UID: \"2ceea4b5-bcc4-405c-9364-c2dc815b0c97\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-cxsgw"
Jul 01 19:22:33.366047 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.366018    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq489\" (UniqueName: \"kubernetes.io/projected/074f32fe-7406-4e01-9868-da1ec706d589-kube-api-access-wq489\") pod \"kube-state-metrics-845b954455-gjdxq\" (UID: \"074f32fe-7406-4e01-9868-da1ec706d589\") " pod="openshift-monitoring/kube-state-metrics-845b954455-gjdxq"
Jul 01 19:22:33.446528 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.446365    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8638dc64-df92-4461-b287-314a1fe5944e-node-exporter-tls\") pod \"node-exporter-ntch4\" (UID: \"8638dc64-df92-4461-b287-314a1fe5944e\") " pod="openshift-monitoring/node-exporter-ntch4"
Jul 01 19:22:33.446528 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.446409    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8638dc64-df92-4461-b287-314a1fe5944e-root\") pod \"node-exporter-ntch4\" (UID: \"8638dc64-df92-4461-b287-314a1fe5944e\") " pod="openshift-monitoring/node-exporter-ntch4"
Jul 01 19:22:33.446528 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.446463    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8638dc64-df92-4461-b287-314a1fe5944e-node-exporter-wtmp\") pod \"node-exporter-ntch4\" (UID: \"8638dc64-df92-4461-b287-314a1fe5944e\") " pod="openshift-monitoring/node-exporter-ntch4"
Jul 01 19:22:33.446528 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.446522    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8638dc64-df92-4461-b287-314a1fe5944e-root\") pod \"node-exporter-ntch4\" (UID: \"8638dc64-df92-4461-b287-314a1fe5944e\") " pod="openshift-monitoring/node-exporter-ntch4"
Jul 01 19:22:33.447014 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.446589    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8638dc64-df92-4461-b287-314a1fe5944e-node-exporter-wtmp\") pod \"node-exporter-ntch4\" (UID: \"8638dc64-df92-4461-b287-314a1fe5944e\") " pod="openshift-monitoring/node-exporter-ntch4"
Jul 01 19:22:33.447014 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.446655    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8638dc64-df92-4461-b287-314a1fe5944e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ntch4\" (UID: \"8638dc64-df92-4461-b287-314a1fe5944e\") " pod="openshift-monitoring/node-exporter-ntch4"
Jul 01 19:22:33.447014 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.446726    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6zkk\" (UniqueName: \"kubernetes.io/projected/8638dc64-df92-4461-b287-314a1fe5944e-kube-api-access-s6zkk\") pod \"node-exporter-ntch4\" (UID: \"8638dc64-df92-4461-b287-314a1fe5944e\") " pod="openshift-monitoring/node-exporter-ntch4"
Jul 01 19:22:33.447014 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.446757    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8638dc64-df92-4461-b287-314a1fe5944e-node-exporter-textfile\") pod \"node-exporter-ntch4\" (UID: \"8638dc64-df92-4461-b287-314a1fe5944e\") " pod="openshift-monitoring/node-exporter-ntch4"
Jul 01 19:22:33.447014 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.446798    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8638dc64-df92-4461-b287-314a1fe5944e-node-exporter-accelerators-collector-config\") pod \"node-exporter-ntch4\" (UID: \"8638dc64-df92-4461-b287-314a1fe5944e\") " pod="openshift-monitoring/node-exporter-ntch4"
Jul 01 19:22:33.447014 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.446831    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8638dc64-df92-4461-b287-314a1fe5944e-sys\") pod \"node-exporter-ntch4\" (UID: \"8638dc64-df92-4461-b287-314a1fe5944e\") " pod="openshift-monitoring/node-exporter-ntch4"
Jul 01 19:22:33.447014 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.446858    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8638dc64-df92-4461-b287-314a1fe5944e-metrics-client-ca\") pod \"node-exporter-ntch4\" (UID: \"8638dc64-df92-4461-b287-314a1fe5944e\") " pod="openshift-monitoring/node-exporter-ntch4"
Jul 01 19:22:33.447484 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.447445    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8638dc64-df92-4461-b287-314a1fe5944e-metrics-client-ca\") pod \"node-exporter-ntch4\" (UID: \"8638dc64-df92-4461-b287-314a1fe5944e\") " pod="openshift-monitoring/node-exporter-ntch4"
Jul 01 19:22:33.447573 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.447513    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8638dc64-df92-4461-b287-314a1fe5944e-sys\") pod \"node-exporter-ntch4\" (UID: \"8638dc64-df92-4461-b287-314a1fe5944e\") " pod="openshift-monitoring/node-exporter-ntch4"
Jul 01 19:22:33.448388 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.447641    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8638dc64-df92-4461-b287-314a1fe5944e-node-exporter-textfile\") pod \"node-exporter-ntch4\" (UID: \"8638dc64-df92-4461-b287-314a1fe5944e\") " pod="openshift-monitoring/node-exporter-ntch4"
Jul 01 19:22:33.448388 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.447636    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8638dc64-df92-4461-b287-314a1fe5944e-node-exporter-accelerators-collector-config\") pod \"node-exporter-ntch4\" (UID: \"8638dc64-df92-4461-b287-314a1fe5944e\") " pod="openshift-monitoring/node-exporter-ntch4"
Jul 01 19:22:33.449871 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.449841    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8638dc64-df92-4461-b287-314a1fe5944e-node-exporter-tls\") pod \"node-exporter-ntch4\" (UID: \"8638dc64-df92-4461-b287-314a1fe5944e\") " pod="openshift-monitoring/node-exporter-ntch4"
Jul 01 19:22:33.450415 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.450372    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8638dc64-df92-4461-b287-314a1fe5944e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ntch4\" (UID: \"8638dc64-df92-4461-b287-314a1fe5944e\") " pod="openshift-monitoring/node-exporter-ntch4"
Jul 01 19:22:33.460100 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.460076    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6zkk\" (UniqueName: \"kubernetes.io/projected/8638dc64-df92-4461-b287-314a1fe5944e-kube-api-access-s6zkk\") pod \"node-exporter-ntch4\" (UID: \"8638dc64-df92-4461-b287-314a1fe5944e\") " pod="openshift-monitoring/node-exporter-ntch4"
Jul 01 19:22:33.487321 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.487293    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-cxsgw"
Jul 01 19:22:33.554251 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.554221    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ntch4"
Jul 01 19:22:33.596043 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:22:33.596011    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8638dc64_df92_4461_b287_314a1fe5944e.slice/crio-1072e304d9c00209703fb8a0583750352d9c20e714dfbd6fd9a88cc3af933491 WatchSource:0}: Error finding container 1072e304d9c00209703fb8a0583750352d9c20e714dfbd6fd9a88cc3af933491: Status 404 returned error can't find the container with id 1072e304d9c00209703fb8a0583750352d9c20e714dfbd6fd9a88cc3af933491
Jul 01 19:22:33.602643 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.602608    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ntch4" event={"ID":"8638dc64-df92-4461-b287-314a1fe5944e","Type":"ContainerStarted","Data":"1072e304d9c00209703fb8a0583750352d9c20e714dfbd6fd9a88cc3af933491"}
Jul 01 19:22:33.675434 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.675399    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-cxsgw"]
Jul 01 19:22:33.678433 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:22:33.678406    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ceea4b5_bcc4_405c_9364_c2dc815b0c97.slice/crio-455f7eec0a2ef3ef02e7f2df9dec83170a9eafcb6fd23831d574aafc902ae138 WatchSource:0}: Error finding container 455f7eec0a2ef3ef02e7f2df9dec83170a9eafcb6fd23831d574aafc902ae138: Status 404 returned error can't find the container with id 455f7eec0a2ef3ef02e7f2df9dec83170a9eafcb6fd23831d574aafc902ae138
Jul 01 19:22:33.850801 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.850761    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/074f32fe-7406-4e01-9868-da1ec706d589-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-gjdxq\" (UID: \"074f32fe-7406-4e01-9868-da1ec706d589\") " pod="openshift-monitoring/kube-state-metrics-845b954455-gjdxq"
Jul 01 19:22:33.853752 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:33.853723    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/074f32fe-7406-4e01-9868-da1ec706d589-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-gjdxq\" (UID: \"074f32fe-7406-4e01-9868-da1ec706d589\") " pod="openshift-monitoring/kube-state-metrics-845b954455-gjdxq"
Jul 01 19:22:34.097541 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.097453    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-845b954455-gjdxq"
Jul 01 19:22:34.148067 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.148035    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 19:22:34.158606 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.158573    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:34.161359 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.161335    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 01 19:22:34.161486 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.161375    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 01 19:22:34.161597 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.161570    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 01 19:22:34.161656 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.161605    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 01 19:22:34.161823 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.161803    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 01 19:22:34.161936 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.161847    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 01 19:22:34.162007 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.161938    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 01 19:22:34.162007 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.161951    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 01 19:22:34.162183 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.162064    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rds5f\""
Jul 01 19:22:34.162287 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.162269    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 01 19:22:34.172932 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.172886    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 19:22:34.254885 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.254843    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:34.254885 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.254889    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:34.255123 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.254933    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:34.255123 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.254969    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-config-out\") pod \"alertmanager-main-0\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:34.255123 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.255055    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:34.255123 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.255093    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:34.255313 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.255205    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:34.255313 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.255281    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-config-volume\") pod \"alertmanager-main-0\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:34.255411 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.255309    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8b6d\" (UniqueName: \"kubernetes.io/projected/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-kube-api-access-r8b6d\") pod \"alertmanager-main-0\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:34.255411 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.255341    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:34.255411 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.255388    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:34.255559 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.255424    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-web-config\") pod \"alertmanager-main-0\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:34.255559 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.255469    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:34.356455 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.356011    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:34.356455 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.356063    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:34.356455 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.356110    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:34.356733 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.356461    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-config-volume\") pod \"alertmanager-main-0\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:34.356733 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.356512    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8b6d\" (UniqueName: \"kubernetes.io/projected/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-kube-api-access-r8b6d\") pod \"alertmanager-main-0\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:34.356733 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.356545    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:34.356733 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.356585    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:34.356733 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.356616    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-web-config\") pod \"alertmanager-main-0\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:34.356733 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.356665    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:34.356733 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.356706    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:34.357083 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.356735    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:34.357083 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.356762    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:34.357083 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.356786    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-config-out\") pod \"alertmanager-main-0\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:34.357083 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.357037    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:34.357461 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.357261    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:34.359414 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.359390    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-config-out\") pod \"alertmanager-main-0\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:34.361948 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.361929    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 01 19:22:34.362784 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.362405    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 01 19:22:34.362784 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.362491    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 01 19:22:34.362784 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.362552    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 01 19:22:34.362784 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.362631    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 01 19:22:34.362784 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.362660    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 01 19:22:34.363337 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.363177    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 01 19:22:34.364456 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.364160    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 01 19:22:34.370576 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.370100    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 01 19:22:34.371211 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.371187    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8b6d\" (UniqueName: \"kubernetes.io/projected/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-kube-api-access-r8b6d\") pod \"alertmanager-main-0\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:34.371529 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.371508    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:34.372550 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.372395    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-config-volume\") pod \"alertmanager-main-0\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:34.372636 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.372612    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-web-config\") pod \"alertmanager-main-0\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:34.373229 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.373206    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:34.373697 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.373672    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:34.373785 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.373706    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:34.374563 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.374540    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:34.375076 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.374991    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:34.377540 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.377516    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:34.471825 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.471793    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rds5f\""
Jul 01 19:22:34.479814 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.479788    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:34.609458 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.609384    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-cxsgw" event={"ID":"2ceea4b5-bcc4-405c-9364-c2dc815b0c97","Type":"ContainerStarted","Data":"7f9a263d48e351e8fd41de5f6c91d77172216eaff57d35e15685c4612e075afb"}
Jul 01 19:22:34.609458 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.609441    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-cxsgw" event={"ID":"2ceea4b5-bcc4-405c-9364-c2dc815b0c97","Type":"ContainerStarted","Data":"455f7eec0a2ef3ef02e7f2df9dec83170a9eafcb6fd23831d574aafc902ae138"}
Jul 01 19:22:34.619988 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.619960    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55cc98d7f-fgkth"]
Jul 01 19:22:34.634092 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.634068    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55cc98d7f-fgkth"
Jul 01 19:22:34.635961 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.635935    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55cc98d7f-fgkth"]
Jul 01 19:22:34.637079 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.637054    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6zs54\""
Jul 01 19:22:34.637246 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.637228    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 01 19:22:34.637464 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.637434    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 01 19:22:34.637708 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.637679    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 01 19:22:34.637797 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.637054    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 01 19:22:34.637880 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.637863    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 01 19:22:34.761098 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.761057    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9cd41253-3f14-42d8-9893-3b95d85270c1-console-serving-cert\") pod \"console-55cc98d7f-fgkth\" (UID: \"9cd41253-3f14-42d8-9893-3b95d85270c1\") " pod="openshift-console/console-55cc98d7f-fgkth"
Jul 01 19:22:34.761284 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.761168    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hz5b\" (UniqueName: \"kubernetes.io/projected/9cd41253-3f14-42d8-9893-3b95d85270c1-kube-api-access-9hz5b\") pod \"console-55cc98d7f-fgkth\" (UID: \"9cd41253-3f14-42d8-9893-3b95d85270c1\") " pod="openshift-console/console-55cc98d7f-fgkth"
Jul 01 19:22:34.761284 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.761201    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd41253-3f14-42d8-9893-3b95d85270c1-service-ca\") pod \"console-55cc98d7f-fgkth\" (UID: \"9cd41253-3f14-42d8-9893-3b95d85270c1\") " pod="openshift-console/console-55cc98d7f-fgkth"
Jul 01 19:22:34.761284 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.761234    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9cd41253-3f14-42d8-9893-3b95d85270c1-oauth-serving-cert\") pod \"console-55cc98d7f-fgkth\" (UID: \"9cd41253-3f14-42d8-9893-3b95d85270c1\") " pod="openshift-console/console-55cc98d7f-fgkth"
Jul 01 19:22:34.761408 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.761322    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9cd41253-3f14-42d8-9893-3b95d85270c1-console-oauth-config\") pod \"console-55cc98d7f-fgkth\" (UID: \"9cd41253-3f14-42d8-9893-3b95d85270c1\") " pod="openshift-console/console-55cc98d7f-fgkth"
Jul 01 19:22:34.761454 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.761420    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9cd41253-3f14-42d8-9893-3b95d85270c1-console-config\") pod \"console-55cc98d7f-fgkth\" (UID: \"9cd41253-3f14-42d8-9893-3b95d85270c1\") " pod="openshift-console/console-55cc98d7f-fgkth"
Jul 01 19:22:34.862717 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.862626    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9cd41253-3f14-42d8-9893-3b95d85270c1-console-serving-cert\") pod \"console-55cc98d7f-fgkth\" (UID: \"9cd41253-3f14-42d8-9893-3b95d85270c1\") " pod="openshift-console/console-55cc98d7f-fgkth"
Jul 01 19:22:34.862882 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.862717    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hz5b\" (UniqueName: \"kubernetes.io/projected/9cd41253-3f14-42d8-9893-3b95d85270c1-kube-api-access-9hz5b\") pod \"console-55cc98d7f-fgkth\" (UID: \"9cd41253-3f14-42d8-9893-3b95d85270c1\") " pod="openshift-console/console-55cc98d7f-fgkth"
Jul 01 19:22:34.862882 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.862749    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd41253-3f14-42d8-9893-3b95d85270c1-service-ca\") pod \"console-55cc98d7f-fgkth\" (UID: \"9cd41253-3f14-42d8-9893-3b95d85270c1\") " pod="openshift-console/console-55cc98d7f-fgkth"
Jul 01 19:22:34.862882 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.862783    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9cd41253-3f14-42d8-9893-3b95d85270c1-oauth-serving-cert\") pod \"console-55cc98d7f-fgkth\" (UID: \"9cd41253-3f14-42d8-9893-3b95d85270c1\") " pod="openshift-console/console-55cc98d7f-fgkth"
Jul 01 19:22:34.862882 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.862824    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9cd41253-3f14-42d8-9893-3b95d85270c1-console-oauth-config\") pod \"console-55cc98d7f-fgkth\" (UID: \"9cd41253-3f14-42d8-9893-3b95d85270c1\") " pod="openshift-console/console-55cc98d7f-fgkth"
Jul 01 19:22:34.863113 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.862923    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9cd41253-3f14-42d8-9893-3b95d85270c1-console-config\") pod \"console-55cc98d7f-fgkth\" (UID: \"9cd41253-3f14-42d8-9893-3b95d85270c1\") " pod="openshift-console/console-55cc98d7f-fgkth"
Jul 01 19:22:34.864124 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.864073    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd41253-3f14-42d8-9893-3b95d85270c1-service-ca\") pod \"console-55cc98d7f-fgkth\" (UID: \"9cd41253-3f14-42d8-9893-3b95d85270c1\") " pod="openshift-console/console-55cc98d7f-fgkth"
Jul 01 19:22:34.864244 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.864153    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9cd41253-3f14-42d8-9893-3b95d85270c1-console-config\") pod \"console-55cc98d7f-fgkth\" (UID: \"9cd41253-3f14-42d8-9893-3b95d85270c1\") " pod="openshift-console/console-55cc98d7f-fgkth"
Jul 01 19:22:34.864491 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.864465    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9cd41253-3f14-42d8-9893-3b95d85270c1-oauth-serving-cert\") pod \"console-55cc98d7f-fgkth\" (UID: \"9cd41253-3f14-42d8-9893-3b95d85270c1\") " pod="openshift-console/console-55cc98d7f-fgkth"
Jul 01 19:22:34.865794 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.865763    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9cd41253-3f14-42d8-9893-3b95d85270c1-console-oauth-config\") pod \"console-55cc98d7f-fgkth\" (UID: \"9cd41253-3f14-42d8-9893-3b95d85270c1\") " pod="openshift-console/console-55cc98d7f-fgkth"
Jul 01 19:22:34.866323 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.866301    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9cd41253-3f14-42d8-9893-3b95d85270c1-console-serving-cert\") pod \"console-55cc98d7f-fgkth\" (UID: \"9cd41253-3f14-42d8-9893-3b95d85270c1\") " pod="openshift-console/console-55cc98d7f-fgkth"
Jul 01 19:22:34.872156 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.872113    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hz5b\" (UniqueName: \"kubernetes.io/projected/9cd41253-3f14-42d8-9893-3b95d85270c1-kube-api-access-9hz5b\") pod \"console-55cc98d7f-fgkth\" (UID: \"9cd41253-3f14-42d8-9893-3b95d85270c1\") " pod="openshift-console/console-55cc98d7f-fgkth"
Jul 01 19:22:34.946979 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:34.946939    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55cc98d7f-fgkth"
Jul 01 19:22:35.309250 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:35.308968    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-gjdxq"]
Jul 01 19:22:35.310466 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:22:35.310436    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod074f32fe_7406_4e01_9868_da1ec706d589.slice/crio-df80d039dbc9cfe95547b93b3e5912afa43e711d6b62c18fbf8376d3cc0c16f1 WatchSource:0}: Error finding container df80d039dbc9cfe95547b93b3e5912afa43e711d6b62c18fbf8376d3cc0c16f1: Status 404 returned error can't find the container with id df80d039dbc9cfe95547b93b3e5912afa43e711d6b62c18fbf8376d3cc0c16f1
Jul 01 19:22:35.332020 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:35.331986    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 19:22:35.338241 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:22:35.338198    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff6f6d4c_aa49_47bf_aa73_71da2eb3e80a.slice/crio-1bd296bd21051436e268698ff00b04deaac4ab800d3e86459d10e31832ecc4f8 WatchSource:0}: Error finding container 1bd296bd21051436e268698ff00b04deaac4ab800d3e86459d10e31832ecc4f8: Status 404 returned error can't find the container with id 1bd296bd21051436e268698ff00b04deaac4ab800d3e86459d10e31832ecc4f8
Jul 01 19:22:35.357361 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:35.357296    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55cc98d7f-fgkth"]
Jul 01 19:22:35.361836 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:22:35.361809    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cd41253_3f14_42d8_9893_3b95d85270c1.slice/crio-e2e62c2adfdff666c98d2d320ebbceae0953c2eb31b3b31ba00933cc9aad1da6 WatchSource:0}: Error finding container e2e62c2adfdff666c98d2d320ebbceae0953c2eb31b3b31ba00933cc9aad1da6: Status 404 returned error can't find the container with id e2e62c2adfdff666c98d2d320ebbceae0953c2eb31b3b31ba00933cc9aad1da6
Jul 01 19:22:35.505122 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:35.505092    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bplsm"
Jul 01 19:22:35.617535 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:35.617446    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-cxsgw" event={"ID":"2ceea4b5-bcc4-405c-9364-c2dc815b0c97","Type":"ContainerStarted","Data":"294bd61ff043d86be61ac0f995158c42d6edd9c9b6ec490f7f8773ce3a3aee3a"}
Jul 01 19:22:35.619081 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:35.619016    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55cc98d7f-fgkth" event={"ID":"9cd41253-3f14-42d8-9893-3b95d85270c1","Type":"ContainerStarted","Data":"e2e62c2adfdff666c98d2d320ebbceae0953c2eb31b3b31ba00933cc9aad1da6"}
Jul 01 19:22:35.620405 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:35.620377    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a","Type":"ContainerStarted","Data":"1bd296bd21051436e268698ff00b04deaac4ab800d3e86459d10e31832ecc4f8"}
Jul 01 19:22:35.623184 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:35.623119    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n8l46" event={"ID":"3359c7fa-a51a-4dc7-b46e-c99a59154ba9","Type":"ContainerStarted","Data":"0a2e5294fb5a31154bf565fbfe25ded6e0ea40e469e4bf1bb280bfbeb367bb1d"}
Jul 01 19:22:35.624785 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:35.624762    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-gjdxq" event={"ID":"074f32fe-7406-4e01-9868-da1ec706d589","Type":"ContainerStarted","Data":"df80d039dbc9cfe95547b93b3e5912afa43e711d6b62c18fbf8376d3cc0c16f1"}
Jul 01 19:22:35.654050 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:35.653986    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-n8l46" podStartSLOduration=1.4409764219999999 podStartE2EDuration="10.653716873s" totalImagesPullingTime="9.212740451s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:22:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:35.652008102 +0000 UTC m=+61.744007576" watchObservedRunningTime="2026-07-01 19:22:35.653716873 +0000 UTC m=+61.745716331"
Jul 01 19:22:36.201688 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:36.201582    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6df8f7d99d-rdvgb"]
Jul 01 19:22:36.205161 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:36.205136    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6df8f7d99d-rdvgb"
Jul 01 19:22:36.210794 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:36.210588    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-73ed26q23e7v3\""
Jul 01 19:22:36.210794 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:36.210610    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 01 19:22:36.210794 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:36.210637    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 01 19:22:36.210794 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:36.210705    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-qb6x6\""
Jul 01 19:22:36.211102 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:36.210837    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 01 19:22:36.211347 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:36.211179    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 01 19:22:36.211347 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:36.211250    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 01 19:22:36.230035 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:36.229768    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6df8f7d99d-rdvgb"]
Jul 01 19:22:36.275372 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:36.274915    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/95c3faaa-6932-4f96-bb46-24fdd489b653-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6df8f7d99d-rdvgb\" (UID: \"95c3faaa-6932-4f96-bb46-24fdd489b653\") " pod="openshift-monitoring/thanos-querier-6df8f7d99d-rdvgb"
Jul 01 19:22:36.275372 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:36.274957    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/95c3faaa-6932-4f96-bb46-24fdd489b653-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6df8f7d99d-rdvgb\" (UID: \"95c3faaa-6932-4f96-bb46-24fdd489b653\") " pod="openshift-monitoring/thanos-querier-6df8f7d99d-rdvgb"
Jul 01 19:22:36.275372 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:36.275020    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/95c3faaa-6932-4f96-bb46-24fdd489b653-secret-grpc-tls\") pod \"thanos-querier-6df8f7d99d-rdvgb\" (UID: \"95c3faaa-6932-4f96-bb46-24fdd489b653\") " pod="openshift-monitoring/thanos-querier-6df8f7d99d-rdvgb"
Jul 01 19:22:36.275372 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:36.275048    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/95c3faaa-6932-4f96-bb46-24fdd489b653-secret-thanos-querier-tls\") pod \"thanos-querier-6df8f7d99d-rdvgb\" (UID: \"95c3faaa-6932-4f96-bb46-24fdd489b653\") " pod="openshift-monitoring/thanos-querier-6df8f7d99d-rdvgb"
Jul 01 19:22:36.275372 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:36.275122    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jbm6\" (UniqueName: \"kubernetes.io/projected/95c3faaa-6932-4f96-bb46-24fdd489b653-kube-api-access-7jbm6\") pod \"thanos-querier-6df8f7d99d-rdvgb\" (UID: \"95c3faaa-6932-4f96-bb46-24fdd489b653\") " pod="openshift-monitoring/thanos-querier-6df8f7d99d-rdvgb"
Jul 01 19:22:36.275372 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:36.275159    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/95c3faaa-6932-4f96-bb46-24fdd489b653-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6df8f7d99d-rdvgb\" (UID: \"95c3faaa-6932-4f96-bb46-24fdd489b653\") " pod="openshift-monitoring/thanos-querier-6df8f7d99d-rdvgb"
Jul 01 19:22:36.275372 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:36.275190    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/95c3faaa-6932-4f96-bb46-24fdd489b653-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6df8f7d99d-rdvgb\" (UID: \"95c3faaa-6932-4f96-bb46-24fdd489b653\") " pod="openshift-monitoring/thanos-querier-6df8f7d99d-rdvgb"
Jul 01 19:22:36.275372 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:36.275227    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95c3faaa-6932-4f96-bb46-24fdd489b653-metrics-client-ca\") pod \"thanos-querier-6df8f7d99d-rdvgb\" (UID: \"95c3faaa-6932-4f96-bb46-24fdd489b653\") " pod="openshift-monitoring/thanos-querier-6df8f7d99d-rdvgb"
Jul 01 19:22:36.377148 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:36.376451    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jbm6\" (UniqueName: \"kubernetes.io/projected/95c3faaa-6932-4f96-bb46-24fdd489b653-kube-api-access-7jbm6\") pod \"thanos-querier-6df8f7d99d-rdvgb\" (UID: \"95c3faaa-6932-4f96-bb46-24fdd489b653\") " pod="openshift-monitoring/thanos-querier-6df8f7d99d-rdvgb"
Jul 01 19:22:36.377148 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:36.376507    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/95c3faaa-6932-4f96-bb46-24fdd489b653-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6df8f7d99d-rdvgb\" (UID: \"95c3faaa-6932-4f96-bb46-24fdd489b653\") " pod="openshift-monitoring/thanos-querier-6df8f7d99d-rdvgb"
Jul 01 19:22:36.377148 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:36.376540    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/95c3faaa-6932-4f96-bb46-24fdd489b653-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6df8f7d99d-rdvgb\" (UID: \"95c3faaa-6932-4f96-bb46-24fdd489b653\") " pod="openshift-monitoring/thanos-querier-6df8f7d99d-rdvgb"
Jul 01 19:22:36.377148 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:36.376573    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95c3faaa-6932-4f96-bb46-24fdd489b653-metrics-client-ca\") pod \"thanos-querier-6df8f7d99d-rdvgb\" (UID: \"95c3faaa-6932-4f96-bb46-24fdd489b653\") " pod="openshift-monitoring/thanos-querier-6df8f7d99d-rdvgb"
Jul 01 19:22:36.377148 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:36.376627    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/95c3faaa-6932-4f96-bb46-24fdd489b653-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6df8f7d99d-rdvgb\" (UID: \"95c3faaa-6932-4f96-bb46-24fdd489b653\") " pod="openshift-monitoring/thanos-querier-6df8f7d99d-rdvgb"
Jul 01 19:22:36.377148 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:36.376655    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/95c3faaa-6932-4f96-bb46-24fdd489b653-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6df8f7d99d-rdvgb\" (UID: \"95c3faaa-6932-4f96-bb46-24fdd489b653\") " pod="openshift-monitoring/thanos-querier-6df8f7d99d-rdvgb"
Jul 01 19:22:36.377148 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:36.376709    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/95c3faaa-6932-4f96-bb46-24fdd489b653-secret-grpc-tls\") pod \"thanos-querier-6df8f7d99d-rdvgb\" (UID: \"95c3faaa-6932-4f96-bb46-24fdd489b653\") " pod="openshift-monitoring/thanos-querier-6df8f7d99d-rdvgb"
Jul 01 19:22:36.377148 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:36.376736    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/95c3faaa-6932-4f96-bb46-24fdd489b653-secret-thanos-querier-tls\") pod \"thanos-querier-6df8f7d99d-rdvgb\" (UID: \"95c3faaa-6932-4f96-bb46-24fdd489b653\") " pod="openshift-monitoring/thanos-querier-6df8f7d99d-rdvgb"
Jul 01 19:22:36.378108 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:36.377713    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95c3faaa-6932-4f96-bb46-24fdd489b653-metrics-client-ca\") pod \"thanos-querier-6df8f7d99d-rdvgb\" (UID: \"95c3faaa-6932-4f96-bb46-24fdd489b653\") " pod="openshift-monitoring/thanos-querier-6df8f7d99d-rdvgb"
Jul 01 19:22:36.381296 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:36.381251    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/95c3faaa-6932-4f96-bb46-24fdd489b653-secret-grpc-tls\") pod \"thanos-querier-6df8f7d99d-rdvgb\" (UID: \"95c3faaa-6932-4f96-bb46-24fdd489b653\") " pod="openshift-monitoring/thanos-querier-6df8f7d99d-rdvgb"
Jul 01 19:22:36.381739 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:36.381702    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/95c3faaa-6932-4f96-bb46-24fdd489b653-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6df8f7d99d-rdvgb\" (UID: \"95c3faaa-6932-4f96-bb46-24fdd489b653\") " pod="openshift-monitoring/thanos-querier-6df8f7d99d-rdvgb"
Jul 01 19:22:36.383127 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:36.383099    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/95c3faaa-6932-4f96-bb46-24fdd489b653-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6df8f7d99d-rdvgb\" (UID: \"95c3faaa-6932-4f96-bb46-24fdd489b653\") " pod="openshift-monitoring/thanos-querier-6df8f7d99d-rdvgb"
Jul 01 19:22:36.384250 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:36.384218    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/95c3faaa-6932-4f96-bb46-24fdd489b653-secret-thanos-querier-tls\") pod \"thanos-querier-6df8f7d99d-rdvgb\" (UID: \"95c3faaa-6932-4f96-bb46-24fdd489b653\") " pod="openshift-monitoring/thanos-querier-6df8f7d99d-rdvgb"
Jul 01 19:22:36.384679 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:36.384623    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/95c3faaa-6932-4f96-bb46-24fdd489b653-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6df8f7d99d-rdvgb\" (UID: \"95c3faaa-6932-4f96-bb46-24fdd489b653\") " pod="openshift-monitoring/thanos-querier-6df8f7d99d-rdvgb"
Jul 01 19:22:36.385937 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:36.385912    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/95c3faaa-6932-4f96-bb46-24fdd489b653-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6df8f7d99d-rdvgb\" (UID: \"95c3faaa-6932-4f96-bb46-24fdd489b653\") " pod="openshift-monitoring/thanos-querier-6df8f7d99d-rdvgb"
Jul 01 19:22:36.389579 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:36.389542    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jbm6\" (UniqueName: \"kubernetes.io/projected/95c3faaa-6932-4f96-bb46-24fdd489b653-kube-api-access-7jbm6\") pod \"thanos-querier-6df8f7d99d-rdvgb\" (UID: \"95c3faaa-6932-4f96-bb46-24fdd489b653\") " pod="openshift-monitoring/thanos-querier-6df8f7d99d-rdvgb"
Jul 01 19:22:36.518286 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:36.518200    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6df8f7d99d-rdvgb"
Jul 01 19:22:36.635927 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:36.635158    2574 generic.go:356] "Generic (PLEG): container finished" podID="8638dc64-df92-4461-b287-314a1fe5944e" containerID="52224420eb174f3720483196ad1d58838b009227177db6b4dcd7add2d3c54fbc" exitCode=0
Jul 01 19:22:36.637298 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:36.637267    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ntch4" event={"ID":"8638dc64-df92-4461-b287-314a1fe5944e","Type":"ContainerDied","Data":"52224420eb174f3720483196ad1d58838b009227177db6b4dcd7add2d3c54fbc"}
Jul 01 19:22:36.997631 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:36.997597    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6df8f7d99d-rdvgb"]
Jul 01 19:22:37.642408 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:37.642376    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ntch4" event={"ID":"8638dc64-df92-4461-b287-314a1fe5944e","Type":"ContainerStarted","Data":"302b170a5a2a8fa1c789de76a38dc41b0bc039c903d7da33bb8f6dd68de339e7"}
Jul 01 19:22:37.644095 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:37.643995    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6df8f7d99d-rdvgb" event={"ID":"95c3faaa-6932-4f96-bb46-24fdd489b653","Type":"ContainerStarted","Data":"f9ca84fd427a476c398f24170d92cc4bac8593cb9686bd2b94cbe2b434b47aff"}
Jul 01 19:22:37.648930 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:37.648891    2574 generic.go:356] "Generic (PLEG): container finished" podID="ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" containerID="f6a7f7e46315cb04dc38830294efaee355c2a4cb6190213b102541ef6a6cb41c" exitCode=0
Jul 01 19:22:37.649032 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:37.648949    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a","Type":"ContainerDied","Data":"f6a7f7e46315cb04dc38830294efaee355c2a4cb6190213b102541ef6a6cb41c"}
Jul 01 19:22:38.656973 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:38.656850    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-gjdxq" event={"ID":"074f32fe-7406-4e01-9868-da1ec706d589","Type":"ContainerStarted","Data":"f03fd3680cc9cd075e86a91f02f6a86bed575a7d5857f20626b5b0ae88e4e29b"}
Jul 01 19:22:38.656973 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:38.656942    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-gjdxq" event={"ID":"074f32fe-7406-4e01-9868-da1ec706d589","Type":"ContainerStarted","Data":"5c03026e219ee4720b1ac01ed6decd2caee9faaf2be377719d8f1fe92a05e953"}
Jul 01 19:22:38.656973 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:38.656973    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-gjdxq" event={"ID":"074f32fe-7406-4e01-9868-da1ec706d589","Type":"ContainerStarted","Data":"d62872e96cb3840a4729564c51462f723e173bc2dae41746a0aa7d8132999052"}
Jul 01 19:22:38.660388 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:38.660327    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ntch4" event={"ID":"8638dc64-df92-4461-b287-314a1fe5944e","Type":"ContainerStarted","Data":"68ddc1d25f8c63408d86c17ecf96811a177762d2db215fa8523ca1c76481b39c"}
Jul 01 19:22:38.663294 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:38.663267    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-cxsgw" event={"ID":"2ceea4b5-bcc4-405c-9364-c2dc815b0c97","Type":"ContainerStarted","Data":"bb386d36dc6e6bbe7c1d6d9df241f56c3bce3e341c0324c82624e4ed4e1240c6"}
Jul 01 19:22:38.675863 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:38.675822    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-845b954455-gjdxq" podStartSLOduration=3.380378508 podStartE2EDuration="5.675807997s" totalImagesPullingTime="2.295429489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:22:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:38.674419489 +0000 UTC m=+64.766418966" watchObservedRunningTime="2026-07-01 19:22:38.675807997 +0000 UTC m=+64.767807476"
Jul 01 19:22:38.697212 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:38.697157    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ntch4" podStartSLOduration=3.458677552 podStartE2EDuration="5.697136581s" totalImagesPullingTime="2.168971606s" totalInitContainerRuntime="69.487423ms" isStatefulPod=true podCreationTimestamp="2026-07-01 19:22:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:38.694633883 +0000 UTC m=+64.786633356" watchObservedRunningTime="2026-07-01 19:22:38.697136581 +0000 UTC m=+64.789136029"
Jul 01 19:22:38.745573 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:38.745508    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-cxsgw" podStartSLOduration=3.357776104 podStartE2EDuration="5.745484438s" totalImagesPullingTime="2.387708334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:22:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:38.743066445 +0000 UTC m=+64.835065913" watchObservedRunningTime="2026-07-01 19:22:38.745484438 +0000 UTC m=+64.837483888"
Jul 01 19:22:40.114637 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:40.114585    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9eaa621-a5fa-49a2-ad92-8dcd5729803a-original-pull-secret\") pod \"global-pull-secret-syncer-mq22h\" (UID: \"f9eaa621-a5fa-49a2-ad92-8dcd5729803a\") " pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:22:40.115163 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:40.114679    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98d86579-8161-4185-8cf7-5f427d712b2e-metrics-certs\") pod \"network-metrics-daemon-m6lm4\" (UID: \"98d86579-8161-4185-8cf7-5f427d712b2e\") " pod="openshift-multus/network-metrics-daemon-m6lm4"
Jul 01 19:22:40.117392 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:40.117358    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 01 19:22:40.117537 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:40.117440    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 01 19:22:40.127975 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:40.127949    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98d86579-8161-4185-8cf7-5f427d712b2e-metrics-certs\") pod \"network-metrics-daemon-m6lm4\" (UID: \"98d86579-8161-4185-8cf7-5f427d712b2e\") " pod="openshift-multus/network-metrics-daemon-m6lm4"
Jul 01 19:22:40.127975 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:40.127960    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9eaa621-a5fa-49a2-ad92-8dcd5729803a-original-pull-secret\") pod \"global-pull-secret-syncer-mq22h\" (UID: \"f9eaa621-a5fa-49a2-ad92-8dcd5729803a\") " pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:22:40.169801 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:40.169728    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k4h7l\""
Jul 01 19:22:40.173813 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:40.173791    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq22h"
Jul 01 19:22:40.180201 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:40.180181    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6lm4"
Jul 01 19:22:40.215798 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:40.215765    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q775\" (UniqueName: \"kubernetes.io/projected/9e551421-5f11-40eb-8a22-e594b6f4eb61-kube-api-access-8q775\") pod \"network-check-target-7g8pr\" (UID: \"9e551421-5f11-40eb-8a22-e594b6f4eb61\") " pod="openshift-network-diagnostics/network-check-target-7g8pr"
Jul 01 19:22:40.218660 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:40.218639    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 01 19:22:40.229204 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:40.229181    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 01 19:22:40.239552 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:40.239533    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q775\" (UniqueName: \"kubernetes.io/projected/9e551421-5f11-40eb-8a22-e594b6f4eb61-kube-api-access-8q775\") pod \"network-check-target-7g8pr\" (UID: \"9e551421-5f11-40eb-8a22-e594b6f4eb61\") " pod="openshift-network-diagnostics/network-check-target-7g8pr"
Jul 01 19:22:40.362495 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:40.362462    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-j4bvv\""
Jul 01 19:22:40.368635 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:40.368609    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g8pr"
Jul 01 19:22:41.068684 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:41.068163    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m6lm4"]
Jul 01 19:22:41.081289 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:22:41.081252    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98d86579_8161_4185_8cf7_5f427d712b2e.slice/crio-7cc9d8ea8c4b7ded7788bca6c533e754dc805e8f910f78c6452e7132f171824a WatchSource:0}: Error finding container 7cc9d8ea8c4b7ded7788bca6c533e754dc805e8f910f78c6452e7132f171824a: Status 404 returned error can't find the container with id 7cc9d8ea8c4b7ded7788bca6c533e754dc805e8f910f78c6452e7132f171824a
Jul 01 19:22:41.298091 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:41.298011    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7g8pr"]
Jul 01 19:22:41.304771 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:41.304005    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mq22h"]
Jul 01 19:22:41.310279 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:22:41.310018    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9eaa621_a5fa_49a2_ad92_8dcd5729803a.slice/crio-ca4180327226f72769a6a41a71bcf334647264b3f10454a1c14d8908b9774365 WatchSource:0}: Error finding container ca4180327226f72769a6a41a71bcf334647264b3f10454a1c14d8908b9774365: Status 404 returned error can't find the container with id ca4180327226f72769a6a41a71bcf334647264b3f10454a1c14d8908b9774365
Jul 01 19:22:41.677813 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:41.677776    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55cc98d7f-fgkth" event={"ID":"9cd41253-3f14-42d8-9893-3b95d85270c1","Type":"ContainerStarted","Data":"024888106a300806f9a16c14f1dd7534cac1282c42f3552a78fe28b245fceeea"}
Jul 01 19:22:41.680367 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:41.680341    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6df8f7d99d-rdvgb" event={"ID":"95c3faaa-6932-4f96-bb46-24fdd489b653","Type":"ContainerStarted","Data":"f1dd7ca6db529511c8d86334a89433d9b6d8ff758c5f20b22c8fb183d93a2932"}
Jul 01 19:22:41.680495 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:41.680407    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6df8f7d99d-rdvgb" event={"ID":"95c3faaa-6932-4f96-bb46-24fdd489b653","Type":"ContainerStarted","Data":"4295463b882a46a3afa6357db973dae89f30af080b1e8c3b2e80602638708fc3"}
Jul 01 19:22:41.680495 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:41.680447    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6df8f7d99d-rdvgb" event={"ID":"95c3faaa-6932-4f96-bb46-24fdd489b653","Type":"ContainerStarted","Data":"aec9cc5b188346ccd39ff8d31a935ad02209f3b200aa1abf4c54fc527f0b21d3"}
Jul 01 19:22:41.681863 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:41.681836    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mq22h" event={"ID":"f9eaa621-a5fa-49a2-ad92-8dcd5729803a","Type":"ContainerStarted","Data":"ca4180327226f72769a6a41a71bcf334647264b3f10454a1c14d8908b9774365"}
Jul 01 19:22:41.685196 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:41.685169    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a","Type":"ContainerStarted","Data":"837a073082791ba7ef268a62cbd44160f0774a10c8759c1e960ce2a54f775026"}
Jul 01 19:22:41.685280 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:41.685235    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a","Type":"ContainerStarted","Data":"566cd1574b74e59dd60ff6740a6c14af08c4fab4aa93670ac95f4c303ee4733f"}
Jul 01 19:22:41.685349 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:41.685278    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a","Type":"ContainerStarted","Data":"ef7e1a29538a5be6215571aa29dcae391ab761663ce70c8275c83f10c9692768"}
Jul 01 19:22:41.685349 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:41.685318    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a","Type":"ContainerStarted","Data":"d00eaef05e60cf038464cfc9d0019a2ee1c3a8587f50964a4b284b4b5ef05d97"}
Jul 01 19:22:41.685446 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:41.685357    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a","Type":"ContainerStarted","Data":"154069e46af698dc0af853d98e266e4650425bd004f701bf286bc107776d499e"}
Jul 01 19:22:41.686456 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:41.686431    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7g8pr" event={"ID":"9e551421-5f11-40eb-8a22-e594b6f4eb61","Type":"ContainerStarted","Data":"324031f118ea889d396f15b10f538be574ffe951145680ba3848b8c4cb11d11a"}
Jul 01 19:22:41.687828 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:41.687807    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m6lm4" event={"ID":"98d86579-8161-4185-8cf7-5f427d712b2e","Type":"ContainerStarted","Data":"7cc9d8ea8c4b7ded7788bca6c533e754dc805e8f910f78c6452e7132f171824a"}
Jul 01 19:22:41.700413 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:41.700367    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55cc98d7f-fgkth" podStartSLOduration=2.195128968 podStartE2EDuration="7.700348277s" totalImagesPullingTime="5.505219309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:22:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:41.699631387 +0000 UTC m=+67.791630869" watchObservedRunningTime="2026-07-01 19:22:41.700348277 +0000 UTC m=+67.792347725"
Jul 01 19:22:42.606041 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:42.605931    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-n29hh"
Jul 01 19:22:43.705859 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:43.705818    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6df8f7d99d-rdvgb" event={"ID":"95c3faaa-6932-4f96-bb46-24fdd489b653","Type":"ContainerStarted","Data":"9ae9d61a7ebba6ee1367f3b6930cfcf307f78ddf4d2fb8a8f8955a22822d6efe"}
Jul 01 19:22:43.706372 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:43.705922    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6df8f7d99d-rdvgb" event={"ID":"95c3faaa-6932-4f96-bb46-24fdd489b653","Type":"ContainerStarted","Data":"1c873000dfa8d56f6c2f82ea5880a8a6ce4c2d80d72e1d1cdd0adc583e912727"}
Jul 01 19:22:43.706372 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:43.705967    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6df8f7d99d-rdvgb" event={"ID":"95c3faaa-6932-4f96-bb46-24fdd489b653","Type":"ContainerStarted","Data":"eb770dd925797039318984693027a2b227bfe70814c01c7ca03bdb9a2434ca48"}
Jul 01 19:22:43.706372 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:43.706011    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6df8f7d99d-rdvgb"
Jul 01 19:22:43.710524 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:43.710448    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a","Type":"ContainerStarted","Data":"c5056edb07a43f1668bb0660a170225f2325a073e40c00947b9214996f063949"}
Jul 01 19:22:43.714066 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:43.714005    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m6lm4" event={"ID":"98d86579-8161-4185-8cf7-5f427d712b2e","Type":"ContainerStarted","Data":"eba3ecbacda4643872663b06abfdeaacdbaa342eaaa27549d9aede0efbbe5e6e"}
Jul 01 19:22:43.714186 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:43.714092    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m6lm4" event={"ID":"98d86579-8161-4185-8cf7-5f427d712b2e","Type":"ContainerStarted","Data":"733ea3fd074149703c94db7ff17836a23168386509dd42ab52ff7ca083da29f3"}
Jul 01 19:22:43.800710 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:43.800304    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.659867637 podStartE2EDuration="9.800282246s" totalImagesPullingTime="6.11851698s" totalInitContainerRuntime="21.897629ms" isStatefulPod=false podCreationTimestamp="2026-07-01 19:22:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:43.798470821 +0000 UTC m=+69.890470298" watchObservedRunningTime="2026-07-01 19:22:43.800282246 +0000 UTC m=+69.892281694"
Jul 01 19:22:43.802015 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:43.800943    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6df8f7d99d-rdvgb" podStartSLOduration=2.441766959 podStartE2EDuration="7.80092656s" totalImagesPullingTime="5.359159601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:22:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:43.750627968 +0000 UTC m=+69.842627442" watchObservedRunningTime="2026-07-01 19:22:43.80092656 +0000 UTC m=+69.892926016"
Jul 01 19:22:43.848142 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:43.847075    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-m6lm4" podStartSLOduration=67.916608399 podStartE2EDuration="1m9.847051268s" totalImagesPullingTime="1.930442869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:21:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:43.843161281 +0000 UTC m=+69.935160724" watchObservedRunningTime="2026-07-01 19:22:43.847051268 +0000 UTC m=+69.939050717"
Jul 01 19:22:44.622795 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:44.622647    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5bb4976c74-5rs24"]
Jul 01 19:22:44.628707 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:44.626141    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bb4976c74-5rs24"
Jul 01 19:22:44.636739 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:44.636709    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 01 19:22:44.652305 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:44.652232    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bb4976c74-5rs24"]
Jul 01 19:22:44.762006 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:44.761564    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b81156d0-bfc6-4255-95d5-55844116eebe-service-ca\") pod \"console-5bb4976c74-5rs24\" (UID: \"b81156d0-bfc6-4255-95d5-55844116eebe\") " pod="openshift-console/console-5bb4976c74-5rs24"
Jul 01 19:22:44.762006 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:44.761633    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b81156d0-bfc6-4255-95d5-55844116eebe-console-serving-cert\") pod \"console-5bb4976c74-5rs24\" (UID: \"b81156d0-bfc6-4255-95d5-55844116eebe\") " pod="openshift-console/console-5bb4976c74-5rs24"
Jul 01 19:22:44.762006 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:44.761666    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d86mb\" (UniqueName: \"kubernetes.io/projected/b81156d0-bfc6-4255-95d5-55844116eebe-kube-api-access-d86mb\") pod \"console-5bb4976c74-5rs24\" (UID: \"b81156d0-bfc6-4255-95d5-55844116eebe\") " pod="openshift-console/console-5bb4976c74-5rs24"
Jul 01 19:22:44.762006 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:44.761694    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b81156d0-bfc6-4255-95d5-55844116eebe-console-oauth-config\") pod \"console-5bb4976c74-5rs24\" (UID: \"b81156d0-bfc6-4255-95d5-55844116eebe\") " pod="openshift-console/console-5bb4976c74-5rs24"
Jul 01 19:22:44.762006 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:44.761737    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b81156d0-bfc6-4255-95d5-55844116eebe-oauth-serving-cert\") pod \"console-5bb4976c74-5rs24\" (UID: \"b81156d0-bfc6-4255-95d5-55844116eebe\") " pod="openshift-console/console-5bb4976c74-5rs24"
Jul 01 19:22:44.762006 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:44.761770    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b81156d0-bfc6-4255-95d5-55844116eebe-trusted-ca-bundle\") pod \"console-5bb4976c74-5rs24\" (UID: \"b81156d0-bfc6-4255-95d5-55844116eebe\") " pod="openshift-console/console-5bb4976c74-5rs24"
Jul 01 19:22:44.762006 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:44.761816    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b81156d0-bfc6-4255-95d5-55844116eebe-console-config\") pod \"console-5bb4976c74-5rs24\" (UID: \"b81156d0-bfc6-4255-95d5-55844116eebe\") " pod="openshift-console/console-5bb4976c74-5rs24"
Jul 01 19:22:44.862497 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:44.862460    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b81156d0-bfc6-4255-95d5-55844116eebe-service-ca\") pod \"console-5bb4976c74-5rs24\" (UID: \"b81156d0-bfc6-4255-95d5-55844116eebe\") " pod="openshift-console/console-5bb4976c74-5rs24"
Jul 01 19:22:44.862716 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:44.862571    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b81156d0-bfc6-4255-95d5-55844116eebe-console-serving-cert\") pod \"console-5bb4976c74-5rs24\" (UID: \"b81156d0-bfc6-4255-95d5-55844116eebe\") " pod="openshift-console/console-5bb4976c74-5rs24"
Jul 01 19:22:44.862716 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:44.862596    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d86mb\" (UniqueName: \"kubernetes.io/projected/b81156d0-bfc6-4255-95d5-55844116eebe-kube-api-access-d86mb\") pod \"console-5bb4976c74-5rs24\" (UID: \"b81156d0-bfc6-4255-95d5-55844116eebe\") " pod="openshift-console/console-5bb4976c74-5rs24"
Jul 01 19:22:44.862716 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:44.862620    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b81156d0-bfc6-4255-95d5-55844116eebe-console-oauth-config\") pod \"console-5bb4976c74-5rs24\" (UID: \"b81156d0-bfc6-4255-95d5-55844116eebe\") " pod="openshift-console/console-5bb4976c74-5rs24"
Jul 01 19:22:44.862873 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:44.862725    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b81156d0-bfc6-4255-95d5-55844116eebe-oauth-serving-cert\") pod \"console-5bb4976c74-5rs24\" (UID: \"b81156d0-bfc6-4255-95d5-55844116eebe\") " pod="openshift-console/console-5bb4976c74-5rs24"
Jul 01 19:22:44.862873 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:44.862793    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b81156d0-bfc6-4255-95d5-55844116eebe-trusted-ca-bundle\") pod \"console-5bb4976c74-5rs24\" (UID: \"b81156d0-bfc6-4255-95d5-55844116eebe\") " pod="openshift-console/console-5bb4976c74-5rs24"
Jul 01 19:22:44.862873 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:44.862856    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b81156d0-bfc6-4255-95d5-55844116eebe-console-config\") pod \"console-5bb4976c74-5rs24\" (UID: \"b81156d0-bfc6-4255-95d5-55844116eebe\") " pod="openshift-console/console-5bb4976c74-5rs24"
Jul 01 19:22:44.864043 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:44.863352    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b81156d0-bfc6-4255-95d5-55844116eebe-service-ca\") pod \"console-5bb4976c74-5rs24\" (UID: \"b81156d0-bfc6-4255-95d5-55844116eebe\") " pod="openshift-console/console-5bb4976c74-5rs24"
Jul 01 19:22:44.864163 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:44.864080    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b81156d0-bfc6-4255-95d5-55844116eebe-console-config\") pod \"console-5bb4976c74-5rs24\" (UID: \"b81156d0-bfc6-4255-95d5-55844116eebe\") " pod="openshift-console/console-5bb4976c74-5rs24"
Jul 01 19:22:44.864759 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:44.864448    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b81156d0-bfc6-4255-95d5-55844116eebe-oauth-serving-cert\") pod \"console-5bb4976c74-5rs24\" (UID: \"b81156d0-bfc6-4255-95d5-55844116eebe\") " pod="openshift-console/console-5bb4976c74-5rs24"
Jul 01 19:22:44.864759 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:44.864511    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b81156d0-bfc6-4255-95d5-55844116eebe-trusted-ca-bundle\") pod \"console-5bb4976c74-5rs24\" (UID: \"b81156d0-bfc6-4255-95d5-55844116eebe\") " pod="openshift-console/console-5bb4976c74-5rs24"
Jul 01 19:22:44.867236 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:44.867192    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b81156d0-bfc6-4255-95d5-55844116eebe-console-oauth-config\") pod \"console-5bb4976c74-5rs24\" (UID: \"b81156d0-bfc6-4255-95d5-55844116eebe\") " pod="openshift-console/console-5bb4976c74-5rs24"
Jul 01 19:22:44.867323 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:44.867285    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b81156d0-bfc6-4255-95d5-55844116eebe-console-serving-cert\") pod \"console-5bb4976c74-5rs24\" (UID: \"b81156d0-bfc6-4255-95d5-55844116eebe\") " pod="openshift-console/console-5bb4976c74-5rs24"
Jul 01 19:22:44.873594 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:44.873543    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d86mb\" (UniqueName: \"kubernetes.io/projected/b81156d0-bfc6-4255-95d5-55844116eebe-kube-api-access-d86mb\") pod \"console-5bb4976c74-5rs24\" (UID: \"b81156d0-bfc6-4255-95d5-55844116eebe\") " pod="openshift-console/console-5bb4976c74-5rs24"
Jul 01 19:22:44.942028 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:44.941984    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bb4976c74-5rs24"
Jul 01 19:22:44.947642 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:44.947605    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55cc98d7f-fgkth"
Jul 01 19:22:44.947752 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:44.947675    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55cc98d7f-fgkth"
Jul 01 19:22:44.949493 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:44.949465    2574 patch_prober.go:28] interesting pod/console-55cc98d7f-fgkth container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.14:8443/health\": dial tcp 10.134.0.14:8443: connect: connection refused" start-of-body=
Jul 01 19:22:44.949602 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:44.949531    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-55cc98d7f-fgkth" podUID="9cd41253-3f14-42d8-9893-3b95d85270c1" containerName="console" probeResult="failure" output="Get \"https://10.134.0.14:8443/health\": dial tcp 10.134.0.14:8443: connect: connection refused"
Jul 01 19:22:49.730507 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:49.730471    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6df8f7d99d-rdvgb"
Jul 01 19:22:54.457408 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:54.457332    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bb4976c74-5rs24"]
Jul 01 19:22:54.461752 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:22:54.461715    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb81156d0_bfc6_4255_95d5_55844116eebe.slice/crio-ae4e74e88c327eaccc5ca6a827fcb9cc3236c8171805579b3a2055a1ed1dbab7 WatchSource:0}: Error finding container ae4e74e88c327eaccc5ca6a827fcb9cc3236c8171805579b3a2055a1ed1dbab7: Status 404 returned error can't find the container with id ae4e74e88c327eaccc5ca6a827fcb9cc3236c8171805579b3a2055a1ed1dbab7
Jul 01 19:22:54.770676 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:54.770573    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bb4976c74-5rs24" event={"ID":"b81156d0-bfc6-4255-95d5-55844116eebe","Type":"ContainerStarted","Data":"8de4f66bd6761882b546ae2256db99154999acdb6312adf663ccdbfb2e3b8046"}
Jul 01 19:22:54.770676 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:54.770630    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bb4976c74-5rs24" event={"ID":"b81156d0-bfc6-4255-95d5-55844116eebe","Type":"ContainerStarted","Data":"ae4e74e88c327eaccc5ca6a827fcb9cc3236c8171805579b3a2055a1ed1dbab7"}
Jul 01 19:22:54.772388 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:54.772357    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mq22h" event={"ID":"f9eaa621-a5fa-49a2-ad92-8dcd5729803a","Type":"ContainerStarted","Data":"3f538868425dc9caf4d5bfbced0880731e0b6c9b95b1bb6185ba7a78d6d9df6f"}
Jul 01 19:22:54.774103 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:54.774077    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7g8pr" event={"ID":"9e551421-5f11-40eb-8a22-e594b6f4eb61","Type":"ContainerStarted","Data":"30facc918e0ed932509b450e2262482eb6c98c03b83a8f271bcbc2d51b56c93d"}
Jul 01 19:22:54.774295 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:54.774268    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7g8pr"
Jul 01 19:22:54.775673 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:54.775649    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5499b6fb8f-g2fv2" event={"ID":"d20bf32b-d17e-47a2-a6d0-809d3e465a08","Type":"ContainerStarted","Data":"a1b847f59b648f987f6e95675c8fc8ec7024dc9169117722fe2a6a61c3cfa0fd"}
Jul 01 19:22:54.775856 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:54.775823    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5499b6fb8f-g2fv2"
Jul 01 19:22:54.786049 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:54.786020    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5499b6fb8f-g2fv2"
Jul 01 19:22:54.861321 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:54.861261    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5bb4976c74-5rs24" podStartSLOduration=10.861239981 podStartE2EDuration="10.861239981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:22:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:54.860393467 +0000 UTC m=+80.952392938" watchObservedRunningTime="2026-07-01 19:22:54.861239981 +0000 UTC m=+80.953239432"
Jul 01 19:22:54.901275 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:54.901218    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mq22h" podStartSLOduration=67.932447881 podStartE2EDuration="1m20.901193084s" totalImagesPullingTime="12.968745203s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:21:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:54.900121395 +0000 UTC m=+80.992120867" watchObservedRunningTime="2026-07-01 19:22:54.901193084 +0000 UTC m=+80.993192534"
Jul 01 19:22:54.942935 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:54.942879    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5bb4976c74-5rs24"
Jul 01 19:22:54.943114 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:54.942971    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5bb4976c74-5rs24"
Jul 01 19:22:54.948747 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:54.948722    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5bb4976c74-5rs24"
Jul 01 19:22:54.952043 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:54.952018    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-55cc98d7f-fgkth"
Jul 01 19:22:54.956593 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:54.956567    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-55cc98d7f-fgkth"
Jul 01 19:22:54.958135 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:54.958086    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7g8pr" podStartSLOduration=67.987449596 podStartE2EDuration="1m20.958070546s" totalImagesPullingTime="12.97062095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:21:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:54.946360716 +0000 UTC m=+81.038360164" watchObservedRunningTime="2026-07-01 19:22:54.958070546 +0000 UTC m=+81.050070021"
Jul 01 19:22:55.017459 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:55.017401    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5499b6fb8f-g2fv2" podStartSLOduration=2.240474749 podStartE2EDuration="25.017383855s" totalImagesPullingTime="22.776909106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:22:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:55.016346763 +0000 UTC m=+81.108346212" watchObservedRunningTime="2026-07-01 19:22:55.017383855 +0000 UTC m=+81.109383302"
Jul 01 19:22:55.801510 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:55.801363    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5bb4976c74-5rs24"
Jul 01 19:22:55.899352 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:22:55.899321    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55cc98d7f-fgkth"]
Jul 01 19:23:21.808527 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:21.808446    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-55cc98d7f-fgkth" podUID="9cd41253-3f14-42d8-9893-3b95d85270c1" containerName="console" containerID="cri-o://024888106a300806f9a16c14f1dd7534cac1282c42f3552a78fe28b245fceeea" gracePeriod=15
Jul 01 19:23:22.088723 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:22.088664    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55cc98d7f-fgkth"
Jul 01 19:23:22.192215 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:22.192180    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9cd41253-3f14-42d8-9893-3b95d85270c1-console-config\") pod \"9cd41253-3f14-42d8-9893-3b95d85270c1\" (UID: \"9cd41253-3f14-42d8-9893-3b95d85270c1\") "
Jul 01 19:23:22.192399 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:22.192264    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9cd41253-3f14-42d8-9893-3b95d85270c1-console-serving-cert\") pod \"9cd41253-3f14-42d8-9893-3b95d85270c1\" (UID: \"9cd41253-3f14-42d8-9893-3b95d85270c1\") "
Jul 01 19:23:22.192399 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:22.192281    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd41253-3f14-42d8-9893-3b95d85270c1-service-ca\") pod \"9cd41253-3f14-42d8-9893-3b95d85270c1\" (UID: \"9cd41253-3f14-42d8-9893-3b95d85270c1\") "
Jul 01 19:23:22.192399 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:22.192299    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hz5b\" (UniqueName: \"kubernetes.io/projected/9cd41253-3f14-42d8-9893-3b95d85270c1-kube-api-access-9hz5b\") pod \"9cd41253-3f14-42d8-9893-3b95d85270c1\" (UID: \"9cd41253-3f14-42d8-9893-3b95d85270c1\") "
Jul 01 19:23:22.192399 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:22.192338    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9cd41253-3f14-42d8-9893-3b95d85270c1-console-oauth-config\") pod \"9cd41253-3f14-42d8-9893-3b95d85270c1\" (UID: \"9cd41253-3f14-42d8-9893-3b95d85270c1\") "
Jul 01 19:23:22.192399 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:22.192358    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9cd41253-3f14-42d8-9893-3b95d85270c1-oauth-serving-cert\") pod \"9cd41253-3f14-42d8-9893-3b95d85270c1\" (UID: \"9cd41253-3f14-42d8-9893-3b95d85270c1\") "
Jul 01 19:23:22.192737 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:22.192700    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cd41253-3f14-42d8-9893-3b95d85270c1-console-config" (OuterVolumeSpecName: "console-config") pod "9cd41253-3f14-42d8-9893-3b95d85270c1" (UID: "9cd41253-3f14-42d8-9893-3b95d85270c1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:23:22.192844 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:22.192751    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cd41253-3f14-42d8-9893-3b95d85270c1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9cd41253-3f14-42d8-9893-3b95d85270c1" (UID: "9cd41253-3f14-42d8-9893-3b95d85270c1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:23:22.192844 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:22.192709    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cd41253-3f14-42d8-9893-3b95d85270c1-service-ca" (OuterVolumeSpecName: "service-ca") pod "9cd41253-3f14-42d8-9893-3b95d85270c1" (UID: "9cd41253-3f14-42d8-9893-3b95d85270c1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:23:22.194600 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:22.194574    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cd41253-3f14-42d8-9893-3b95d85270c1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9cd41253-3f14-42d8-9893-3b95d85270c1" (UID: "9cd41253-3f14-42d8-9893-3b95d85270c1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:23:22.194683 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:22.194665    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd41253-3f14-42d8-9893-3b95d85270c1-kube-api-access-9hz5b" (OuterVolumeSpecName: "kube-api-access-9hz5b") pod "9cd41253-3f14-42d8-9893-3b95d85270c1" (UID: "9cd41253-3f14-42d8-9893-3b95d85270c1"). InnerVolumeSpecName "kube-api-access-9hz5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:23:22.194735 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:22.194685    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cd41253-3f14-42d8-9893-3b95d85270c1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9cd41253-3f14-42d8-9893-3b95d85270c1" (UID: "9cd41253-3f14-42d8-9893-3b95d85270c1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:23:22.293407 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:22.293367    2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9cd41253-3f14-42d8-9893-3b95d85270c1-console-oauth-config\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:23:22.293407 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:22.293401    2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9cd41253-3f14-42d8-9893-3b95d85270c1-oauth-serving-cert\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:23:22.293407 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:22.293414    2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9cd41253-3f14-42d8-9893-3b95d85270c1-console-config\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:23:22.293639 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:22.293426    2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9cd41253-3f14-42d8-9893-3b95d85270c1-console-serving-cert\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:23:22.293639 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:22.293437    2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd41253-3f14-42d8-9893-3b95d85270c1-service-ca\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:23:22.293639 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:22.293447    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hz5b\" (UniqueName: \"kubernetes.io/projected/9cd41253-3f14-42d8-9893-3b95d85270c1-kube-api-access-9hz5b\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:23:22.895958 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:22.895895    2574 generic.go:356] "Generic (PLEG): container finished" podID="9cd41253-3f14-42d8-9893-3b95d85270c1" containerID="024888106a300806f9a16c14f1dd7534cac1282c42f3552a78fe28b245fceeea" exitCode=0
Jul 01 19:23:22.896375 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:22.895987    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55cc98d7f-fgkth" event={"ID":"9cd41253-3f14-42d8-9893-3b95d85270c1","Type":"ContainerDied","Data":"024888106a300806f9a16c14f1dd7534cac1282c42f3552a78fe28b245fceeea"}
Jul 01 19:23:22.896375 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:22.896069    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55cc98d7f-fgkth"
Jul 01 19:23:22.896375 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:22.896136    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55cc98d7f-fgkth" event={"ID":"9cd41253-3f14-42d8-9893-3b95d85270c1","Type":"ContainerDied","Data":"e2e62c2adfdff666c98d2d320ebbceae0953c2eb31b3b31ba00933cc9aad1da6"}
Jul 01 19:23:22.896375 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:22.896202    2574 scope.go:117] "RemoveContainer" containerID="024888106a300806f9a16c14f1dd7534cac1282c42f3552a78fe28b245fceeea"
Jul 01 19:23:22.904435 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:22.904416    2574 scope.go:117] "RemoveContainer" containerID="024888106a300806f9a16c14f1dd7534cac1282c42f3552a78fe28b245fceeea"
Jul 01 19:23:22.904695 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:23:22.904670    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"024888106a300806f9a16c14f1dd7534cac1282c42f3552a78fe28b245fceeea\": container with ID starting with 024888106a300806f9a16c14f1dd7534cac1282c42f3552a78fe28b245fceeea not found: ID does not exist" containerID="024888106a300806f9a16c14f1dd7534cac1282c42f3552a78fe28b245fceeea"
Jul 01 19:23:22.904749 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:22.904704    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"024888106a300806f9a16c14f1dd7534cac1282c42f3552a78fe28b245fceeea"} err="failed to get container status \"024888106a300806f9a16c14f1dd7534cac1282c42f3552a78fe28b245fceeea\": rpc error: code = NotFound desc = could not find container \"024888106a300806f9a16c14f1dd7534cac1282c42f3552a78fe28b245fceeea\": container with ID starting with 024888106a300806f9a16c14f1dd7534cac1282c42f3552a78fe28b245fceeea not found: ID does not exist"
Jul 01 19:23:22.915465 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:22.915442    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55cc98d7f-fgkth"]
Jul 01 19:23:22.919948 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:22.919927    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55cc98d7f-fgkth"]
Jul 01 19:23:24.355302 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:24.355267    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cd41253-3f14-42d8-9893-3b95d85270c1" path="/var/lib/kubelet/pods/9cd41253-3f14-42d8-9893-3b95d85270c1/volumes"
Jul 01 19:23:25.782077 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:25.782037    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7g8pr"
Jul 01 19:23:53.634022 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:53.633989    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 19:23:53.634607 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:53.634295    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" containerName="alertmanager" containerID="cri-o://154069e46af698dc0af853d98e266e4650425bd004f701bf286bc107776d499e" gracePeriod=120
Jul 01 19:23:53.634607 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:53.634365    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" containerName="kube-rbac-proxy-web" containerID="cri-o://ef7e1a29538a5be6215571aa29dcae391ab761663ce70c8275c83f10c9692768" gracePeriod=120
Jul 01 19:23:53.634607 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:53.634364    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" containerName="kube-rbac-proxy-metric" containerID="cri-o://837a073082791ba7ef268a62cbd44160f0774a10c8759c1e960ce2a54f775026" gracePeriod=120
Jul 01 19:23:53.634607 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:53.634378    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" containerName="config-reloader" containerID="cri-o://d00eaef05e60cf038464cfc9d0019a2ee1c3a8587f50964a4b284b4b5ef05d97" gracePeriod=120
Jul 01 19:23:53.634607 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:53.634442    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" containerName="kube-rbac-proxy" containerID="cri-o://566cd1574b74e59dd60ff6740a6c14af08c4fab4aa93670ac95f4c303ee4733f" gracePeriod=120
Jul 01 19:23:53.634607 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:53.634522    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" containerName="prom-label-proxy" containerID="cri-o://c5056edb07a43f1668bb0660a170225f2325a073e40c00947b9214996f063949" gracePeriod=120
Jul 01 19:23:54.019404 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:54.019325    2574 generic.go:356] "Generic (PLEG): container finished" podID="ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" containerID="c5056edb07a43f1668bb0660a170225f2325a073e40c00947b9214996f063949" exitCode=0
Jul 01 19:23:54.019404 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:54.019349    2574 generic.go:356] "Generic (PLEG): container finished" podID="ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" containerID="566cd1574b74e59dd60ff6740a6c14af08c4fab4aa93670ac95f4c303ee4733f" exitCode=0
Jul 01 19:23:54.019404 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:54.019356    2574 generic.go:356] "Generic (PLEG): container finished" podID="ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" containerID="d00eaef05e60cf038464cfc9d0019a2ee1c3a8587f50964a4b284b4b5ef05d97" exitCode=0
Jul 01 19:23:54.019404 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:54.019362    2574 generic.go:356] "Generic (PLEG): container finished" podID="ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" containerID="154069e46af698dc0af853d98e266e4650425bd004f701bf286bc107776d499e" exitCode=0
Jul 01 19:23:54.019404 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:54.019395    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a","Type":"ContainerDied","Data":"c5056edb07a43f1668bb0660a170225f2325a073e40c00947b9214996f063949"}
Jul 01 19:23:54.019659 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:54.019469    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a","Type":"ContainerDied","Data":"566cd1574b74e59dd60ff6740a6c14af08c4fab4aa93670ac95f4c303ee4733f"}
Jul 01 19:23:54.019659 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:54.019498    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a","Type":"ContainerDied","Data":"d00eaef05e60cf038464cfc9d0019a2ee1c3a8587f50964a4b284b4b5ef05d97"}
Jul 01 19:23:54.019659 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:54.019529    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a","Type":"ContainerDied","Data":"154069e46af698dc0af853d98e266e4650425bd004f701bf286bc107776d499e"}
Jul 01 19:23:54.882491 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:54.882468    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:23:54.956566 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:54.956483    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-web-config\") pod \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") "
Jul 01 19:23:54.956566 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:54.956541    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-config-out\") pod \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") "
Jul 01 19:23:54.956780 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:54.956579    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-config-volume\") pod \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") "
Jul 01 19:23:54.956780 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:54.956603    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-cluster-tls-config\") pod \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") "
Jul 01 19:23:54.956780 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:54.956649    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-secret-alertmanager-main-tls\") pod \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") "
Jul 01 19:23:54.956780 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:54.956698    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") "
Jul 01 19:23:54.956780 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:54.956725    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-alertmanager-trusted-ca-bundle\") pod \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") "
Jul 01 19:23:54.956780 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:54.956765    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-tls-assets\") pod \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") "
Jul 01 19:23:54.957135 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:54.956803    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-metrics-client-ca\") pod \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") "
Jul 01 19:23:54.957135 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:54.956835    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-alertmanager-main-db\") pod \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") "
Jul 01 19:23:54.957135 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:54.956860    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-secret-alertmanager-kube-rbac-proxy\") pod \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") "
Jul 01 19:23:54.957135 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:54.956886    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-secret-alertmanager-kube-rbac-proxy-web\") pod \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") "
Jul 01 19:23:54.957135 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:54.956940    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8b6d\" (UniqueName: \"kubernetes.io/projected/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-kube-api-access-r8b6d\") pod \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\" (UID: \"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a\") "
Jul 01 19:23:54.957881 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:54.957714    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" (UID: "ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:23:54.957881 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:54.957824    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" (UID: "ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:23:54.958178 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:54.958131    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" (UID: "ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:23:54.960345 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:54.960050    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-config-volume" (OuterVolumeSpecName: "config-volume") pod "ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" (UID: "ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:23:54.960734 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:54.960704    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-kube-api-access-r8b6d" (OuterVolumeSpecName: "kube-api-access-r8b6d") pod "ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" (UID: "ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a"). InnerVolumeSpecName "kube-api-access-r8b6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:23:54.961086 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:54.961050    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" (UID: "ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:23:54.961086 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:54.961059    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" (UID: "ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:23:54.961533 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:54.961492    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-config-out" (OuterVolumeSpecName: "config-out") pod "ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" (UID: "ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:23:54.961653 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:54.961524    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" (UID: "ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:23:54.961951 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:54.961925    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" (UID: "ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:23:54.962439 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:54.962410    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" (UID: "ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:23:54.964323 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:54.964298    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" (UID: "ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:23:54.970559 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:54.970534    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-web-config" (OuterVolumeSpecName: "web-config") pod "ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" (UID: "ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:23:55.025850 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.025817    2574 generic.go:356] "Generic (PLEG): container finished" podID="ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" containerID="837a073082791ba7ef268a62cbd44160f0774a10c8759c1e960ce2a54f775026" exitCode=0
Jul 01 19:23:55.025850 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.025842    2574 generic.go:356] "Generic (PLEG): container finished" podID="ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" containerID="ef7e1a29538a5be6215571aa29dcae391ab761663ce70c8275c83f10c9692768" exitCode=0
Jul 01 19:23:55.026099 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.025922    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:23:55.026099 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.025926    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a","Type":"ContainerDied","Data":"837a073082791ba7ef268a62cbd44160f0774a10c8759c1e960ce2a54f775026"}
Jul 01 19:23:55.026099 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.026028    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a","Type":"ContainerDied","Data":"ef7e1a29538a5be6215571aa29dcae391ab761663ce70c8275c83f10c9692768"}
Jul 01 19:23:55.026099 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.026057    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a","Type":"ContainerDied","Data":"1bd296bd21051436e268698ff00b04deaac4ab800d3e86459d10e31832ecc4f8"}
Jul 01 19:23:55.026099 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.026100    2574 scope.go:117] "RemoveContainer" containerID="c5056edb07a43f1668bb0660a170225f2325a073e40c00947b9214996f063949"
Jul 01 19:23:55.034813 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.034794    2574 scope.go:117] "RemoveContainer" containerID="837a073082791ba7ef268a62cbd44160f0774a10c8759c1e960ce2a54f775026"
Jul 01 19:23:55.042033 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.041975    2574 scope.go:117] "RemoveContainer" containerID="566cd1574b74e59dd60ff6740a6c14af08c4fab4aa93670ac95f4c303ee4733f"
Jul 01 19:23:55.048366 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.048350    2574 scope.go:117] "RemoveContainer" containerID="ef7e1a29538a5be6215571aa29dcae391ab761663ce70c8275c83f10c9692768"
Jul 01 19:23:55.051326 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.051306    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 19:23:55.055560 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.055538    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 19:23:55.055847 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.055828    2574 scope.go:117] "RemoveContainer" containerID="d00eaef05e60cf038464cfc9d0019a2ee1c3a8587f50964a4b284b4b5ef05d97"
Jul 01 19:23:55.058069 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.058051    2574 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-secret-alertmanager-main-tls\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:23:55.058168 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.058073    2574 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:23:55.058168 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.058089    2574 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:23:55.058168 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.058112    2574 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-tls-assets\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:23:55.058168 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.058125    2574 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-metrics-client-ca\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:23:55.058168 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.058139    2574 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-alertmanager-main-db\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:23:55.058168 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.058156    2574 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:23:55.058399 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.058171    2574 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:23:55.058399 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.058186    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8b6d\" (UniqueName: \"kubernetes.io/projected/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-kube-api-access-r8b6d\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:23:55.058399 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.058228    2574 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-web-config\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:23:55.058399 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.058242    2574 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-config-out\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:23:55.058399 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.058255    2574 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-config-volume\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:23:55.058399 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.058269    2574 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a-cluster-tls-config\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:23:55.062728 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.062711    2574 scope.go:117] "RemoveContainer" containerID="154069e46af698dc0af853d98e266e4650425bd004f701bf286bc107776d499e"
Jul 01 19:23:55.069224 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.069207    2574 scope.go:117] "RemoveContainer" containerID="f6a7f7e46315cb04dc38830294efaee355c2a4cb6190213b102541ef6a6cb41c"
Jul 01 19:23:55.076108 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.076089    2574 scope.go:117] "RemoveContainer" containerID="c5056edb07a43f1668bb0660a170225f2325a073e40c00947b9214996f063949"
Jul 01 19:23:55.076355 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:23:55.076335    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5056edb07a43f1668bb0660a170225f2325a073e40c00947b9214996f063949\": container with ID starting with c5056edb07a43f1668bb0660a170225f2325a073e40c00947b9214996f063949 not found: ID does not exist" containerID="c5056edb07a43f1668bb0660a170225f2325a073e40c00947b9214996f063949"
Jul 01 19:23:55.076413 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.076368    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5056edb07a43f1668bb0660a170225f2325a073e40c00947b9214996f063949"} err="failed to get container status \"c5056edb07a43f1668bb0660a170225f2325a073e40c00947b9214996f063949\": rpc error: code = NotFound desc = could not find container \"c5056edb07a43f1668bb0660a170225f2325a073e40c00947b9214996f063949\": container with ID starting with c5056edb07a43f1668bb0660a170225f2325a073e40c00947b9214996f063949 not found: ID does not exist"
Jul 01 19:23:55.076413 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.076392    2574 scope.go:117] "RemoveContainer" containerID="837a073082791ba7ef268a62cbd44160f0774a10c8759c1e960ce2a54f775026"
Jul 01 19:23:55.076629 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:23:55.076611    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"837a073082791ba7ef268a62cbd44160f0774a10c8759c1e960ce2a54f775026\": container with ID starting with 837a073082791ba7ef268a62cbd44160f0774a10c8759c1e960ce2a54f775026 not found: ID does not exist" containerID="837a073082791ba7ef268a62cbd44160f0774a10c8759c1e960ce2a54f775026"
Jul 01 19:23:55.076665 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.076632    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"837a073082791ba7ef268a62cbd44160f0774a10c8759c1e960ce2a54f775026"} err="failed to get container status \"837a073082791ba7ef268a62cbd44160f0774a10c8759c1e960ce2a54f775026\": rpc error: code = NotFound desc = could not find container \"837a073082791ba7ef268a62cbd44160f0774a10c8759c1e960ce2a54f775026\": container with ID starting with 837a073082791ba7ef268a62cbd44160f0774a10c8759c1e960ce2a54f775026 not found: ID does not exist"
Jul 01 19:23:55.076665 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.076645    2574 scope.go:117] "RemoveContainer" containerID="566cd1574b74e59dd60ff6740a6c14af08c4fab4aa93670ac95f4c303ee4733f"
Jul 01 19:23:55.076855 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:23:55.076839    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"566cd1574b74e59dd60ff6740a6c14af08c4fab4aa93670ac95f4c303ee4733f\": container with ID starting with 566cd1574b74e59dd60ff6740a6c14af08c4fab4aa93670ac95f4c303ee4733f not found: ID does not exist" containerID="566cd1574b74e59dd60ff6740a6c14af08c4fab4aa93670ac95f4c303ee4733f"
Jul 01 19:23:55.076894 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.076862    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"566cd1574b74e59dd60ff6740a6c14af08c4fab4aa93670ac95f4c303ee4733f"} err="failed to get container status \"566cd1574b74e59dd60ff6740a6c14af08c4fab4aa93670ac95f4c303ee4733f\": rpc error: code = NotFound desc = could not find container \"566cd1574b74e59dd60ff6740a6c14af08c4fab4aa93670ac95f4c303ee4733f\": container with ID starting with 566cd1574b74e59dd60ff6740a6c14af08c4fab4aa93670ac95f4c303ee4733f not found: ID does not exist"
Jul 01 19:23:55.076894 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.076885    2574 scope.go:117] "RemoveContainer" containerID="ef7e1a29538a5be6215571aa29dcae391ab761663ce70c8275c83f10c9692768"
Jul 01 19:23:55.077115 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:23:55.077100    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef7e1a29538a5be6215571aa29dcae391ab761663ce70c8275c83f10c9692768\": container with ID starting with ef7e1a29538a5be6215571aa29dcae391ab761663ce70c8275c83f10c9692768 not found: ID does not exist" containerID="ef7e1a29538a5be6215571aa29dcae391ab761663ce70c8275c83f10c9692768"
Jul 01 19:23:55.077157 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.077120    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef7e1a29538a5be6215571aa29dcae391ab761663ce70c8275c83f10c9692768"} err="failed to get container status \"ef7e1a29538a5be6215571aa29dcae391ab761663ce70c8275c83f10c9692768\": rpc error: code = NotFound desc = could not find container \"ef7e1a29538a5be6215571aa29dcae391ab761663ce70c8275c83f10c9692768\": container with ID starting with ef7e1a29538a5be6215571aa29dcae391ab761663ce70c8275c83f10c9692768 not found: ID does not exist"
Jul 01 19:23:55.077157 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.077132    2574 scope.go:117] "RemoveContainer" containerID="d00eaef05e60cf038464cfc9d0019a2ee1c3a8587f50964a4b284b4b5ef05d97"
Jul 01 19:23:55.077362 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:23:55.077344    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d00eaef05e60cf038464cfc9d0019a2ee1c3a8587f50964a4b284b4b5ef05d97\": container with ID starting with d00eaef05e60cf038464cfc9d0019a2ee1c3a8587f50964a4b284b4b5ef05d97 not found: ID does not exist" containerID="d00eaef05e60cf038464cfc9d0019a2ee1c3a8587f50964a4b284b4b5ef05d97"
Jul 01 19:23:55.077409 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.077365    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d00eaef05e60cf038464cfc9d0019a2ee1c3a8587f50964a4b284b4b5ef05d97"} err="failed to get container status \"d00eaef05e60cf038464cfc9d0019a2ee1c3a8587f50964a4b284b4b5ef05d97\": rpc error: code = NotFound desc = could not find container \"d00eaef05e60cf038464cfc9d0019a2ee1c3a8587f50964a4b284b4b5ef05d97\": container with ID starting with d00eaef05e60cf038464cfc9d0019a2ee1c3a8587f50964a4b284b4b5ef05d97 not found: ID does not exist"
Jul 01 19:23:55.077409 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.077390    2574 scope.go:117] "RemoveContainer" containerID="154069e46af698dc0af853d98e266e4650425bd004f701bf286bc107776d499e"
Jul 01 19:23:55.077641 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:23:55.077624    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"154069e46af698dc0af853d98e266e4650425bd004f701bf286bc107776d499e\": container with ID starting with 154069e46af698dc0af853d98e266e4650425bd004f701bf286bc107776d499e not found: ID does not exist" containerID="154069e46af698dc0af853d98e266e4650425bd004f701bf286bc107776d499e"
Jul 01 19:23:55.077677 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.077644    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"154069e46af698dc0af853d98e266e4650425bd004f701bf286bc107776d499e"} err="failed to get container status \"154069e46af698dc0af853d98e266e4650425bd004f701bf286bc107776d499e\": rpc error: code = NotFound desc = could not find container \"154069e46af698dc0af853d98e266e4650425bd004f701bf286bc107776d499e\": container with ID starting with 154069e46af698dc0af853d98e266e4650425bd004f701bf286bc107776d499e not found: ID does not exist"
Jul 01 19:23:55.077677 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.077656    2574 scope.go:117] "RemoveContainer" containerID="f6a7f7e46315cb04dc38830294efaee355c2a4cb6190213b102541ef6a6cb41c"
Jul 01 19:23:55.077845 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:23:55.077828    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6a7f7e46315cb04dc38830294efaee355c2a4cb6190213b102541ef6a6cb41c\": container with ID starting with f6a7f7e46315cb04dc38830294efaee355c2a4cb6190213b102541ef6a6cb41c not found: ID does not exist" containerID="f6a7f7e46315cb04dc38830294efaee355c2a4cb6190213b102541ef6a6cb41c"
Jul 01 19:23:55.077883 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.077848    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6a7f7e46315cb04dc38830294efaee355c2a4cb6190213b102541ef6a6cb41c"} err="failed to get container status \"f6a7f7e46315cb04dc38830294efaee355c2a4cb6190213b102541ef6a6cb41c\": rpc error: code = NotFound desc = could not find container \"f6a7f7e46315cb04dc38830294efaee355c2a4cb6190213b102541ef6a6cb41c\": container with ID starting with f6a7f7e46315cb04dc38830294efaee355c2a4cb6190213b102541ef6a6cb41c not found: ID does not exist"
Jul 01 19:23:55.077883 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.077861    2574 scope.go:117] "RemoveContainer" containerID="c5056edb07a43f1668bb0660a170225f2325a073e40c00947b9214996f063949"
Jul 01 19:23:55.078088 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.078071    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5056edb07a43f1668bb0660a170225f2325a073e40c00947b9214996f063949"} err="failed to get container status \"c5056edb07a43f1668bb0660a170225f2325a073e40c00947b9214996f063949\": rpc error: code = NotFound desc = could not find container \"c5056edb07a43f1668bb0660a170225f2325a073e40c00947b9214996f063949\": container with ID starting with c5056edb07a43f1668bb0660a170225f2325a073e40c00947b9214996f063949 not found: ID does not exist"
Jul 01 19:23:55.078088 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.078087    2574 scope.go:117] "RemoveContainer" containerID="837a073082791ba7ef268a62cbd44160f0774a10c8759c1e960ce2a54f775026"
Jul 01 19:23:55.078306 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.078288    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"837a073082791ba7ef268a62cbd44160f0774a10c8759c1e960ce2a54f775026"} err="failed to get container status \"837a073082791ba7ef268a62cbd44160f0774a10c8759c1e960ce2a54f775026\": rpc error: code = NotFound desc = could not find container \"837a073082791ba7ef268a62cbd44160f0774a10c8759c1e960ce2a54f775026\": container with ID starting with 837a073082791ba7ef268a62cbd44160f0774a10c8759c1e960ce2a54f775026 not found: ID does not exist"
Jul 01 19:23:55.078347 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.078307    2574 scope.go:117] "RemoveContainer" containerID="566cd1574b74e59dd60ff6740a6c14af08c4fab4aa93670ac95f4c303ee4733f"
Jul 01 19:23:55.078504 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.078488    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"566cd1574b74e59dd60ff6740a6c14af08c4fab4aa93670ac95f4c303ee4733f"} err="failed to get container status \"566cd1574b74e59dd60ff6740a6c14af08c4fab4aa93670ac95f4c303ee4733f\": rpc error: code = NotFound desc = could not find container \"566cd1574b74e59dd60ff6740a6c14af08c4fab4aa93670ac95f4c303ee4733f\": container with ID starting with 566cd1574b74e59dd60ff6740a6c14af08c4fab4aa93670ac95f4c303ee4733f not found: ID does not exist"
Jul 01 19:23:55.078547 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.078504    2574 scope.go:117] "RemoveContainer" containerID="ef7e1a29538a5be6215571aa29dcae391ab761663ce70c8275c83f10c9692768"
Jul 01 19:23:55.078703 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.078682    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef7e1a29538a5be6215571aa29dcae391ab761663ce70c8275c83f10c9692768"} err="failed to get container status \"ef7e1a29538a5be6215571aa29dcae391ab761663ce70c8275c83f10c9692768\": rpc error: code = NotFound desc = could not find container \"ef7e1a29538a5be6215571aa29dcae391ab761663ce70c8275c83f10c9692768\": container with ID starting with ef7e1a29538a5be6215571aa29dcae391ab761663ce70c8275c83f10c9692768 not found: ID does not exist"
Jul 01 19:23:55.078743 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.078702    2574 scope.go:117] "RemoveContainer" containerID="d00eaef05e60cf038464cfc9d0019a2ee1c3a8587f50964a4b284b4b5ef05d97"
Jul 01 19:23:55.078895 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.078878    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d00eaef05e60cf038464cfc9d0019a2ee1c3a8587f50964a4b284b4b5ef05d97"} err="failed to get container status \"d00eaef05e60cf038464cfc9d0019a2ee1c3a8587f50964a4b284b4b5ef05d97\": rpc error: code = NotFound desc = could not find container \"d00eaef05e60cf038464cfc9d0019a2ee1c3a8587f50964a4b284b4b5ef05d97\": container with ID starting with d00eaef05e60cf038464cfc9d0019a2ee1c3a8587f50964a4b284b4b5ef05d97 not found: ID does not exist"
Jul 01 19:23:55.078970 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.078948    2574 scope.go:117] "RemoveContainer" containerID="154069e46af698dc0af853d98e266e4650425bd004f701bf286bc107776d499e"
Jul 01 19:23:55.079137 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.079122    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"154069e46af698dc0af853d98e266e4650425bd004f701bf286bc107776d499e"} err="failed to get container status \"154069e46af698dc0af853d98e266e4650425bd004f701bf286bc107776d499e\": rpc error: code = NotFound desc = could not find container \"154069e46af698dc0af853d98e266e4650425bd004f701bf286bc107776d499e\": container with ID starting with 154069e46af698dc0af853d98e266e4650425bd004f701bf286bc107776d499e not found: ID does not exist"
Jul 01 19:23:55.079178 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.079138    2574 scope.go:117] "RemoveContainer" containerID="f6a7f7e46315cb04dc38830294efaee355c2a4cb6190213b102541ef6a6cb41c"
Jul 01 19:23:55.079316 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.079299    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6a7f7e46315cb04dc38830294efaee355c2a4cb6190213b102541ef6a6cb41c"} err="failed to get container status \"f6a7f7e46315cb04dc38830294efaee355c2a4cb6190213b102541ef6a6cb41c\": rpc error: code = NotFound desc = could not find container \"f6a7f7e46315cb04dc38830294efaee355c2a4cb6190213b102541ef6a6cb41c\": container with ID starting with f6a7f7e46315cb04dc38830294efaee355c2a4cb6190213b102541ef6a6cb41c not found: ID does not exist"
Jul 01 19:23:55.084674 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.084654    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 19:23:55.084964 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.084951    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" containerName="kube-rbac-proxy"
Jul 01 19:23:55.085009 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.084966    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" containerName="kube-rbac-proxy"
Jul 01 19:23:55.085009 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.084977    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cd41253-3f14-42d8-9893-3b95d85270c1" containerName="console"
Jul 01 19:23:55.085009 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.084983    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cd41253-3f14-42d8-9893-3b95d85270c1" containerName="console"
Jul 01 19:23:55.085009 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.084988    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" containerName="prom-label-proxy"
Jul 01 19:23:55.085009 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.084993    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" containerName="prom-label-proxy"
Jul 01 19:23:55.085009 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.085001    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" containerName="config-reloader"
Jul 01 19:23:55.085009 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.085006    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" containerName="config-reloader"
Jul 01 19:23:55.085208 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.085013    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" containerName="alertmanager"
Jul 01 19:23:55.085208 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.085018    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" containerName="alertmanager"
Jul 01 19:23:55.085208 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.085029    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" containerName="init-config-reloader"
Jul 01 19:23:55.085208 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.085035    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" containerName="init-config-reloader"
Jul 01 19:23:55.085208 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.085044    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" containerName="kube-rbac-proxy-web"
Jul 01 19:23:55.085208 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.085050    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" containerName="kube-rbac-proxy-web"
Jul 01 19:23:55.085208 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.085058    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" containerName="kube-rbac-proxy-metric"
Jul 01 19:23:55.085208 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.085063    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" containerName="kube-rbac-proxy-metric"
Jul 01 19:23:55.085208 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.085099    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" containerName="alertmanager"
Jul 01 19:23:55.085208 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.085105    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" containerName="kube-rbac-proxy-web"
Jul 01 19:23:55.085208 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.085110    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" containerName="kube-rbac-proxy-metric"
Jul 01 19:23:55.085208 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.085114    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" containerName="prom-label-proxy"
Jul 01 19:23:55.085208 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.085122    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cd41253-3f14-42d8-9893-3b95d85270c1" containerName="console"
Jul 01 19:23:55.085208 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.085127    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" containerName="config-reloader"
Jul 01 19:23:55.085208 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.085133    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" containerName="kube-rbac-proxy"
Jul 01 19:23:55.088347 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.088331    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:23:55.097256 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.097234    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 01 19:23:55.097352 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.097247    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 01 19:23:55.097424 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.097408    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 01 19:23:55.097542 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.097521    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 01 19:23:55.097683 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.097602    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 01 19:23:55.097683 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.097616    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rds5f\""
Jul 01 19:23:55.097683 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.097642    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 01 19:23:55.097892 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.097696    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 01 19:23:55.097892 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.097751    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 01 19:23:55.101856 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.101839    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 01 19:23:55.113443 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.113422    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 19:23:55.158999 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.158971    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3bf3ec80-44ad-4d65-a942-94c74f8aa1b3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:23:55.158999 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.159002    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bf3ec80-44ad-4d65-a942-94c74f8aa1b3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:23:55.159246 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.159027    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3bf3ec80-44ad-4d65-a942-94c74f8aa1b3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:23:55.159246 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.159076    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3bf3ec80-44ad-4d65-a942-94c74f8aa1b3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:23:55.159246 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.159124    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3bf3ec80-44ad-4d65-a942-94c74f8aa1b3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:23:55.159246 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.159178    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3bf3ec80-44ad-4d65-a942-94c74f8aa1b3-config-out\") pod \"alertmanager-main-0\" (UID: \"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:23:55.159246 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.159223    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3bf3ec80-44ad-4d65-a942-94c74f8aa1b3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:23:55.159464 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.159252    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3bf3ec80-44ad-4d65-a942-94c74f8aa1b3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:23:55.159464 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.159279    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3bf3ec80-44ad-4d65-a942-94c74f8aa1b3-web-config\") pod \"alertmanager-main-0\" (UID: \"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:23:55.159464 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.159309    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3bf3ec80-44ad-4d65-a942-94c74f8aa1b3-config-volume\") pod \"alertmanager-main-0\" (UID: \"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:23:55.159464 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.159332    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-299bl\" (UniqueName: \"kubernetes.io/projected/3bf3ec80-44ad-4d65-a942-94c74f8aa1b3-kube-api-access-299bl\") pod \"alertmanager-main-0\" (UID: \"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:23:55.159464 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.159360    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3bf3ec80-44ad-4d65-a942-94c74f8aa1b3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:23:55.159464 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.159413    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3bf3ec80-44ad-4d65-a942-94c74f8aa1b3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:23:55.260231 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.260156    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3bf3ec80-44ad-4d65-a942-94c74f8aa1b3-web-config\") pod \"alertmanager-main-0\" (UID: \"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:23:55.260231 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.260192    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3bf3ec80-44ad-4d65-a942-94c74f8aa1b3-config-volume\") pod \"alertmanager-main-0\" (UID: \"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:23:55.260231 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.260209    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-299bl\" (UniqueName: \"kubernetes.io/projected/3bf3ec80-44ad-4d65-a942-94c74f8aa1b3-kube-api-access-299bl\") pod \"alertmanager-main-0\" (UID: \"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:23:55.260231 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.260225    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3bf3ec80-44ad-4d65-a942-94c74f8aa1b3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:23:55.260499 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.260260    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3bf3ec80-44ad-4d65-a942-94c74f8aa1b3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:23:55.260499 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.260291    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3bf3ec80-44ad-4d65-a942-94c74f8aa1b3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:23:55.260499 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.260313    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bf3ec80-44ad-4d65-a942-94c74f8aa1b3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:23:55.260499 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.260346    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3bf3ec80-44ad-4d65-a942-94c74f8aa1b3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:23:55.260499 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.260367    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3bf3ec80-44ad-4d65-a942-94c74f8aa1b3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:23:55.260499 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.260395    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3bf3ec80-44ad-4d65-a942-94c74f8aa1b3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:23:55.260499 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.260446    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3bf3ec80-44ad-4d65-a942-94c74f8aa1b3-config-out\") pod \"alertmanager-main-0\" (UID: \"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:23:55.260499 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.260478    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3bf3ec80-44ad-4d65-a942-94c74f8aa1b3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:23:55.260499 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.260504    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3bf3ec80-44ad-4d65-a942-94c74f8aa1b3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:23:55.260969 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.260853    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3bf3ec80-44ad-4d65-a942-94c74f8aa1b3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:23:55.261348 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.261320    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bf3ec80-44ad-4d65-a942-94c74f8aa1b3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:23:55.263524 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.263326    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3bf3ec80-44ad-4d65-a942-94c74f8aa1b3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:23:55.263524 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.263349    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3bf3ec80-44ad-4d65-a942-94c74f8aa1b3-config-out\") pod \"alertmanager-main-0\" (UID: \"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:23:55.263524 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.263354    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3bf3ec80-44ad-4d65-a942-94c74f8aa1b3-config-volume\") pod \"alertmanager-main-0\" (UID: \"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:23:55.263524 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.263470    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3bf3ec80-44ad-4d65-a942-94c74f8aa1b3-web-config\") pod \"alertmanager-main-0\" (UID: \"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:23:55.263524 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.263492    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3bf3ec80-44ad-4d65-a942-94c74f8aa1b3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:23:55.263779 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.263647    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3bf3ec80-44ad-4d65-a942-94c74f8aa1b3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:23:55.263779 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.263662    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3bf3ec80-44ad-4d65-a942-94c74f8aa1b3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:23:55.263843 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.263804    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3bf3ec80-44ad-4d65-a942-94c74f8aa1b3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:23:55.263843 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.263805    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3bf3ec80-44ad-4d65-a942-94c74f8aa1b3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:23:55.265202 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.265184    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3bf3ec80-44ad-4d65-a942-94c74f8aa1b3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:23:55.272118 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.272101    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-299bl\" (UniqueName: \"kubernetes.io/projected/3bf3ec80-44ad-4d65-a942-94c74f8aa1b3-kube-api-access-299bl\") pod \"alertmanager-main-0\" (UID: \"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:23:55.398069 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.398025    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:23:55.537974 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:55.537949    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 19:23:55.539698 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:23:55.539669    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf3ec80_44ad_4d65_a942_94c74f8aa1b3.slice/crio-2589e17c6baa70879ba72828349ce32c71c651dfd3ec4949f3b31730f62eea32 WatchSource:0}: Error finding container 2589e17c6baa70879ba72828349ce32c71c651dfd3ec4949f3b31730f62eea32: Status 404 returned error can't find the container with id 2589e17c6baa70879ba72828349ce32c71c651dfd3ec4949f3b31730f62eea32
Jul 01 19:23:56.030794 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:56.030763    2574 generic.go:356] "Generic (PLEG): container finished" podID="3bf3ec80-44ad-4d65-a942-94c74f8aa1b3" containerID="bf77d66b78bb83c104942bd83e857972de3e887584f4de3c13e0385ce7ac47d7" exitCode=0
Jul 01 19:23:56.031185 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:56.030850    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3","Type":"ContainerDied","Data":"bf77d66b78bb83c104942bd83e857972de3e887584f4de3c13e0385ce7ac47d7"}
Jul 01 19:23:56.031185 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:56.030949    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3","Type":"ContainerStarted","Data":"2589e17c6baa70879ba72828349ce32c71c651dfd3ec4949f3b31730f62eea32"}
Jul 01 19:23:56.356005 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:56.355981    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a" path="/var/lib/kubelet/pods/ff6f6d4c-aa49-47bf-aa73-71da2eb3e80a/volumes"
Jul 01 19:23:57.039033 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:57.038998    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3","Type":"ContainerStarted","Data":"19d02e78e9692c1945922409ad692d1611fb79aa92bd5ddfdd251d7c70a5f50d"}
Jul 01 19:23:57.039405 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:57.039066    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3","Type":"ContainerStarted","Data":"d15edfa654d9e22a356bc005228f5433dd4bf3881d28742c63fdcba302037c39"}
Jul 01 19:23:57.039405 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:57.039094    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3","Type":"ContainerStarted","Data":"61db8f8afcd483e64f3df9f7a409853566a31fd143b72b547048f313ed175fb5"}
Jul 01 19:23:57.039405 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:57.039120    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3","Type":"ContainerStarted","Data":"9c11cc965502e1efb9fa6a2c8f02dd2a14df111f7c301a39509e0d5242e76895"}
Jul 01 19:23:57.039405 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:57.039145    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3","Type":"ContainerStarted","Data":"9050c17439b4885fe9c2e292e0d2fe15c1b1f946ef4caff36a5afcec95e632ae"}
Jul 01 19:23:57.039405 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:57.039169    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3bf3ec80-44ad-4d65-a942-94c74f8aa1b3","Type":"ContainerStarted","Data":"955c3bcd89084ddd1fc24beb93b6843f6e7ac37f344756de1eccf6447632ad5f"}
Jul 01 19:23:57.072252 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:57.072184    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.051885709 podStartE2EDuration="2.072165316s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.279607ms" isStatefulPod=false podCreationTimestamp="2026-07-01 19:23:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:23:57.071805075 +0000 UTC m=+143.163804544" watchObservedRunningTime="2026-07-01 19:23:57.072165316 +0000 UTC m=+143.164164766"
Jul 01 19:23:57.701681 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:57.701641    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7cdc9756fd-hntp7"]
Jul 01 19:23:57.704388 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:57.704365    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7cdc9756fd-hntp7"
Jul 01 19:23:57.708607 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:57.708589    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 01 19:23:57.709396 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:57.709377    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 01 19:23:57.709614 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:57.709591    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 01 19:23:57.709717 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:57.709633    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 01 19:23:57.710140 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:57.710123    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 01 19:23:57.713108 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:57.713087    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 01 19:23:57.715573 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:57.715557    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-d92nl\""
Jul 01 19:23:57.728654 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:57.728632    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7cdc9756fd-hntp7"]
Jul 01 19:23:57.777518 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:57.777488    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/79bc6c96-6740-4296-8fa4-a0da34a98370-telemeter-client-tls\") pod \"telemeter-client-7cdc9756fd-hntp7\" (UID: \"79bc6c96-6740-4296-8fa4-a0da34a98370\") " pod="openshift-monitoring/telemeter-client-7cdc9756fd-hntp7"
Jul 01 19:23:57.777663 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:57.777596    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5w6j\" (UniqueName: \"kubernetes.io/projected/79bc6c96-6740-4296-8fa4-a0da34a98370-kube-api-access-k5w6j\") pod \"telemeter-client-7cdc9756fd-hntp7\" (UID: \"79bc6c96-6740-4296-8fa4-a0da34a98370\") " pod="openshift-monitoring/telemeter-client-7cdc9756fd-hntp7"
Jul 01 19:23:57.777663 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:57.777634    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/79bc6c96-6740-4296-8fa4-a0da34a98370-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7cdc9756fd-hntp7\" (UID: \"79bc6c96-6740-4296-8fa4-a0da34a98370\") " pod="openshift-monitoring/telemeter-client-7cdc9756fd-hntp7"
Jul 01 19:23:57.777783 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:57.777674    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/79bc6c96-6740-4296-8fa4-a0da34a98370-secret-telemeter-client\") pod \"telemeter-client-7cdc9756fd-hntp7\" (UID: \"79bc6c96-6740-4296-8fa4-a0da34a98370\") " pod="openshift-monitoring/telemeter-client-7cdc9756fd-hntp7"
Jul 01 19:23:57.777783 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:57.777703    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79bc6c96-6740-4296-8fa4-a0da34a98370-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7cdc9756fd-hntp7\" (UID: \"79bc6c96-6740-4296-8fa4-a0da34a98370\") " pod="openshift-monitoring/telemeter-client-7cdc9756fd-hntp7"
Jul 01 19:23:57.777783 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:57.777742    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/79bc6c96-6740-4296-8fa4-a0da34a98370-federate-client-tls\") pod \"telemeter-client-7cdc9756fd-hntp7\" (UID: \"79bc6c96-6740-4296-8fa4-a0da34a98370\") " pod="openshift-monitoring/telemeter-client-7cdc9756fd-hntp7"
Jul 01 19:23:57.777783 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:57.777764    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79bc6c96-6740-4296-8fa4-a0da34a98370-metrics-client-ca\") pod \"telemeter-client-7cdc9756fd-hntp7\" (UID: \"79bc6c96-6740-4296-8fa4-a0da34a98370\") " pod="openshift-monitoring/telemeter-client-7cdc9756fd-hntp7"
Jul 01 19:23:57.777989 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:57.777794    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79bc6c96-6740-4296-8fa4-a0da34a98370-serving-certs-ca-bundle\") pod \"telemeter-client-7cdc9756fd-hntp7\" (UID: \"79bc6c96-6740-4296-8fa4-a0da34a98370\") " pod="openshift-monitoring/telemeter-client-7cdc9756fd-hntp7"
Jul 01 19:23:57.879041 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:57.878996    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/79bc6c96-6740-4296-8fa4-a0da34a98370-secret-telemeter-client\") pod \"telemeter-client-7cdc9756fd-hntp7\" (UID: \"79bc6c96-6740-4296-8fa4-a0da34a98370\") " pod="openshift-monitoring/telemeter-client-7cdc9756fd-hntp7"
Jul 01 19:23:57.879041 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:57.879044    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79bc6c96-6740-4296-8fa4-a0da34a98370-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7cdc9756fd-hntp7\" (UID: \"79bc6c96-6740-4296-8fa4-a0da34a98370\") " pod="openshift-monitoring/telemeter-client-7cdc9756fd-hntp7"
Jul 01 19:23:57.879286 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:57.879080    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/79bc6c96-6740-4296-8fa4-a0da34a98370-federate-client-tls\") pod \"telemeter-client-7cdc9756fd-hntp7\" (UID: \"79bc6c96-6740-4296-8fa4-a0da34a98370\") " pod="openshift-monitoring/telemeter-client-7cdc9756fd-hntp7"
Jul 01 19:23:57.879286 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:57.879102    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79bc6c96-6740-4296-8fa4-a0da34a98370-metrics-client-ca\") pod \"telemeter-client-7cdc9756fd-hntp7\" (UID: \"79bc6c96-6740-4296-8fa4-a0da34a98370\") " pod="openshift-monitoring/telemeter-client-7cdc9756fd-hntp7"
Jul 01 19:23:57.879286 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:57.879128    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79bc6c96-6740-4296-8fa4-a0da34a98370-serving-certs-ca-bundle\") pod \"telemeter-client-7cdc9756fd-hntp7\" (UID: \"79bc6c96-6740-4296-8fa4-a0da34a98370\") " pod="openshift-monitoring/telemeter-client-7cdc9756fd-hntp7"
Jul 01 19:23:57.879286 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:57.879159    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/79bc6c96-6740-4296-8fa4-a0da34a98370-telemeter-client-tls\") pod \"telemeter-client-7cdc9756fd-hntp7\" (UID: \"79bc6c96-6740-4296-8fa4-a0da34a98370\") " pod="openshift-monitoring/telemeter-client-7cdc9756fd-hntp7"
Jul 01 19:23:57.879286 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:57.879204    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5w6j\" (UniqueName: \"kubernetes.io/projected/79bc6c96-6740-4296-8fa4-a0da34a98370-kube-api-access-k5w6j\") pod \"telemeter-client-7cdc9756fd-hntp7\" (UID: \"79bc6c96-6740-4296-8fa4-a0da34a98370\") " pod="openshift-monitoring/telemeter-client-7cdc9756fd-hntp7"
Jul 01 19:23:57.879286 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:57.879238    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/79bc6c96-6740-4296-8fa4-a0da34a98370-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7cdc9756fd-hntp7\" (UID: \"79bc6c96-6740-4296-8fa4-a0da34a98370\") " pod="openshift-monitoring/telemeter-client-7cdc9756fd-hntp7"
Jul 01 19:23:57.880114 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:57.880085    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79bc6c96-6740-4296-8fa4-a0da34a98370-metrics-client-ca\") pod \"telemeter-client-7cdc9756fd-hntp7\" (UID: \"79bc6c96-6740-4296-8fa4-a0da34a98370\") " pod="openshift-monitoring/telemeter-client-7cdc9756fd-hntp7"
Jul 01 19:23:57.880484 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:57.880434    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79bc6c96-6740-4296-8fa4-a0da34a98370-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7cdc9756fd-hntp7\" (UID: \"79bc6c96-6740-4296-8fa4-a0da34a98370\") " pod="openshift-monitoring/telemeter-client-7cdc9756fd-hntp7"
Jul 01 19:23:57.883144 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:57.880662    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79bc6c96-6740-4296-8fa4-a0da34a98370-serving-certs-ca-bundle\") pod \"telemeter-client-7cdc9756fd-hntp7\" (UID: \"79bc6c96-6740-4296-8fa4-a0da34a98370\") " pod="openshift-monitoring/telemeter-client-7cdc9756fd-hntp7"
Jul 01 19:23:57.883144 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:57.882160    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/79bc6c96-6740-4296-8fa4-a0da34a98370-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7cdc9756fd-hntp7\" (UID: \"79bc6c96-6740-4296-8fa4-a0da34a98370\") " pod="openshift-monitoring/telemeter-client-7cdc9756fd-hntp7"
Jul 01 19:23:57.883144 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:57.882236    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/79bc6c96-6740-4296-8fa4-a0da34a98370-secret-telemeter-client\") pod \"telemeter-client-7cdc9756fd-hntp7\" (UID: \"79bc6c96-6740-4296-8fa4-a0da34a98370\") " pod="openshift-monitoring/telemeter-client-7cdc9756fd-hntp7"
Jul 01 19:23:57.883144 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:57.882553    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/79bc6c96-6740-4296-8fa4-a0da34a98370-telemeter-client-tls\") pod \"telemeter-client-7cdc9756fd-hntp7\" (UID: \"79bc6c96-6740-4296-8fa4-a0da34a98370\") " pod="openshift-monitoring/telemeter-client-7cdc9756fd-hntp7"
Jul 01 19:23:57.883652 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:57.883636    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/79bc6c96-6740-4296-8fa4-a0da34a98370-federate-client-tls\") pod \"telemeter-client-7cdc9756fd-hntp7\" (UID: \"79bc6c96-6740-4296-8fa4-a0da34a98370\") " pod="openshift-monitoring/telemeter-client-7cdc9756fd-hntp7"
Jul 01 19:23:57.888622 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:57.888597    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5w6j\" (UniqueName: \"kubernetes.io/projected/79bc6c96-6740-4296-8fa4-a0da34a98370-kube-api-access-k5w6j\") pod \"telemeter-client-7cdc9756fd-hntp7\" (UID: \"79bc6c96-6740-4296-8fa4-a0da34a98370\") " pod="openshift-monitoring/telemeter-client-7cdc9756fd-hntp7"
Jul 01 19:23:58.014479 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:58.014378    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7cdc9756fd-hntp7"
Jul 01 19:23:58.191432 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:58.191393    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7cdc9756fd-hntp7"]
Jul 01 19:23:58.195489 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:23:58.195461    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79bc6c96_6740_4296_8fa4_a0da34a98370.slice/crio-4719e1540934961029c353fab8360d2c03cf244881d867505b468723212efa5a WatchSource:0}: Error finding container 4719e1540934961029c353fab8360d2c03cf244881d867505b468723212efa5a: Status 404 returned error can't find the container with id 4719e1540934961029c353fab8360d2c03cf244881d867505b468723212efa5a
Jul 01 19:23:59.051428 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:23:59.051396    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7cdc9756fd-hntp7" event={"ID":"79bc6c96-6740-4296-8fa4-a0da34a98370","Type":"ContainerStarted","Data":"4719e1540934961029c353fab8360d2c03cf244881d867505b468723212efa5a"}
Jul 01 19:24:01.062552 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:01.062513    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7cdc9756fd-hntp7" event={"ID":"79bc6c96-6740-4296-8fa4-a0da34a98370","Type":"ContainerStarted","Data":"0d1753892257b6e3d7fe50661aee705e2bc13efe3a04fcdf0f49ee19fd8734cc"}
Jul 01 19:24:01.062973 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:01.062572    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7cdc9756fd-hntp7" event={"ID":"79bc6c96-6740-4296-8fa4-a0da34a98370","Type":"ContainerStarted","Data":"2587bcb6b1448ce24247bc9b2440bab107478351bc8cbc1917e217bc583b4312"}
Jul 01 19:24:01.062973 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:01.062593    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7cdc9756fd-hntp7" event={"ID":"79bc6c96-6740-4296-8fa4-a0da34a98370","Type":"ContainerStarted","Data":"0e0740d686bfc58c6440eb523e1b54e9507c76fe48cd93843b5f303e6b3dc417"}
Jul 01 19:24:01.103365 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:01.103312    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7cdc9756fd-hntp7" podStartSLOduration=1.907965122 podStartE2EDuration="4.103296239s" totalImagesPullingTime="2.195331117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:23:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:24:01.102474296 +0000 UTC m=+147.194473745" watchObservedRunningTime="2026-07-01 19:24:01.103296239 +0000 UTC m=+147.195295686"
Jul 01 19:24:01.987921 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:01.987860    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5595bf5c5b-6dntk"]
Jul 01 19:24:01.990465 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:01.990437    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5595bf5c5b-6dntk"
Jul 01 19:24:02.006410 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:02.006382    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5595bf5c5b-6dntk"]
Jul 01 19:24:02.128586 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:02.128554    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30212148-a8b1-496b-9ba3-e0504ba8a3f3-console-config\") pod \"console-5595bf5c5b-6dntk\" (UID: \"30212148-a8b1-496b-9ba3-e0504ba8a3f3\") " pod="openshift-console/console-5595bf5c5b-6dntk"
Jul 01 19:24:02.129133 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:02.128601    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30212148-a8b1-496b-9ba3-e0504ba8a3f3-console-serving-cert\") pod \"console-5595bf5c5b-6dntk\" (UID: \"30212148-a8b1-496b-9ba3-e0504ba8a3f3\") " pod="openshift-console/console-5595bf5c5b-6dntk"
Jul 01 19:24:02.129133 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:02.128633    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq4qk\" (UniqueName: \"kubernetes.io/projected/30212148-a8b1-496b-9ba3-e0504ba8a3f3-kube-api-access-tq4qk\") pod \"console-5595bf5c5b-6dntk\" (UID: \"30212148-a8b1-496b-9ba3-e0504ba8a3f3\") " pod="openshift-console/console-5595bf5c5b-6dntk"
Jul 01 19:24:02.129133 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:02.128754    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30212148-a8b1-496b-9ba3-e0504ba8a3f3-service-ca\") pod \"console-5595bf5c5b-6dntk\" (UID: \"30212148-a8b1-496b-9ba3-e0504ba8a3f3\") " pod="openshift-console/console-5595bf5c5b-6dntk"
Jul 01 19:24:02.129133 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:02.128787    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30212148-a8b1-496b-9ba3-e0504ba8a3f3-oauth-serving-cert\") pod \"console-5595bf5c5b-6dntk\" (UID: \"30212148-a8b1-496b-9ba3-e0504ba8a3f3\") " pod="openshift-console/console-5595bf5c5b-6dntk"
Jul 01 19:24:02.129133 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:02.128866    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30212148-a8b1-496b-9ba3-e0504ba8a3f3-console-oauth-config\") pod \"console-5595bf5c5b-6dntk\" (UID: \"30212148-a8b1-496b-9ba3-e0504ba8a3f3\") " pod="openshift-console/console-5595bf5c5b-6dntk"
Jul 01 19:24:02.129133 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:02.128929    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30212148-a8b1-496b-9ba3-e0504ba8a3f3-trusted-ca-bundle\") pod \"console-5595bf5c5b-6dntk\" (UID: \"30212148-a8b1-496b-9ba3-e0504ba8a3f3\") " pod="openshift-console/console-5595bf5c5b-6dntk"
Jul 01 19:24:02.229530 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:02.229488    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30212148-a8b1-496b-9ba3-e0504ba8a3f3-service-ca\") pod \"console-5595bf5c5b-6dntk\" (UID: \"30212148-a8b1-496b-9ba3-e0504ba8a3f3\") " pod="openshift-console/console-5595bf5c5b-6dntk"
Jul 01 19:24:02.229530 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:02.229540    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30212148-a8b1-496b-9ba3-e0504ba8a3f3-oauth-serving-cert\") pod \"console-5595bf5c5b-6dntk\" (UID: \"30212148-a8b1-496b-9ba3-e0504ba8a3f3\") " pod="openshift-console/console-5595bf5c5b-6dntk"
Jul 01 19:24:02.229797 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:02.229582    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30212148-a8b1-496b-9ba3-e0504ba8a3f3-console-oauth-config\") pod \"console-5595bf5c5b-6dntk\" (UID: \"30212148-a8b1-496b-9ba3-e0504ba8a3f3\") " pod="openshift-console/console-5595bf5c5b-6dntk"
Jul 01 19:24:02.229797 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:02.229607    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30212148-a8b1-496b-9ba3-e0504ba8a3f3-trusted-ca-bundle\") pod \"console-5595bf5c5b-6dntk\" (UID: \"30212148-a8b1-496b-9ba3-e0504ba8a3f3\") " pod="openshift-console/console-5595bf5c5b-6dntk"
Jul 01 19:24:02.229797 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:02.229631    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30212148-a8b1-496b-9ba3-e0504ba8a3f3-console-config\") pod \"console-5595bf5c5b-6dntk\" (UID: \"30212148-a8b1-496b-9ba3-e0504ba8a3f3\") " pod="openshift-console/console-5595bf5c5b-6dntk"
Jul 01 19:24:02.229797 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:02.229656    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30212148-a8b1-496b-9ba3-e0504ba8a3f3-console-serving-cert\") pod \"console-5595bf5c5b-6dntk\" (UID: \"30212148-a8b1-496b-9ba3-e0504ba8a3f3\") " pod="openshift-console/console-5595bf5c5b-6dntk"
Jul 01 19:24:02.229797 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:02.229690    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq4qk\" (UniqueName: \"kubernetes.io/projected/30212148-a8b1-496b-9ba3-e0504ba8a3f3-kube-api-access-tq4qk\") pod \"console-5595bf5c5b-6dntk\" (UID: \"30212148-a8b1-496b-9ba3-e0504ba8a3f3\") " pod="openshift-console/console-5595bf5c5b-6dntk"
Jul 01 19:24:02.231029 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:02.230854    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30212148-a8b1-496b-9ba3-e0504ba8a3f3-service-ca\") pod \"console-5595bf5c5b-6dntk\" (UID: \"30212148-a8b1-496b-9ba3-e0504ba8a3f3\") " pod="openshift-console/console-5595bf5c5b-6dntk"
Jul 01 19:24:02.231029 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:02.230854    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30212148-a8b1-496b-9ba3-e0504ba8a3f3-console-config\") pod \"console-5595bf5c5b-6dntk\" (UID: \"30212148-a8b1-496b-9ba3-e0504ba8a3f3\") " pod="openshift-console/console-5595bf5c5b-6dntk"
Jul 01 19:24:02.231736 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:02.231505    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30212148-a8b1-496b-9ba3-e0504ba8a3f3-oauth-serving-cert\") pod \"console-5595bf5c5b-6dntk\" (UID: \"30212148-a8b1-496b-9ba3-e0504ba8a3f3\") " pod="openshift-console/console-5595bf5c5b-6dntk"
Jul 01 19:24:02.231912 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:02.231884    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30212148-a8b1-496b-9ba3-e0504ba8a3f3-trusted-ca-bundle\") pod \"console-5595bf5c5b-6dntk\" (UID: \"30212148-a8b1-496b-9ba3-e0504ba8a3f3\") " pod="openshift-console/console-5595bf5c5b-6dntk"
Jul 01 19:24:02.233831 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:02.233810    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30212148-a8b1-496b-9ba3-e0504ba8a3f3-console-oauth-config\") pod \"console-5595bf5c5b-6dntk\" (UID: \"30212148-a8b1-496b-9ba3-e0504ba8a3f3\") " pod="openshift-console/console-5595bf5c5b-6dntk"
Jul 01 19:24:02.233967 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:02.233950    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30212148-a8b1-496b-9ba3-e0504ba8a3f3-console-serving-cert\") pod \"console-5595bf5c5b-6dntk\" (UID: \"30212148-a8b1-496b-9ba3-e0504ba8a3f3\") " pod="openshift-console/console-5595bf5c5b-6dntk"
Jul 01 19:24:02.243667 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:02.243578    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq4qk\" (UniqueName: \"kubernetes.io/projected/30212148-a8b1-496b-9ba3-e0504ba8a3f3-kube-api-access-tq4qk\") pod \"console-5595bf5c5b-6dntk\" (UID: \"30212148-a8b1-496b-9ba3-e0504ba8a3f3\") " pod="openshift-console/console-5595bf5c5b-6dntk"
Jul 01 19:24:02.302292 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:02.302259    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5595bf5c5b-6dntk"
Jul 01 19:24:02.443476 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:02.443430    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5595bf5c5b-6dntk"]
Jul 01 19:24:02.445593 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:24:02.445564    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30212148_a8b1_496b_9ba3_e0504ba8a3f3.slice/crio-76ffe216b4993166a93a6dcb0410804832208ccbe43b2cf0305805b24d04aed6 WatchSource:0}: Error finding container 76ffe216b4993166a93a6dcb0410804832208ccbe43b2cf0305805b24d04aed6: Status 404 returned error can't find the container with id 76ffe216b4993166a93a6dcb0410804832208ccbe43b2cf0305805b24d04aed6
Jul 01 19:24:03.072126 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:03.072085    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5595bf5c5b-6dntk" event={"ID":"30212148-a8b1-496b-9ba3-e0504ba8a3f3","Type":"ContainerStarted","Data":"48e0a54714ff496143a9084e51ec8d3df929cb20b0b08b26a4613f7f17768221"}
Jul 01 19:24:03.072290 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:03.072145    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5595bf5c5b-6dntk" event={"ID":"30212148-a8b1-496b-9ba3-e0504ba8a3f3","Type":"ContainerStarted","Data":"76ffe216b4993166a93a6dcb0410804832208ccbe43b2cf0305805b24d04aed6"}
Jul 01 19:24:03.093692 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:03.093650    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5595bf5c5b-6dntk" podStartSLOduration=2.093635736 podStartE2EDuration="2.093635736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:24:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:24:03.092423959 +0000 UTC m=+149.184423422" watchObservedRunningTime="2026-07-01 19:24:03.093635736 +0000 UTC m=+149.185635185"
Jul 01 19:24:12.302773 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:12.302734    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5595bf5c5b-6dntk"
Jul 01 19:24:12.303207 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:12.302794    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5595bf5c5b-6dntk"
Jul 01 19:24:12.307371 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:12.307347    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5595bf5c5b-6dntk"
Jul 01 19:24:13.113647 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:13.113619    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5595bf5c5b-6dntk"
Jul 01 19:24:13.167804 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:13.167775    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bb4976c74-5rs24"]
Jul 01 19:24:38.194131 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:38.194069    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-5bb4976c74-5rs24" podUID="b81156d0-bfc6-4255-95d5-55844116eebe" containerName="console" containerID="cri-o://8de4f66bd6761882b546ae2256db99154999acdb6312adf663ccdbfb2e3b8046" gracePeriod=15
Jul 01 19:24:38.449887 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:38.449831    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bb4976c74-5rs24"
Jul 01 19:24:38.538116 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:38.538081    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b81156d0-bfc6-4255-95d5-55844116eebe-console-serving-cert\") pod \"b81156d0-bfc6-4255-95d5-55844116eebe\" (UID: \"b81156d0-bfc6-4255-95d5-55844116eebe\") "
Jul 01 19:24:38.538280 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:38.538128    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b81156d0-bfc6-4255-95d5-55844116eebe-oauth-serving-cert\") pod \"b81156d0-bfc6-4255-95d5-55844116eebe\" (UID: \"b81156d0-bfc6-4255-95d5-55844116eebe\") "
Jul 01 19:24:38.538280 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:38.538145    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b81156d0-bfc6-4255-95d5-55844116eebe-trusted-ca-bundle\") pod \"b81156d0-bfc6-4255-95d5-55844116eebe\" (UID: \"b81156d0-bfc6-4255-95d5-55844116eebe\") "
Jul 01 19:24:38.538280 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:38.538180    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b81156d0-bfc6-4255-95d5-55844116eebe-service-ca\") pod \"b81156d0-bfc6-4255-95d5-55844116eebe\" (UID: \"b81156d0-bfc6-4255-95d5-55844116eebe\") "
Jul 01 19:24:38.538280 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:38.538208    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b81156d0-bfc6-4255-95d5-55844116eebe-console-oauth-config\") pod \"b81156d0-bfc6-4255-95d5-55844116eebe\" (UID: \"b81156d0-bfc6-4255-95d5-55844116eebe\") "
Jul 01 19:24:38.538280 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:38.538261    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d86mb\" (UniqueName: \"kubernetes.io/projected/b81156d0-bfc6-4255-95d5-55844116eebe-kube-api-access-d86mb\") pod \"b81156d0-bfc6-4255-95d5-55844116eebe\" (UID: \"b81156d0-bfc6-4255-95d5-55844116eebe\") "
Jul 01 19:24:38.538530 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:38.538289    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b81156d0-bfc6-4255-95d5-55844116eebe-console-config\") pod \"b81156d0-bfc6-4255-95d5-55844116eebe\" (UID: \"b81156d0-bfc6-4255-95d5-55844116eebe\") "
Jul 01 19:24:38.538530 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:38.538495    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b81156d0-bfc6-4255-95d5-55844116eebe-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b81156d0-bfc6-4255-95d5-55844116eebe" (UID: "b81156d0-bfc6-4255-95d5-55844116eebe"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:24:38.538633 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:38.538537    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b81156d0-bfc6-4255-95d5-55844116eebe-service-ca" (OuterVolumeSpecName: "service-ca") pod "b81156d0-bfc6-4255-95d5-55844116eebe" (UID: "b81156d0-bfc6-4255-95d5-55844116eebe"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:24:38.538765 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:38.538739    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b81156d0-bfc6-4255-95d5-55844116eebe-console-config" (OuterVolumeSpecName: "console-config") pod "b81156d0-bfc6-4255-95d5-55844116eebe" (UID: "b81156d0-bfc6-4255-95d5-55844116eebe"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:24:38.538833 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:38.538817    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b81156d0-bfc6-4255-95d5-55844116eebe-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b81156d0-bfc6-4255-95d5-55844116eebe" (UID: "b81156d0-bfc6-4255-95d5-55844116eebe"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:24:38.540525 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:38.540498    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b81156d0-bfc6-4255-95d5-55844116eebe-kube-api-access-d86mb" (OuterVolumeSpecName: "kube-api-access-d86mb") pod "b81156d0-bfc6-4255-95d5-55844116eebe" (UID: "b81156d0-bfc6-4255-95d5-55844116eebe"). InnerVolumeSpecName "kube-api-access-d86mb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:24:38.540525 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:38.540509    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b81156d0-bfc6-4255-95d5-55844116eebe-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b81156d0-bfc6-4255-95d5-55844116eebe" (UID: "b81156d0-bfc6-4255-95d5-55844116eebe"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:24:38.540666 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:38.540557    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b81156d0-bfc6-4255-95d5-55844116eebe-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b81156d0-bfc6-4255-95d5-55844116eebe" (UID: "b81156d0-bfc6-4255-95d5-55844116eebe"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:24:38.639090 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:38.639042    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d86mb\" (UniqueName: \"kubernetes.io/projected/b81156d0-bfc6-4255-95d5-55844116eebe-kube-api-access-d86mb\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:24:38.639090 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:38.639085    2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b81156d0-bfc6-4255-95d5-55844116eebe-console-config\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:24:38.639090 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:38.639095    2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b81156d0-bfc6-4255-95d5-55844116eebe-console-serving-cert\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:24:38.639090 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:38.639103    2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b81156d0-bfc6-4255-95d5-55844116eebe-oauth-serving-cert\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:24:38.639353 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:38.639112    2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b81156d0-bfc6-4255-95d5-55844116eebe-trusted-ca-bundle\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:24:38.639353 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:38.639121    2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b81156d0-bfc6-4255-95d5-55844116eebe-service-ca\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:24:38.639353 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:38.639130    2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b81156d0-bfc6-4255-95d5-55844116eebe-console-oauth-config\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:24:39.206154 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:39.206121    2574 generic.go:356] "Generic (PLEG): container finished" podID="b81156d0-bfc6-4255-95d5-55844116eebe" containerID="8de4f66bd6761882b546ae2256db99154999acdb6312adf663ccdbfb2e3b8046" exitCode=0
Jul 01 19:24:39.206626 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:39.206174    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bb4976c74-5rs24"
Jul 01 19:24:39.206626 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:39.206208    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bb4976c74-5rs24" event={"ID":"b81156d0-bfc6-4255-95d5-55844116eebe","Type":"ContainerDied","Data":"8de4f66bd6761882b546ae2256db99154999acdb6312adf663ccdbfb2e3b8046"}
Jul 01 19:24:39.206626 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:39.206257    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bb4976c74-5rs24" event={"ID":"b81156d0-bfc6-4255-95d5-55844116eebe","Type":"ContainerDied","Data":"ae4e74e88c327eaccc5ca6a827fcb9cc3236c8171805579b3a2055a1ed1dbab7"}
Jul 01 19:24:39.206626 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:39.206277    2574 scope.go:117] "RemoveContainer" containerID="8de4f66bd6761882b546ae2256db99154999acdb6312adf663ccdbfb2e3b8046"
Jul 01 19:24:39.214559 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:39.214544    2574 scope.go:117] "RemoveContainer" containerID="8de4f66bd6761882b546ae2256db99154999acdb6312adf663ccdbfb2e3b8046"
Jul 01 19:24:39.214820 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:24:39.214802    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8de4f66bd6761882b546ae2256db99154999acdb6312adf663ccdbfb2e3b8046\": container with ID starting with 8de4f66bd6761882b546ae2256db99154999acdb6312adf663ccdbfb2e3b8046 not found: ID does not exist" containerID="8de4f66bd6761882b546ae2256db99154999acdb6312adf663ccdbfb2e3b8046"
Jul 01 19:24:39.214861 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:39.214827    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8de4f66bd6761882b546ae2256db99154999acdb6312adf663ccdbfb2e3b8046"} err="failed to get container status \"8de4f66bd6761882b546ae2256db99154999acdb6312adf663ccdbfb2e3b8046\": rpc error: code = NotFound desc = could not find container \"8de4f66bd6761882b546ae2256db99154999acdb6312adf663ccdbfb2e3b8046\": container with ID starting with 8de4f66bd6761882b546ae2256db99154999acdb6312adf663ccdbfb2e3b8046 not found: ID does not exist"
Jul 01 19:24:39.226602 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:39.226581    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bb4976c74-5rs24"]
Jul 01 19:24:39.230276 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:39.230257    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5bb4976c74-5rs24"]
Jul 01 19:24:40.355378 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:24:40.355345    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b81156d0-bfc6-4255-95d5-55844116eebe" path="/var/lib/kubelet/pods/b81156d0-bfc6-4255-95d5-55844116eebe/volumes"
Jul 01 19:25:24.076912 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:24.076869    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c748d989c-rn88n"]
Jul 01 19:25:24.077380 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:24.077231    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b81156d0-bfc6-4255-95d5-55844116eebe" containerName="console"
Jul 01 19:25:24.077380 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:24.077244    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b81156d0-bfc6-4255-95d5-55844116eebe" containerName="console"
Jul 01 19:25:24.077380 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:24.077299    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b81156d0-bfc6-4255-95d5-55844116eebe" containerName="console"
Jul 01 19:25:24.079319 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:24.079304    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c748d989c-rn88n"
Jul 01 19:25:24.094636 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:24.094610    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c748d989c-rn88n"]
Jul 01 19:25:24.102280 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:24.102252    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d1522f35-27a3-4a30-9550-2e10b2f98893-oauth-serving-cert\") pod \"console-6c748d989c-rn88n\" (UID: \"d1522f35-27a3-4a30-9550-2e10b2f98893\") " pod="openshift-console/console-6c748d989c-rn88n"
Jul 01 19:25:24.102403 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:24.102290    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d1522f35-27a3-4a30-9550-2e10b2f98893-console-config\") pod \"console-6c748d989c-rn88n\" (UID: \"d1522f35-27a3-4a30-9550-2e10b2f98893\") " pod="openshift-console/console-6c748d989c-rn88n"
Jul 01 19:25:24.102403 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:24.102313    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d1522f35-27a3-4a30-9550-2e10b2f98893-service-ca\") pod \"console-6c748d989c-rn88n\" (UID: \"d1522f35-27a3-4a30-9550-2e10b2f98893\") " pod="openshift-console/console-6c748d989c-rn88n"
Jul 01 19:25:24.102484 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:24.102434    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1522f35-27a3-4a30-9550-2e10b2f98893-trusted-ca-bundle\") pod \"console-6c748d989c-rn88n\" (UID: \"d1522f35-27a3-4a30-9550-2e10b2f98893\") " pod="openshift-console/console-6c748d989c-rn88n"
Jul 01 19:25:24.102540 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:24.102481    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d1522f35-27a3-4a30-9550-2e10b2f98893-console-oauth-config\") pod \"console-6c748d989c-rn88n\" (UID: \"d1522f35-27a3-4a30-9550-2e10b2f98893\") " pod="openshift-console/console-6c748d989c-rn88n"
Jul 01 19:25:24.102591 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:24.102533    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d1522f35-27a3-4a30-9550-2e10b2f98893-console-serving-cert\") pod \"console-6c748d989c-rn88n\" (UID: \"d1522f35-27a3-4a30-9550-2e10b2f98893\") " pod="openshift-console/console-6c748d989c-rn88n"
Jul 01 19:25:24.102591 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:24.102573    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqck8\" (UniqueName: \"kubernetes.io/projected/d1522f35-27a3-4a30-9550-2e10b2f98893-kube-api-access-wqck8\") pod \"console-6c748d989c-rn88n\" (UID: \"d1522f35-27a3-4a30-9550-2e10b2f98893\") " pod="openshift-console/console-6c748d989c-rn88n"
Jul 01 19:25:24.203326 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:24.203285    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d1522f35-27a3-4a30-9550-2e10b2f98893-oauth-serving-cert\") pod \"console-6c748d989c-rn88n\" (UID: \"d1522f35-27a3-4a30-9550-2e10b2f98893\") " pod="openshift-console/console-6c748d989c-rn88n"
Jul 01 19:25:24.203523 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:24.203337    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d1522f35-27a3-4a30-9550-2e10b2f98893-console-config\") pod \"console-6c748d989c-rn88n\" (UID: \"d1522f35-27a3-4a30-9550-2e10b2f98893\") " pod="openshift-console/console-6c748d989c-rn88n"
Jul 01 19:25:24.203523 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:24.203380    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d1522f35-27a3-4a30-9550-2e10b2f98893-service-ca\") pod \"console-6c748d989c-rn88n\" (UID: \"d1522f35-27a3-4a30-9550-2e10b2f98893\") " pod="openshift-console/console-6c748d989c-rn88n"
Jul 01 19:25:24.203523 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:24.203429    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1522f35-27a3-4a30-9550-2e10b2f98893-trusted-ca-bundle\") pod \"console-6c748d989c-rn88n\" (UID: \"d1522f35-27a3-4a30-9550-2e10b2f98893\") " pod="openshift-console/console-6c748d989c-rn88n"
Jul 01 19:25:24.203523 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:24.203460    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d1522f35-27a3-4a30-9550-2e10b2f98893-console-oauth-config\") pod \"console-6c748d989c-rn88n\" (UID: \"d1522f35-27a3-4a30-9550-2e10b2f98893\") " pod="openshift-console/console-6c748d989c-rn88n"
Jul 01 19:25:24.203523 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:24.203490    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d1522f35-27a3-4a30-9550-2e10b2f98893-console-serving-cert\") pod \"console-6c748d989c-rn88n\" (UID: \"d1522f35-27a3-4a30-9550-2e10b2f98893\") " pod="openshift-console/console-6c748d989c-rn88n"
Jul 01 19:25:24.203776 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:24.203523    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqck8\" (UniqueName: \"kubernetes.io/projected/d1522f35-27a3-4a30-9550-2e10b2f98893-kube-api-access-wqck8\") pod \"console-6c748d989c-rn88n\" (UID: \"d1522f35-27a3-4a30-9550-2e10b2f98893\") " pod="openshift-console/console-6c748d989c-rn88n"
Jul 01 19:25:24.204206 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:24.204163    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d1522f35-27a3-4a30-9550-2e10b2f98893-oauth-serving-cert\") pod \"console-6c748d989c-rn88n\" (UID: \"d1522f35-27a3-4a30-9550-2e10b2f98893\") " pod="openshift-console/console-6c748d989c-rn88n"
Jul 01 19:25:24.204206 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:24.204204    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d1522f35-27a3-4a30-9550-2e10b2f98893-console-config\") pod \"console-6c748d989c-rn88n\" (UID: \"d1522f35-27a3-4a30-9550-2e10b2f98893\") " pod="openshift-console/console-6c748d989c-rn88n"
Jul 01 19:25:24.204410 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:24.204206    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d1522f35-27a3-4a30-9550-2e10b2f98893-service-ca\") pod \"console-6c748d989c-rn88n\" (UID: \"d1522f35-27a3-4a30-9550-2e10b2f98893\") " pod="openshift-console/console-6c748d989c-rn88n"
Jul 01 19:25:24.204518 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:24.204493    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1522f35-27a3-4a30-9550-2e10b2f98893-trusted-ca-bundle\") pod \"console-6c748d989c-rn88n\" (UID: \"d1522f35-27a3-4a30-9550-2e10b2f98893\") " pod="openshift-console/console-6c748d989c-rn88n"
Jul 01 19:25:24.206457 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:24.206431    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d1522f35-27a3-4a30-9550-2e10b2f98893-console-oauth-config\") pod \"console-6c748d989c-rn88n\" (UID: \"d1522f35-27a3-4a30-9550-2e10b2f98893\") " pod="openshift-console/console-6c748d989c-rn88n"
Jul 01 19:25:24.206642 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:24.206618    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d1522f35-27a3-4a30-9550-2e10b2f98893-console-serving-cert\") pod \"console-6c748d989c-rn88n\" (UID: \"d1522f35-27a3-4a30-9550-2e10b2f98893\") " pod="openshift-console/console-6c748d989c-rn88n"
Jul 01 19:25:24.211386 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:24.211363    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqck8\" (UniqueName: \"kubernetes.io/projected/d1522f35-27a3-4a30-9550-2e10b2f98893-kube-api-access-wqck8\") pod \"console-6c748d989c-rn88n\" (UID: \"d1522f35-27a3-4a30-9550-2e10b2f98893\") " pod="openshift-console/console-6c748d989c-rn88n"
Jul 01 19:25:24.387675 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:24.387648    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c748d989c-rn88n"
Jul 01 19:25:24.510188 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:24.510162    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c748d989c-rn88n"]
Jul 01 19:25:24.512621 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:25:24.512592    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1522f35_27a3_4a30_9550_2e10b2f98893.slice/crio-b2d9ea57f8f2147b4d5804dd3c94fbb1decba86502e309de4215921a27b3dc53 WatchSource:0}: Error finding container b2d9ea57f8f2147b4d5804dd3c94fbb1decba86502e309de4215921a27b3dc53: Status 404 returned error can't find the container with id b2d9ea57f8f2147b4d5804dd3c94fbb1decba86502e309de4215921a27b3dc53
Jul 01 19:25:25.379370 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:25.379327    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c748d989c-rn88n" event={"ID":"d1522f35-27a3-4a30-9550-2e10b2f98893","Type":"ContainerStarted","Data":"9e88c913e664935fc49bd75d9af0a7e1460c39ccd64efd3bfc5ce579987a7e00"}
Jul 01 19:25:25.379744 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:25.379418    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c748d989c-rn88n" event={"ID":"d1522f35-27a3-4a30-9550-2e10b2f98893","Type":"ContainerStarted","Data":"b2d9ea57f8f2147b4d5804dd3c94fbb1decba86502e309de4215921a27b3dc53"}
Jul 01 19:25:25.401164 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:25.401114    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c748d989c-rn88n" podStartSLOduration=1.401097242 podStartE2EDuration="1.401097242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:25:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:25:25.399505846 +0000 UTC m=+231.491505308" watchObservedRunningTime="2026-07-01 19:25:25.401097242 +0000 UTC m=+231.493096690"
Jul 01 19:25:34.388291 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:34.388258    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c748d989c-rn88n"
Jul 01 19:25:34.388671 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:34.388352    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c748d989c-rn88n"
Jul 01 19:25:34.393004 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:34.392982    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c748d989c-rn88n"
Jul 01 19:25:34.419976 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:34.419950    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c748d989c-rn88n"
Jul 01 19:25:34.486960 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:34.486925    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5595bf5c5b-6dntk"]
Jul 01 19:25:49.721812 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:49.721776    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fb98b5bb8-cbd8l"]
Jul 01 19:25:49.724743 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:49.724724    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fb98b5bb8-cbd8l"
Jul 01 19:25:49.728007 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:49.727985    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 01 19:25:49.728468 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:49.728448    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 01 19:25:49.728591 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:49.728475    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 01 19:25:49.728591 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:49.728475    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-dhxwc\""
Jul 01 19:25:49.729016 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:49.729000    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 01 19:25:49.740594 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:49.740574    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fb98b5bb8-cbd8l"]
Jul 01 19:25:49.795912 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:49.795864    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fcfcb7fb5-28927"]
Jul 01 19:25:49.798102 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:49.798084    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fcfcb7fb5-28927"
Jul 01 19:25:49.816481 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:49.816460    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 01 19:25:49.822826 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:49.822807    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frltb\" (UniqueName: \"kubernetes.io/projected/80a036fb-d488-4b5d-b51d-0d8f1e975887-kube-api-access-frltb\") pod \"managed-serviceaccount-addon-agent-6fb98b5bb8-cbd8l\" (UID: \"80a036fb-d488-4b5d-b51d-0d8f1e975887\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fb98b5bb8-cbd8l"
Jul 01 19:25:49.822924 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:49.822869    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/80a036fb-d488-4b5d-b51d-0d8f1e975887-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6fb98b5bb8-cbd8l\" (UID: \"80a036fb-d488-4b5d-b51d-0d8f1e975887\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fb98b5bb8-cbd8l"
Jul 01 19:25:49.839470 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:49.839445    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fcfcb7fb5-28927"]
Jul 01 19:25:49.923664 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:49.923628    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d1fc7984-5a74-476b-ac3d-7d3519f2f678-klusterlet-config\") pod \"klusterlet-addon-workmgr-7fcfcb7fb5-28927\" (UID: \"d1fc7984-5a74-476b-ac3d-7d3519f2f678\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fcfcb7fb5-28927"
Jul 01 19:25:49.923826 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:49.923695    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8z5m\" (UniqueName: \"kubernetes.io/projected/d1fc7984-5a74-476b-ac3d-7d3519f2f678-kube-api-access-x8z5m\") pod \"klusterlet-addon-workmgr-7fcfcb7fb5-28927\" (UID: \"d1fc7984-5a74-476b-ac3d-7d3519f2f678\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fcfcb7fb5-28927"
Jul 01 19:25:49.923826 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:49.923721    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/80a036fb-d488-4b5d-b51d-0d8f1e975887-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6fb98b5bb8-cbd8l\" (UID: \"80a036fb-d488-4b5d-b51d-0d8f1e975887\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fb98b5bb8-cbd8l"
Jul 01 19:25:49.923826 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:49.923748    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1fc7984-5a74-476b-ac3d-7d3519f2f678-tmp\") pod \"klusterlet-addon-workmgr-7fcfcb7fb5-28927\" (UID: \"d1fc7984-5a74-476b-ac3d-7d3519f2f678\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fcfcb7fb5-28927"
Jul 01 19:25:49.923959 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:49.923822    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frltb\" (UniqueName: \"kubernetes.io/projected/80a036fb-d488-4b5d-b51d-0d8f1e975887-kube-api-access-frltb\") pod \"managed-serviceaccount-addon-agent-6fb98b5bb8-cbd8l\" (UID: \"80a036fb-d488-4b5d-b51d-0d8f1e975887\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fb98b5bb8-cbd8l"
Jul 01 19:25:49.926222 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:49.926201    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/80a036fb-d488-4b5d-b51d-0d8f1e975887-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6fb98b5bb8-cbd8l\" (UID: \"80a036fb-d488-4b5d-b51d-0d8f1e975887\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fb98b5bb8-cbd8l"
Jul 01 19:25:49.936552 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:49.936531    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frltb\" (UniqueName: \"kubernetes.io/projected/80a036fb-d488-4b5d-b51d-0d8f1e975887-kube-api-access-frltb\") pod \"managed-serviceaccount-addon-agent-6fb98b5bb8-cbd8l\" (UID: \"80a036fb-d488-4b5d-b51d-0d8f1e975887\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fb98b5bb8-cbd8l"
Jul 01 19:25:50.025306 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:50.025232    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8z5m\" (UniqueName: \"kubernetes.io/projected/d1fc7984-5a74-476b-ac3d-7d3519f2f678-kube-api-access-x8z5m\") pod \"klusterlet-addon-workmgr-7fcfcb7fb5-28927\" (UID: \"d1fc7984-5a74-476b-ac3d-7d3519f2f678\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fcfcb7fb5-28927"
Jul 01 19:25:50.025306 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:50.025283    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1fc7984-5a74-476b-ac3d-7d3519f2f678-tmp\") pod \"klusterlet-addon-workmgr-7fcfcb7fb5-28927\" (UID: \"d1fc7984-5a74-476b-ac3d-7d3519f2f678\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fcfcb7fb5-28927"
Jul 01 19:25:50.025525 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:50.025367    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d1fc7984-5a74-476b-ac3d-7d3519f2f678-klusterlet-config\") pod \"klusterlet-addon-workmgr-7fcfcb7fb5-28927\" (UID: \"d1fc7984-5a74-476b-ac3d-7d3519f2f678\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fcfcb7fb5-28927"
Jul 01 19:25:50.025732 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:50.025711    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1fc7984-5a74-476b-ac3d-7d3519f2f678-tmp\") pod \"klusterlet-addon-workmgr-7fcfcb7fb5-28927\" (UID: \"d1fc7984-5a74-476b-ac3d-7d3519f2f678\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fcfcb7fb5-28927"
Jul 01 19:25:50.027758 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:50.027740    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d1fc7984-5a74-476b-ac3d-7d3519f2f678-klusterlet-config\") pod \"klusterlet-addon-workmgr-7fcfcb7fb5-28927\" (UID: \"d1fc7984-5a74-476b-ac3d-7d3519f2f678\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fcfcb7fb5-28927"
Jul 01 19:25:50.046592 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:50.046566    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fb98b5bb8-cbd8l"
Jul 01 19:25:50.055619 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:50.055590    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8z5m\" (UniqueName: \"kubernetes.io/projected/d1fc7984-5a74-476b-ac3d-7d3519f2f678-kube-api-access-x8z5m\") pod \"klusterlet-addon-workmgr-7fcfcb7fb5-28927\" (UID: \"d1fc7984-5a74-476b-ac3d-7d3519f2f678\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fcfcb7fb5-28927"
Jul 01 19:25:50.106641 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:50.106609    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fcfcb7fb5-28927"
Jul 01 19:25:50.185052 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:50.185025    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fb98b5bb8-cbd8l"]
Jul 01 19:25:50.187450 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:25:50.187424    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80a036fb_d488_4b5d_b51d_0d8f1e975887.slice/crio-7e1f8ff31a5a62cc35601c561f3fe89e5fe20895d072e285995ccdbef680967d WatchSource:0}: Error finding container 7e1f8ff31a5a62cc35601c561f3fe89e5fe20895d072e285995ccdbef680967d: Status 404 returned error can't find the container with id 7e1f8ff31a5a62cc35601c561f3fe89e5fe20895d072e285995ccdbef680967d
Jul 01 19:25:50.246370 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:50.246301    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fcfcb7fb5-28927"]
Jul 01 19:25:50.248751 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:25:50.248726    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1fc7984_5a74_476b_ac3d_7d3519f2f678.slice/crio-a69e7d5bfa9a7ea861835290c2f88b4974fab86a2bc2647ee3fce775d9d080e4 WatchSource:0}: Error finding container a69e7d5bfa9a7ea861835290c2f88b4974fab86a2bc2647ee3fce775d9d080e4: Status 404 returned error can't find the container with id a69e7d5bfa9a7ea861835290c2f88b4974fab86a2bc2647ee3fce775d9d080e4
Jul 01 19:25:50.480533 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:50.480490    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fb98b5bb8-cbd8l" event={"ID":"80a036fb-d488-4b5d-b51d-0d8f1e975887","Type":"ContainerStarted","Data":"7e1f8ff31a5a62cc35601c561f3fe89e5fe20895d072e285995ccdbef680967d"}
Jul 01 19:25:50.481460 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:50.481442    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fcfcb7fb5-28927" event={"ID":"d1fc7984-5a74-476b-ac3d-7d3519f2f678","Type":"ContainerStarted","Data":"a69e7d5bfa9a7ea861835290c2f88b4974fab86a2bc2647ee3fce775d9d080e4"}
Jul 01 19:25:53.503220 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:53.503184    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fb98b5bb8-cbd8l" event={"ID":"80a036fb-d488-4b5d-b51d-0d8f1e975887","Type":"ContainerStarted","Data":"5cab578649472a5975e562e8d93535511459604c4402d215269a926afec60170"}
Jul 01 19:25:53.521476 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:53.521427    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fb98b5bb8-cbd8l" podStartSLOduration=1.693787208 podStartE2EDuration="4.521411535s" totalImagesPullingTime="2.827624327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:25:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:25:53.51854044 +0000 UTC m=+259.610539887" watchObservedRunningTime="2026-07-01 19:25:53.521411535 +0000 UTC m=+259.613411013"
Jul 01 19:25:55.512325 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:55.512283    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fcfcb7fb5-28927" event={"ID":"d1fc7984-5a74-476b-ac3d-7d3519f2f678","Type":"ContainerStarted","Data":"1d406cd80314169cd6384786893ee31765d140ece0ca3f59097ed295e5cf9b16"}
Jul 01 19:25:55.512838 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:55.512456    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fcfcb7fb5-28927"
Jul 01 19:25:55.514183 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:55.514161    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fcfcb7fb5-28927"
Jul 01 19:25:55.544219 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:55.544168    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fcfcb7fb5-28927" podStartSLOduration=2.240183449 podStartE2EDuration="6.544153083s" totalImagesPullingTime="4.303969634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:25:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:25:55.543765353 +0000 UTC m=+261.635764802" watchObservedRunningTime="2026-07-01 19:25:55.544153083 +0000 UTC m=+261.636152530"
Jul 01 19:25:57.485568 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:57.485531    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589pxnhv"]
Jul 01 19:25:57.488950 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:57.488929    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589pxnhv"
Jul 01 19:25:57.493568 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:57.493547    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 01 19:25:57.493568 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:57.493561    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-c77b6\""
Jul 01 19:25:57.494097 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:57.494078    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 01 19:25:57.508425 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:57.508400    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589pxnhv"]
Jul 01 19:25:57.594763 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:57.594734    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mjd6\" (UniqueName: \"kubernetes.io/projected/6734c818-0dd9-4d4e-9fbe-72638a8ff7e7-kube-api-access-9mjd6\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589pxnhv\" (UID: \"6734c818-0dd9-4d4e-9fbe-72638a8ff7e7\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589pxnhv"
Jul 01 19:25:57.594966 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:57.594792    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6734c818-0dd9-4d4e-9fbe-72638a8ff7e7-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589pxnhv\" (UID: \"6734c818-0dd9-4d4e-9fbe-72638a8ff7e7\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589pxnhv"
Jul 01 19:25:57.594966 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:57.594856    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6734c818-0dd9-4d4e-9fbe-72638a8ff7e7-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589pxnhv\" (UID: \"6734c818-0dd9-4d4e-9fbe-72638a8ff7e7\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589pxnhv"
Jul 01 19:25:57.695804 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:57.695770    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6734c818-0dd9-4d4e-9fbe-72638a8ff7e7-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589pxnhv\" (UID: \"6734c818-0dd9-4d4e-9fbe-72638a8ff7e7\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589pxnhv"
Jul 01 19:25:57.696003 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:57.695867    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6734c818-0dd9-4d4e-9fbe-72638a8ff7e7-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589pxnhv\" (UID: \"6734c818-0dd9-4d4e-9fbe-72638a8ff7e7\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589pxnhv"
Jul 01 19:25:57.696003 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:57.695955    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mjd6\" (UniqueName: \"kubernetes.io/projected/6734c818-0dd9-4d4e-9fbe-72638a8ff7e7-kube-api-access-9mjd6\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589pxnhv\" (UID: \"6734c818-0dd9-4d4e-9fbe-72638a8ff7e7\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589pxnhv"
Jul 01 19:25:57.696260 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:57.696240    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6734c818-0dd9-4d4e-9fbe-72638a8ff7e7-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589pxnhv\" (UID: \"6734c818-0dd9-4d4e-9fbe-72638a8ff7e7\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589pxnhv"
Jul 01 19:25:57.696298 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:57.696260    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6734c818-0dd9-4d4e-9fbe-72638a8ff7e7-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589pxnhv\" (UID: \"6734c818-0dd9-4d4e-9fbe-72638a8ff7e7\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589pxnhv"
Jul 01 19:25:57.704998 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:57.704974    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mjd6\" (UniqueName: \"kubernetes.io/projected/6734c818-0dd9-4d4e-9fbe-72638a8ff7e7-kube-api-access-9mjd6\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589pxnhv\" (UID: \"6734c818-0dd9-4d4e-9fbe-72638a8ff7e7\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589pxnhv"
Jul 01 19:25:57.798158 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:57.798073    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589pxnhv"
Jul 01 19:25:57.939460 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:57.939343    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589pxnhv"]
Jul 01 19:25:57.942119 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:25:57.942089    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6734c818_0dd9_4d4e_9fbe_72638a8ff7e7.slice/crio-7e738d8b90c804b3afdd8dc9fee41affd8a3fb4a763fa8e32c79052367be1f51 WatchSource:0}: Error finding container 7e738d8b90c804b3afdd8dc9fee41affd8a3fb4a763fa8e32c79052367be1f51: Status 404 returned error can't find the container with id 7e738d8b90c804b3afdd8dc9fee41affd8a3fb4a763fa8e32c79052367be1f51
Jul 01 19:25:58.528876 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:58.528829    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589pxnhv" event={"ID":"6734c818-0dd9-4d4e-9fbe-72638a8ff7e7","Type":"ContainerStarted","Data":"7e738d8b90c804b3afdd8dc9fee41affd8a3fb4a763fa8e32c79052367be1f51"}
Jul 01 19:25:59.506716 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:59.506668    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-5595bf5c5b-6dntk" podUID="30212148-a8b1-496b-9ba3-e0504ba8a3f3" containerName="console" containerID="cri-o://48e0a54714ff496143a9084e51ec8d3df929cb20b0b08b26a4613f7f17768221" gracePeriod=15
Jul 01 19:25:59.751149 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:59.751124    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5595bf5c5b-6dntk"
Jul 01 19:25:59.922946 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:59.922875    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30212148-a8b1-496b-9ba3-e0504ba8a3f3-console-serving-cert\") pod \"30212148-a8b1-496b-9ba3-e0504ba8a3f3\" (UID: \"30212148-a8b1-496b-9ba3-e0504ba8a3f3\") "
Jul 01 19:25:59.923157 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:59.922964    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tq4qk\" (UniqueName: \"kubernetes.io/projected/30212148-a8b1-496b-9ba3-e0504ba8a3f3-kube-api-access-tq4qk\") pod \"30212148-a8b1-496b-9ba3-e0504ba8a3f3\" (UID: \"30212148-a8b1-496b-9ba3-e0504ba8a3f3\") "
Jul 01 19:25:59.923157 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:59.923069    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30212148-a8b1-496b-9ba3-e0504ba8a3f3-console-oauth-config\") pod \"30212148-a8b1-496b-9ba3-e0504ba8a3f3\" (UID: \"30212148-a8b1-496b-9ba3-e0504ba8a3f3\") "
Jul 01 19:25:59.923355 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:59.923177    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30212148-a8b1-496b-9ba3-e0504ba8a3f3-trusted-ca-bundle\") pod \"30212148-a8b1-496b-9ba3-e0504ba8a3f3\" (UID: \"30212148-a8b1-496b-9ba3-e0504ba8a3f3\") "
Jul 01 19:25:59.923355 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:59.923244    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30212148-a8b1-496b-9ba3-e0504ba8a3f3-oauth-serving-cert\") pod \"30212148-a8b1-496b-9ba3-e0504ba8a3f3\" (UID: \"30212148-a8b1-496b-9ba3-e0504ba8a3f3\") "
Jul 01 19:25:59.923355 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:59.923275    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30212148-a8b1-496b-9ba3-e0504ba8a3f3-service-ca\") pod \"30212148-a8b1-496b-9ba3-e0504ba8a3f3\" (UID: \"30212148-a8b1-496b-9ba3-e0504ba8a3f3\") "
Jul 01 19:25:59.923355 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:59.923306    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30212148-a8b1-496b-9ba3-e0504ba8a3f3-console-config\") pod \"30212148-a8b1-496b-9ba3-e0504ba8a3f3\" (UID: \"30212148-a8b1-496b-9ba3-e0504ba8a3f3\") "
Jul 01 19:25:59.924096 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:59.923773    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30212148-a8b1-496b-9ba3-e0504ba8a3f3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "30212148-a8b1-496b-9ba3-e0504ba8a3f3" (UID: "30212148-a8b1-496b-9ba3-e0504ba8a3f3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:25:59.924096 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:59.923838    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30212148-a8b1-496b-9ba3-e0504ba8a3f3-console-config" (OuterVolumeSpecName: "console-config") pod "30212148-a8b1-496b-9ba3-e0504ba8a3f3" (UID: "30212148-a8b1-496b-9ba3-e0504ba8a3f3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:25:59.924296 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:59.924114    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30212148-a8b1-496b-9ba3-e0504ba8a3f3-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "30212148-a8b1-496b-9ba3-e0504ba8a3f3" (UID: "30212148-a8b1-496b-9ba3-e0504ba8a3f3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:25:59.924296 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:59.924164    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30212148-a8b1-496b-9ba3-e0504ba8a3f3-service-ca" (OuterVolumeSpecName: "service-ca") pod "30212148-a8b1-496b-9ba3-e0504ba8a3f3" (UID: "30212148-a8b1-496b-9ba3-e0504ba8a3f3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:25:59.925435 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:59.925374    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30212148-a8b1-496b-9ba3-e0504ba8a3f3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "30212148-a8b1-496b-9ba3-e0504ba8a3f3" (UID: "30212148-a8b1-496b-9ba3-e0504ba8a3f3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:25:59.925435 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:59.925425    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30212148-a8b1-496b-9ba3-e0504ba8a3f3-kube-api-access-tq4qk" (OuterVolumeSpecName: "kube-api-access-tq4qk") pod "30212148-a8b1-496b-9ba3-e0504ba8a3f3" (UID: "30212148-a8b1-496b-9ba3-e0504ba8a3f3"). InnerVolumeSpecName "kube-api-access-tq4qk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:25:59.925632 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:25:59.925608    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30212148-a8b1-496b-9ba3-e0504ba8a3f3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "30212148-a8b1-496b-9ba3-e0504ba8a3f3" (UID: "30212148-a8b1-496b-9ba3-e0504ba8a3f3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:26:00.024784 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:00.024746    2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30212148-a8b1-496b-9ba3-e0504ba8a3f3-oauth-serving-cert\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:26:00.024784 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:00.024774    2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30212148-a8b1-496b-9ba3-e0504ba8a3f3-service-ca\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:26:00.024784 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:00.024786    2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30212148-a8b1-496b-9ba3-e0504ba8a3f3-console-config\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:26:00.024784 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:00.024794    2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30212148-a8b1-496b-9ba3-e0504ba8a3f3-console-serving-cert\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:26:00.025071 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:00.024804    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq4qk\" (UniqueName: \"kubernetes.io/projected/30212148-a8b1-496b-9ba3-e0504ba8a3f3-kube-api-access-tq4qk\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:26:00.025071 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:00.024813    2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30212148-a8b1-496b-9ba3-e0504ba8a3f3-console-oauth-config\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:26:00.025071 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:00.024821    2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30212148-a8b1-496b-9ba3-e0504ba8a3f3-trusted-ca-bundle\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:26:00.538747 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:00.538711    2574 generic.go:356] "Generic (PLEG): container finished" podID="30212148-a8b1-496b-9ba3-e0504ba8a3f3" containerID="48e0a54714ff496143a9084e51ec8d3df929cb20b0b08b26a4613f7f17768221" exitCode=0
Jul 01 19:26:00.538747 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:00.538752    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5595bf5c5b-6dntk"
Jul 01 19:26:00.538961 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:00.538789    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5595bf5c5b-6dntk" event={"ID":"30212148-a8b1-496b-9ba3-e0504ba8a3f3","Type":"ContainerDied","Data":"48e0a54714ff496143a9084e51ec8d3df929cb20b0b08b26a4613f7f17768221"}
Jul 01 19:26:00.538961 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:00.538842    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5595bf5c5b-6dntk" event={"ID":"30212148-a8b1-496b-9ba3-e0504ba8a3f3","Type":"ContainerDied","Data":"76ffe216b4993166a93a6dcb0410804832208ccbe43b2cf0305805b24d04aed6"}
Jul 01 19:26:00.538961 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:00.538864    2574 scope.go:117] "RemoveContainer" containerID="48e0a54714ff496143a9084e51ec8d3df929cb20b0b08b26a4613f7f17768221"
Jul 01 19:26:00.547834 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:00.547808    2574 scope.go:117] "RemoveContainer" containerID="48e0a54714ff496143a9084e51ec8d3df929cb20b0b08b26a4613f7f17768221"
Jul 01 19:26:00.548189 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:26:00.548153    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48e0a54714ff496143a9084e51ec8d3df929cb20b0b08b26a4613f7f17768221\": container with ID starting with 48e0a54714ff496143a9084e51ec8d3df929cb20b0b08b26a4613f7f17768221 not found: ID does not exist" containerID="48e0a54714ff496143a9084e51ec8d3df929cb20b0b08b26a4613f7f17768221"
Jul 01 19:26:00.548259 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:00.548204    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48e0a54714ff496143a9084e51ec8d3df929cb20b0b08b26a4613f7f17768221"} err="failed to get container status \"48e0a54714ff496143a9084e51ec8d3df929cb20b0b08b26a4613f7f17768221\": rpc error: code = NotFound desc = could not find container \"48e0a54714ff496143a9084e51ec8d3df929cb20b0b08b26a4613f7f17768221\": container with ID starting with 48e0a54714ff496143a9084e51ec8d3df929cb20b0b08b26a4613f7f17768221 not found: ID does not exist"
Jul 01 19:26:00.557475 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:00.557452    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5595bf5c5b-6dntk"]
Jul 01 19:26:00.562438 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:00.562387    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5595bf5c5b-6dntk"]
Jul 01 19:26:02.356348 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:02.356305    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30212148-a8b1-496b-9ba3-e0504ba8a3f3" path="/var/lib/kubelet/pods/30212148-a8b1-496b-9ba3-e0504ba8a3f3/volumes"
Jul 01 19:26:06.568590 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:06.568552    2574 generic.go:356] "Generic (PLEG): container finished" podID="6734c818-0dd9-4d4e-9fbe-72638a8ff7e7" containerID="2281b4469f3a1820e797dcd5a00b62b96d1aacedcb2a6546837f2ee82d6a3d1a" exitCode=0
Jul 01 19:26:06.569016 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:06.568617    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589pxnhv" event={"ID":"6734c818-0dd9-4d4e-9fbe-72638a8ff7e7","Type":"ContainerDied","Data":"2281b4469f3a1820e797dcd5a00b62b96d1aacedcb2a6546837f2ee82d6a3d1a"}
Jul 01 19:26:09.582763 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:09.582684    2574 generic.go:356] "Generic (PLEG): container finished" podID="6734c818-0dd9-4d4e-9fbe-72638a8ff7e7" containerID="cae01ed6fea3f182c306a66c0e369451dcb7434871b3993fc267153d5bd48b62" exitCode=0
Jul 01 19:26:09.583167 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:09.582782    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589pxnhv" event={"ID":"6734c818-0dd9-4d4e-9fbe-72638a8ff7e7","Type":"ContainerDied","Data":"cae01ed6fea3f182c306a66c0e369451dcb7434871b3993fc267153d5bd48b62"}
Jul 01 19:26:18.626188 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:18.626151    2574 generic.go:356] "Generic (PLEG): container finished" podID="6734c818-0dd9-4d4e-9fbe-72638a8ff7e7" containerID="0e9bd1ef5dbcc7416c3cefa525f4a0ffdf058737a863a592a13dd5f6a18a73d7" exitCode=0
Jul 01 19:26:18.626593 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:18.626239    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589pxnhv" event={"ID":"6734c818-0dd9-4d4e-9fbe-72638a8ff7e7","Type":"ContainerDied","Data":"0e9bd1ef5dbcc7416c3cefa525f4a0ffdf058737a863a592a13dd5f6a18a73d7"}
Jul 01 19:26:19.755532 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:19.755509    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589pxnhv"
Jul 01 19:26:19.904386 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:19.904290    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6734c818-0dd9-4d4e-9fbe-72638a8ff7e7-util\") pod \"6734c818-0dd9-4d4e-9fbe-72638a8ff7e7\" (UID: \"6734c818-0dd9-4d4e-9fbe-72638a8ff7e7\") "
Jul 01 19:26:19.904386 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:19.904336    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6734c818-0dd9-4d4e-9fbe-72638a8ff7e7-bundle\") pod \"6734c818-0dd9-4d4e-9fbe-72638a8ff7e7\" (UID: \"6734c818-0dd9-4d4e-9fbe-72638a8ff7e7\") "
Jul 01 19:26:19.904386 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:19.904363    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mjd6\" (UniqueName: \"kubernetes.io/projected/6734c818-0dd9-4d4e-9fbe-72638a8ff7e7-kube-api-access-9mjd6\") pod \"6734c818-0dd9-4d4e-9fbe-72638a8ff7e7\" (UID: \"6734c818-0dd9-4d4e-9fbe-72638a8ff7e7\") "
Jul 01 19:26:19.904987 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:19.904960    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6734c818-0dd9-4d4e-9fbe-72638a8ff7e7-bundle" (OuterVolumeSpecName: "bundle") pod "6734c818-0dd9-4d4e-9fbe-72638a8ff7e7" (UID: "6734c818-0dd9-4d4e-9fbe-72638a8ff7e7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:26:19.906679 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:19.906649    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6734c818-0dd9-4d4e-9fbe-72638a8ff7e7-kube-api-access-9mjd6" (OuterVolumeSpecName: "kube-api-access-9mjd6") pod "6734c818-0dd9-4d4e-9fbe-72638a8ff7e7" (UID: "6734c818-0dd9-4d4e-9fbe-72638a8ff7e7"). InnerVolumeSpecName "kube-api-access-9mjd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:26:19.908559 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:19.908537    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6734c818-0dd9-4d4e-9fbe-72638a8ff7e7-util" (OuterVolumeSpecName: "util") pod "6734c818-0dd9-4d4e-9fbe-72638a8ff7e7" (UID: "6734c818-0dd9-4d4e-9fbe-72638a8ff7e7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:26:20.005262 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:20.005209    2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6734c818-0dd9-4d4e-9fbe-72638a8ff7e7-util\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:26:20.005262 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:20.005255    2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6734c818-0dd9-4d4e-9fbe-72638a8ff7e7-bundle\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:26:20.005262 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:20.005264    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mjd6\" (UniqueName: \"kubernetes.io/projected/6734c818-0dd9-4d4e-9fbe-72638a8ff7e7-kube-api-access-9mjd6\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:26:20.635453 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:20.635372    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589pxnhv"
Jul 01 19:26:20.635453 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:20.635391    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589pxnhv" event={"ID":"6734c818-0dd9-4d4e-9fbe-72638a8ff7e7","Type":"ContainerDied","Data":"7e738d8b90c804b3afdd8dc9fee41affd8a3fb4a763fa8e32c79052367be1f51"}
Jul 01 19:26:20.635453 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:20.635438    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e738d8b90c804b3afdd8dc9fee41affd8a3fb4a763fa8e32c79052367be1f51"
Jul 01 19:26:25.810863 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:25.810828    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-g7p6r"]
Jul 01 19:26:25.811327 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:25.811199    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6734c818-0dd9-4d4e-9fbe-72638a8ff7e7" containerName="extract"
Jul 01 19:26:25.811327 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:25.811214    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6734c818-0dd9-4d4e-9fbe-72638a8ff7e7" containerName="extract"
Jul 01 19:26:25.811327 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:25.811226    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6734c818-0dd9-4d4e-9fbe-72638a8ff7e7" containerName="pull"
Jul 01 19:26:25.811327 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:25.811232    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6734c818-0dd9-4d4e-9fbe-72638a8ff7e7" containerName="pull"
Jul 01 19:26:25.811327 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:25.811242    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6734c818-0dd9-4d4e-9fbe-72638a8ff7e7" containerName="util"
Jul 01 19:26:25.811327 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:25.811248    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6734c818-0dd9-4d4e-9fbe-72638a8ff7e7" containerName="util"
Jul 01 19:26:25.811327 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:25.811266    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30212148-a8b1-496b-9ba3-e0504ba8a3f3" containerName="console"
Jul 01 19:26:25.811327 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:25.811272    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="30212148-a8b1-496b-9ba3-e0504ba8a3f3" containerName="console"
Jul 01 19:26:25.811327 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:25.811322    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6734c818-0dd9-4d4e-9fbe-72638a8ff7e7" containerName="extract"
Jul 01 19:26:25.811327 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:25.811331    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="30212148-a8b1-496b-9ba3-e0504ba8a3f3" containerName="console"
Jul 01 19:26:25.813011 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:25.812995    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-g7p6r"
Jul 01 19:26:25.816870 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:25.816847    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 01 19:26:25.817419 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:25.817389    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 01 19:26:25.817535 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:25.817521    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 01 19:26:25.817632 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:25.817615    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-zwkwk\""
Jul 01 19:26:25.844160 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:25.844130    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-g7p6r"]
Jul 01 19:26:25.957513 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:25.957480    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvbbv\" (UniqueName: \"kubernetes.io/projected/87d28096-3d4e-47c0-9c02-5e9337b3b766-kube-api-access-jvbbv\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-g7p6r\" (UID: \"87d28096-3d4e-47c0-9c02-5e9337b3b766\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-g7p6r"
Jul 01 19:26:25.957700 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:25.957544    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/87d28096-3d4e-47c0-9c02-5e9337b3b766-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-g7p6r\" (UID: \"87d28096-3d4e-47c0-9c02-5e9337b3b766\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-g7p6r"
Jul 01 19:26:26.058760 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:26.058720    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/87d28096-3d4e-47c0-9c02-5e9337b3b766-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-g7p6r\" (UID: \"87d28096-3d4e-47c0-9c02-5e9337b3b766\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-g7p6r"
Jul 01 19:26:26.058981 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:26.058812    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvbbv\" (UniqueName: \"kubernetes.io/projected/87d28096-3d4e-47c0-9c02-5e9337b3b766-kube-api-access-jvbbv\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-g7p6r\" (UID: \"87d28096-3d4e-47c0-9c02-5e9337b3b766\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-g7p6r"
Jul 01 19:26:26.061134 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:26.061076    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/87d28096-3d4e-47c0-9c02-5e9337b3b766-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-g7p6r\" (UID: \"87d28096-3d4e-47c0-9c02-5e9337b3b766\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-g7p6r"
Jul 01 19:26:26.067731 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:26.067707    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvbbv\" (UniqueName: \"kubernetes.io/projected/87d28096-3d4e-47c0-9c02-5e9337b3b766-kube-api-access-jvbbv\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-g7p6r\" (UID: \"87d28096-3d4e-47c0-9c02-5e9337b3b766\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-g7p6r"
Jul 01 19:26:26.123712 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:26.123678    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-g7p6r"
Jul 01 19:26:26.265216 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:26.262721    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-g7p6r"]
Jul 01 19:26:26.269499 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:26:26.269471    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87d28096_3d4e_47c0_9c02_5e9337b3b766.slice/crio-fc346319b5a92a07567109e364d5268b7d053984d0848420c1fd3c653d55ab98 WatchSource:0}: Error finding container fc346319b5a92a07567109e364d5268b7d053984d0848420c1fd3c653d55ab98: Status 404 returned error can't find the container with id fc346319b5a92a07567109e364d5268b7d053984d0848420c1fd3c653d55ab98
Jul 01 19:26:26.662588 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:26.662554    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-g7p6r" event={"ID":"87d28096-3d4e-47c0-9c02-5e9337b3b766","Type":"ContainerStarted","Data":"fc346319b5a92a07567109e364d5268b7d053984d0848420c1fd3c653d55ab98"}
Jul 01 19:26:30.683834 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:30.683793    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-g7p6r" event={"ID":"87d28096-3d4e-47c0-9c02-5e9337b3b766","Type":"ContainerStarted","Data":"5bcfaeaa517db721235c0577a874ca6702e5e7f8e05810f667d3ccdb1275068d"}
Jul 01 19:26:30.684326 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:30.683942    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-g7p6r"
Jul 01 19:26:30.777181 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:30.777123    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-g7p6r" podStartSLOduration=2.306107878 podStartE2EDuration="5.777103321s" totalImagesPullingTime="3.470995443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:26:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:26:30.775578199 +0000 UTC m=+296.867577648" watchObservedRunningTime="2026-07-01 19:26:30.777103321 +0000 UTC m=+296.869102770"
Jul 01 19:26:31.446883 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:31.446852    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-m6gzc"]
Jul 01 19:26:31.448996 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:31.448980    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-m6gzc"
Jul 01 19:26:31.451635 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:31.451614    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-tk9dv\""
Jul 01 19:26:31.451635 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:31.451619    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 01 19:26:31.452036 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:31.452012    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 01 19:26:31.465750 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:31.465725    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-m6gzc"]
Jul 01 19:26:31.610144 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:31.610112    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvg8l\" (UniqueName: \"kubernetes.io/projected/bbc3098a-07c8-4d8e-8a5b-e196f7c8f897-kube-api-access-cvg8l\") pod \"keda-admission-5fbd5c4644-m6gzc\" (UID: \"bbc3098a-07c8-4d8e-8a5b-e196f7c8f897\") " pod="openshift-keda/keda-admission-5fbd5c4644-m6gzc"
Jul 01 19:26:31.610312 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:31.610268    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/bbc3098a-07c8-4d8e-8a5b-e196f7c8f897-certificates\") pod \"keda-admission-5fbd5c4644-m6gzc\" (UID: \"bbc3098a-07c8-4d8e-8a5b-e196f7c8f897\") " pod="openshift-keda/keda-admission-5fbd5c4644-m6gzc"
Jul 01 19:26:31.711105 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:31.711021    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvg8l\" (UniqueName: \"kubernetes.io/projected/bbc3098a-07c8-4d8e-8a5b-e196f7c8f897-kube-api-access-cvg8l\") pod \"keda-admission-5fbd5c4644-m6gzc\" (UID: \"bbc3098a-07c8-4d8e-8a5b-e196f7c8f897\") " pod="openshift-keda/keda-admission-5fbd5c4644-m6gzc"
Jul 01 19:26:31.711105 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:31.711082    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/bbc3098a-07c8-4d8e-8a5b-e196f7c8f897-certificates\") pod \"keda-admission-5fbd5c4644-m6gzc\" (UID: \"bbc3098a-07c8-4d8e-8a5b-e196f7c8f897\") " pod="openshift-keda/keda-admission-5fbd5c4644-m6gzc"
Jul 01 19:26:31.711506 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:26:31.711172    2574 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Jul 01 19:26:31.711506 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:26:31.711191    2574 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-5fbd5c4644-m6gzc: secret "keda-admission-webhooks-certs" not found
Jul 01 19:26:31.711506 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:26:31.711241    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bbc3098a-07c8-4d8e-8a5b-e196f7c8f897-certificates podName:bbc3098a-07c8-4d8e-8a5b-e196f7c8f897 nodeName:}" failed. No retries permitted until 2026-07-01 19:26:32.211225793 +0000 UTC m=+298.303225220 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/bbc3098a-07c8-4d8e-8a5b-e196f7c8f897-certificates") pod "keda-admission-5fbd5c4644-m6gzc" (UID: "bbc3098a-07c8-4d8e-8a5b-e196f7c8f897") : secret "keda-admission-webhooks-certs" not found
Jul 01 19:26:31.727251 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:31.727220    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvg8l\" (UniqueName: \"kubernetes.io/projected/bbc3098a-07c8-4d8e-8a5b-e196f7c8f897-kube-api-access-cvg8l\") pod \"keda-admission-5fbd5c4644-m6gzc\" (UID: \"bbc3098a-07c8-4d8e-8a5b-e196f7c8f897\") " pod="openshift-keda/keda-admission-5fbd5c4644-m6gzc"
Jul 01 19:26:32.216769 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:32.216726    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/bbc3098a-07c8-4d8e-8a5b-e196f7c8f897-certificates\") pod \"keda-admission-5fbd5c4644-m6gzc\" (UID: \"bbc3098a-07c8-4d8e-8a5b-e196f7c8f897\") " pod="openshift-keda/keda-admission-5fbd5c4644-m6gzc"
Jul 01 19:26:32.219205 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:32.219174    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/bbc3098a-07c8-4d8e-8a5b-e196f7c8f897-certificates\") pod \"keda-admission-5fbd5c4644-m6gzc\" (UID: \"bbc3098a-07c8-4d8e-8a5b-e196f7c8f897\") " pod="openshift-keda/keda-admission-5fbd5c4644-m6gzc"
Jul 01 19:26:32.359599 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:32.359559    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-m6gzc"
Jul 01 19:26:32.488938 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:32.488894    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-m6gzc"]
Jul 01 19:26:32.491606 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:26:32.491582    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbc3098a_07c8_4d8e_8a5b_e196f7c8f897.slice/crio-5d8c08c660bf2b9771a29baddb468cc43ace4e0f43352efdb63a9fd3ea84d275 WatchSource:0}: Error finding container 5d8c08c660bf2b9771a29baddb468cc43ace4e0f43352efdb63a9fd3ea84d275: Status 404 returned error can't find the container with id 5d8c08c660bf2b9771a29baddb468cc43ace4e0f43352efdb63a9fd3ea84d275
Jul 01 19:26:32.694867 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:32.694832    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-m6gzc" event={"ID":"bbc3098a-07c8-4d8e-8a5b-e196f7c8f897","Type":"ContainerStarted","Data":"5d8c08c660bf2b9771a29baddb468cc43ace4e0f43352efdb63a9fd3ea84d275"}
Jul 01 19:26:34.311640 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:34.311606    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bplsm_bfb845e6-341d-492e-91c9-9c698a0b8d29/ovn-acl-logging/0.log"
Jul 01 19:26:34.313753 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:34.313730    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bplsm_bfb845e6-341d-492e-91c9-9c698a0b8d29/ovn-acl-logging/0.log"
Jul 01 19:26:34.317091 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:34.317059    2574 kubelet.go:1648] "Image garbage collection succeeded"
Jul 01 19:26:35.710390 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:35.710352    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-m6gzc" event={"ID":"bbc3098a-07c8-4d8e-8a5b-e196f7c8f897","Type":"ContainerStarted","Data":"6816bf45e40aa177259c3292634e4bd217d60e286b843d1b8c9d7bb1b6dacca7"}
Jul 01 19:26:35.710796 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:35.710506    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-5fbd5c4644-m6gzc"
Jul 01 19:26:35.746173 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:35.746127    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-5fbd5c4644-m6gzc" podStartSLOduration=2.076466177 podStartE2EDuration="4.746112487s" totalImagesPullingTime="2.66964631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:26:35.743973221 +0000 UTC m=+301.835972687" watchObservedRunningTime="2026-07-01 19:26:35.746112487 +0000 UTC m=+301.838111935"
Jul 01 19:26:51.690559 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:51.690526    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-g7p6r"
Jul 01 19:26:56.717039 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:26:56.717003    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-5fbd5c4644-m6gzc"
Jul 01 19:27:40.684796 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:27:40.684761    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-j4m6t"]
Jul 01 19:27:40.688175 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:27:40.688159    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-67d95fdc48-j4m6t"
Jul 01 19:27:40.690658 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:27:40.690638    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 01 19:27:40.696978 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:27:40.696948    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 01 19:27:40.697089 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:27:40.696997    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 01 19:27:40.697661 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:27:40.697641    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-7pz7b\""
Jul 01 19:27:40.728804 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:27:40.728780    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-j4m6t"]
Jul 01 19:27:40.787076 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:27:40.787047    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cf6j\" (UniqueName: \"kubernetes.io/projected/68177740-cc72-4d09-9c42-9d9146ba216d-kube-api-access-7cf6j\") pod \"llmisvc-controller-manager-67d95fdc48-j4m6t\" (UID: \"68177740-cc72-4d09-9c42-9d9146ba216d\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-j4m6t"
Jul 01 19:27:40.787227 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:27:40.787086    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68177740-cc72-4d09-9c42-9d9146ba216d-cert\") pod \"llmisvc-controller-manager-67d95fdc48-j4m6t\" (UID: \"68177740-cc72-4d09-9c42-9d9146ba216d\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-j4m6t"
Jul 01 19:27:40.888212 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:27:40.888183    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cf6j\" (UniqueName: \"kubernetes.io/projected/68177740-cc72-4d09-9c42-9d9146ba216d-kube-api-access-7cf6j\") pod \"llmisvc-controller-manager-67d95fdc48-j4m6t\" (UID: \"68177740-cc72-4d09-9c42-9d9146ba216d\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-j4m6t"
Jul 01 19:27:40.888376 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:27:40.888225    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68177740-cc72-4d09-9c42-9d9146ba216d-cert\") pod \"llmisvc-controller-manager-67d95fdc48-j4m6t\" (UID: \"68177740-cc72-4d09-9c42-9d9146ba216d\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-j4m6t"
Jul 01 19:27:40.888376 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:27:40.888336    2574 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 01 19:27:40.888452 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:27:40.888396    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68177740-cc72-4d09-9c42-9d9146ba216d-cert podName:68177740-cc72-4d09-9c42-9d9146ba216d nodeName:}" failed. No retries permitted until 2026-07-01 19:27:41.388377873 +0000 UTC m=+367.480377298 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/68177740-cc72-4d09-9c42-9d9146ba216d-cert") pod "llmisvc-controller-manager-67d95fdc48-j4m6t" (UID: "68177740-cc72-4d09-9c42-9d9146ba216d") : secret "llmisvc-webhook-server-cert" not found
Jul 01 19:27:40.898765 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:27:40.898739    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cf6j\" (UniqueName: \"kubernetes.io/projected/68177740-cc72-4d09-9c42-9d9146ba216d-kube-api-access-7cf6j\") pod \"llmisvc-controller-manager-67d95fdc48-j4m6t\" (UID: \"68177740-cc72-4d09-9c42-9d9146ba216d\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-j4m6t"
Jul 01 19:27:41.391671 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:27:41.391634    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68177740-cc72-4d09-9c42-9d9146ba216d-cert\") pod \"llmisvc-controller-manager-67d95fdc48-j4m6t\" (UID: \"68177740-cc72-4d09-9c42-9d9146ba216d\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-j4m6t"
Jul 01 19:27:41.394124 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:27:41.394105    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68177740-cc72-4d09-9c42-9d9146ba216d-cert\") pod \"llmisvc-controller-manager-67d95fdc48-j4m6t\" (UID: \"68177740-cc72-4d09-9c42-9d9146ba216d\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-j4m6t"
Jul 01 19:27:41.598521 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:27:41.598489    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-67d95fdc48-j4m6t"
Jul 01 19:27:41.728994 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:27:41.728944    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-j4m6t"]
Jul 01 19:27:41.731539 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:27:41.731504    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod68177740_cc72_4d09_9c42_9d9146ba216d.slice/crio-6fc0adde12a88fa6689971e54975f094ba06da60bbda80a77f61aa671ddafa45 WatchSource:0}: Error finding container 6fc0adde12a88fa6689971e54975f094ba06da60bbda80a77f61aa671ddafa45: Status 404 returned error can't find the container with id 6fc0adde12a88fa6689971e54975f094ba06da60bbda80a77f61aa671ddafa45
Jul 01 19:27:41.732807 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:27:41.732790    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 19:27:41.996635 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:27:41.996566    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-67d95fdc48-j4m6t" event={"ID":"68177740-cc72-4d09-9c42-9d9146ba216d","Type":"ContainerStarted","Data":"6fc0adde12a88fa6689971e54975f094ba06da60bbda80a77f61aa671ddafa45"}
Jul 01 19:27:44.007839 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:27:44.007754    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-67d95fdc48-j4m6t" event={"ID":"68177740-cc72-4d09-9c42-9d9146ba216d","Type":"ContainerStarted","Data":"92728de02c5b283da45b7ae0735f2dccd97b09ec196ac4e1c196bd7d44959837"}
Jul 01 19:27:44.007839 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:27:44.007817    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-67d95fdc48-j4m6t"
Jul 01 19:27:44.026030 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:27:44.025980    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-67d95fdc48-j4m6t" podStartSLOduration=2.147741622 podStartE2EDuration="4.025964146s" totalImagesPullingTime="1.878222524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:27:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:27:44.024994566 +0000 UTC m=+370.116994037" watchObservedRunningTime="2026-07-01 19:27:44.025964146 +0000 UTC m=+370.117963596"
Jul 01 19:28:15.015294 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:28:15.015259    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-67d95fdc48-j4m6t"
Jul 01 19:28:50.317131 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:28:50.317100    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-t2bnp"]
Jul 01 19:28:50.320503 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:28:50.320485    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-t2bnp"
Jul 01 19:28:50.324078 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:28:50.324044    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-jssgn\""
Jul 01 19:28:50.324245 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:28:50.324225    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 01 19:28:50.335848 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:28:50.335826    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-t2bnp"]
Jul 01 19:28:50.366199 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:28:50.366172    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxcq7\" (UniqueName: \"kubernetes.io/projected/f23ff531-b808-4adf-ae47-e3955c020b75-kube-api-access-jxcq7\") pod \"odh-model-controller-84b54b7f97-t2bnp\" (UID: \"f23ff531-b808-4adf-ae47-e3955c020b75\") " pod="kserve/odh-model-controller-84b54b7f97-t2bnp"
Jul 01 19:28:50.366346 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:28:50.366236    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f23ff531-b808-4adf-ae47-e3955c020b75-cert\") pod \"odh-model-controller-84b54b7f97-t2bnp\" (UID: \"f23ff531-b808-4adf-ae47-e3955c020b75\") " pod="kserve/odh-model-controller-84b54b7f97-t2bnp"
Jul 01 19:28:50.467339 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:28:50.467306    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f23ff531-b808-4adf-ae47-e3955c020b75-cert\") pod \"odh-model-controller-84b54b7f97-t2bnp\" (UID: \"f23ff531-b808-4adf-ae47-e3955c020b75\") " pod="kserve/odh-model-controller-84b54b7f97-t2bnp"
Jul 01 19:28:50.467464 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:28:50.467354    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxcq7\" (UniqueName: \"kubernetes.io/projected/f23ff531-b808-4adf-ae47-e3955c020b75-kube-api-access-jxcq7\") pod \"odh-model-controller-84b54b7f97-t2bnp\" (UID: \"f23ff531-b808-4adf-ae47-e3955c020b75\") " pod="kserve/odh-model-controller-84b54b7f97-t2bnp"
Jul 01 19:28:50.467508 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:28:50.467456    2574 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 01 19:28:50.467542 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:28:50.467518    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f23ff531-b808-4adf-ae47-e3955c020b75-cert podName:f23ff531-b808-4adf-ae47-e3955c020b75 nodeName:}" failed. No retries permitted until 2026-07-01 19:28:50.967503379 +0000 UTC m=+437.059502806 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f23ff531-b808-4adf-ae47-e3955c020b75-cert") pod "odh-model-controller-84b54b7f97-t2bnp" (UID: "f23ff531-b808-4adf-ae47-e3955c020b75") : secret "odh-model-controller-webhook-cert" not found
Jul 01 19:28:50.485717 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:28:50.485681    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxcq7\" (UniqueName: \"kubernetes.io/projected/f23ff531-b808-4adf-ae47-e3955c020b75-kube-api-access-jxcq7\") pod \"odh-model-controller-84b54b7f97-t2bnp\" (UID: \"f23ff531-b808-4adf-ae47-e3955c020b75\") " pod="kserve/odh-model-controller-84b54b7f97-t2bnp"
Jul 01 19:28:50.972107 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:28:50.972074    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f23ff531-b808-4adf-ae47-e3955c020b75-cert\") pod \"odh-model-controller-84b54b7f97-t2bnp\" (UID: \"f23ff531-b808-4adf-ae47-e3955c020b75\") " pod="kserve/odh-model-controller-84b54b7f97-t2bnp"
Jul 01 19:28:50.974471 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:28:50.974442    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f23ff531-b808-4adf-ae47-e3955c020b75-cert\") pod \"odh-model-controller-84b54b7f97-t2bnp\" (UID: \"f23ff531-b808-4adf-ae47-e3955c020b75\") " pod="kserve/odh-model-controller-84b54b7f97-t2bnp"
Jul 01 19:28:51.231018 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:28:51.230918    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-t2bnp"
Jul 01 19:28:51.362373 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:28:51.362345    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-t2bnp"]
Jul 01 19:28:51.364882 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:28:51.364854    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf23ff531_b808_4adf_ae47_e3955c020b75.slice/crio-e948ed3d59f740baff9aedb2dba214818f2def2ef1b6de8eece197804a00f9c0 WatchSource:0}: Error finding container e948ed3d59f740baff9aedb2dba214818f2def2ef1b6de8eece197804a00f9c0: Status 404 returned error can't find the container with id e948ed3d59f740baff9aedb2dba214818f2def2ef1b6de8eece197804a00f9c0
Jul 01 19:28:52.312369 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:28:52.312329    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-t2bnp" event={"ID":"f23ff531-b808-4adf-ae47-e3955c020b75","Type":"ContainerStarted","Data":"e948ed3d59f740baff9aedb2dba214818f2def2ef1b6de8eece197804a00f9c0"}
Jul 01 19:28:54.324218 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:28:54.324178    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-t2bnp" event={"ID":"f23ff531-b808-4adf-ae47-e3955c020b75","Type":"ContainerStarted","Data":"114dc97c566c8a9334ab3b27ee39bb429b2e41960ba4aea7802f33531fd3423d"}
Jul 01 19:28:54.324655 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:28:54.324254    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-t2bnp"
Jul 01 19:28:54.344299 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:28:54.344251    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-t2bnp" podStartSLOduration=1.502868516 podStartE2EDuration="4.344236117s" totalImagesPullingTime="2.841367601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:28:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:28:54.341835064 +0000 UTC m=+440.433834512" watchObservedRunningTime="2026-07-01 19:28:54.344236117 +0000 UTC m=+440.436235565"
Jul 01 19:29:05.331166 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:05.331084    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-t2bnp"
Jul 01 19:29:06.199486 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:06.199456    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-h8x54"]
Jul 01 19:29:06.206025 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:06.206003    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-h8x54"
Jul 01 19:29:06.208607 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:06.208577    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 01 19:29:06.208824 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:06.208805    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-tdlwf\""
Jul 01 19:29:06.210507 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:06.210486    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-h8x54"]
Jul 01 19:29:06.314978 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:06.314930    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mljq\" (UniqueName: \"kubernetes.io/projected/afd6ed61-31e8-48a6-b2f3-74e5bf203c19-kube-api-access-9mljq\") pod \"s3-init-h8x54\" (UID: \"afd6ed61-31e8-48a6-b2f3-74e5bf203c19\") " pod="kserve/s3-init-h8x54"
Jul 01 19:29:06.315168 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:06.314998    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afd6ed61-31e8-48a6-b2f3-74e5bf203c19-model-cache\") pod \"s3-init-h8x54\" (UID: \"afd6ed61-31e8-48a6-b2f3-74e5bf203c19\") " pod="kserve/s3-init-h8x54"
Jul 01 19:29:06.416093 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:06.416056    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mljq\" (UniqueName: \"kubernetes.io/projected/afd6ed61-31e8-48a6-b2f3-74e5bf203c19-kube-api-access-9mljq\") pod \"s3-init-h8x54\" (UID: \"afd6ed61-31e8-48a6-b2f3-74e5bf203c19\") " pod="kserve/s3-init-h8x54"
Jul 01 19:29:06.416517 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:06.416116    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afd6ed61-31e8-48a6-b2f3-74e5bf203c19-model-cache\") pod \"s3-init-h8x54\" (UID: \"afd6ed61-31e8-48a6-b2f3-74e5bf203c19\") " pod="kserve/s3-init-h8x54"
Jul 01 19:29:06.416613 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:06.416591    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afd6ed61-31e8-48a6-b2f3-74e5bf203c19-model-cache\") pod \"s3-init-h8x54\" (UID: \"afd6ed61-31e8-48a6-b2f3-74e5bf203c19\") " pod="kserve/s3-init-h8x54"
Jul 01 19:29:06.425070 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:06.425049    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mljq\" (UniqueName: \"kubernetes.io/projected/afd6ed61-31e8-48a6-b2f3-74e5bf203c19-kube-api-access-9mljq\") pod \"s3-init-h8x54\" (UID: \"afd6ed61-31e8-48a6-b2f3-74e5bf203c19\") " pod="kserve/s3-init-h8x54"
Jul 01 19:29:06.515715 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:06.515603    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-h8x54"
Jul 01 19:29:06.647968 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:06.647941    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-h8x54"]
Jul 01 19:29:06.650559 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:29:06.650532    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podafd6ed61_31e8_48a6_b2f3_74e5bf203c19.slice/crio-a4620a4643b2020c015c67f23cd1cd0774f8a909924b68c4a3c8721863a13b0a WatchSource:0}: Error finding container a4620a4643b2020c015c67f23cd1cd0774f8a909924b68c4a3c8721863a13b0a: Status 404 returned error can't find the container with id a4620a4643b2020c015c67f23cd1cd0774f8a909924b68c4a3c8721863a13b0a
Jul 01 19:29:07.385347 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:07.385311    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-h8x54" event={"ID":"afd6ed61-31e8-48a6-b2f3-74e5bf203c19","Type":"ContainerStarted","Data":"a4620a4643b2020c015c67f23cd1cd0774f8a909924b68c4a3c8721863a13b0a"}
Jul 01 19:29:10.103665 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:10.103623    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-85df97f456-m66xl"]
Jul 01 19:29:10.121723 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:10.121685    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85df97f456-m66xl"
Jul 01 19:29:10.123147 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:10.123117    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85df97f456-m66xl"]
Jul 01 19:29:10.152869 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:10.152837    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf2583a0-c190-4f5f-8b48-bfef7a3316fa-trusted-ca-bundle\") pod \"console-85df97f456-m66xl\" (UID: \"bf2583a0-c190-4f5f-8b48-bfef7a3316fa\") " pod="openshift-console/console-85df97f456-m66xl"
Jul 01 19:29:10.153031 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:10.152886    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bf2583a0-c190-4f5f-8b48-bfef7a3316fa-oauth-serving-cert\") pod \"console-85df97f456-m66xl\" (UID: \"bf2583a0-c190-4f5f-8b48-bfef7a3316fa\") " pod="openshift-console/console-85df97f456-m66xl"
Jul 01 19:29:10.153031 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:10.152956    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bf2583a0-c190-4f5f-8b48-bfef7a3316fa-console-oauth-config\") pod \"console-85df97f456-m66xl\" (UID: \"bf2583a0-c190-4f5f-8b48-bfef7a3316fa\") " pod="openshift-console/console-85df97f456-m66xl"
Jul 01 19:29:10.153149 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:10.153057    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bf2583a0-c190-4f5f-8b48-bfef7a3316fa-console-config\") pod \"console-85df97f456-m66xl\" (UID: \"bf2583a0-c190-4f5f-8b48-bfef7a3316fa\") " pod="openshift-console/console-85df97f456-m66xl"
Jul 01 19:29:10.153149 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:10.153101    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t49ls\" (UniqueName: \"kubernetes.io/projected/bf2583a0-c190-4f5f-8b48-bfef7a3316fa-kube-api-access-t49ls\") pod \"console-85df97f456-m66xl\" (UID: \"bf2583a0-c190-4f5f-8b48-bfef7a3316fa\") " pod="openshift-console/console-85df97f456-m66xl"
Jul 01 19:29:10.153149 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:10.153130    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bf2583a0-c190-4f5f-8b48-bfef7a3316fa-service-ca\") pod \"console-85df97f456-m66xl\" (UID: \"bf2583a0-c190-4f5f-8b48-bfef7a3316fa\") " pod="openshift-console/console-85df97f456-m66xl"
Jul 01 19:29:10.153303 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:10.153173    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bf2583a0-c190-4f5f-8b48-bfef7a3316fa-console-serving-cert\") pod \"console-85df97f456-m66xl\" (UID: \"bf2583a0-c190-4f5f-8b48-bfef7a3316fa\") " pod="openshift-console/console-85df97f456-m66xl"
Jul 01 19:29:10.254062 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:10.254015    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf2583a0-c190-4f5f-8b48-bfef7a3316fa-trusted-ca-bundle\") pod \"console-85df97f456-m66xl\" (UID: \"bf2583a0-c190-4f5f-8b48-bfef7a3316fa\") " pod="openshift-console/console-85df97f456-m66xl"
Jul 01 19:29:10.254272 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:10.254065    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bf2583a0-c190-4f5f-8b48-bfef7a3316fa-oauth-serving-cert\") pod \"console-85df97f456-m66xl\" (UID: \"bf2583a0-c190-4f5f-8b48-bfef7a3316fa\") " pod="openshift-console/console-85df97f456-m66xl"
Jul 01 19:29:10.254272 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:10.254171    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bf2583a0-c190-4f5f-8b48-bfef7a3316fa-console-oauth-config\") pod \"console-85df97f456-m66xl\" (UID: \"bf2583a0-c190-4f5f-8b48-bfef7a3316fa\") " pod="openshift-console/console-85df97f456-m66xl"
Jul 01 19:29:10.254272 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:10.254265    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bf2583a0-c190-4f5f-8b48-bfef7a3316fa-console-config\") pod \"console-85df97f456-m66xl\" (UID: \"bf2583a0-c190-4f5f-8b48-bfef7a3316fa\") " pod="openshift-console/console-85df97f456-m66xl"
Jul 01 19:29:10.254444 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:10.254308    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t49ls\" (UniqueName: \"kubernetes.io/projected/bf2583a0-c190-4f5f-8b48-bfef7a3316fa-kube-api-access-t49ls\") pod \"console-85df97f456-m66xl\" (UID: \"bf2583a0-c190-4f5f-8b48-bfef7a3316fa\") " pod="openshift-console/console-85df97f456-m66xl"
Jul 01 19:29:10.254444 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:10.254340    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bf2583a0-c190-4f5f-8b48-bfef7a3316fa-service-ca\") pod \"console-85df97f456-m66xl\" (UID: \"bf2583a0-c190-4f5f-8b48-bfef7a3316fa\") " pod="openshift-console/console-85df97f456-m66xl"
Jul 01 19:29:10.254444 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:10.254378    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bf2583a0-c190-4f5f-8b48-bfef7a3316fa-console-serving-cert\") pod \"console-85df97f456-m66xl\" (UID: \"bf2583a0-c190-4f5f-8b48-bfef7a3316fa\") " pod="openshift-console/console-85df97f456-m66xl"
Jul 01 19:29:10.254895 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:10.254855    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bf2583a0-c190-4f5f-8b48-bfef7a3316fa-oauth-serving-cert\") pod \"console-85df97f456-m66xl\" (UID: \"bf2583a0-c190-4f5f-8b48-bfef7a3316fa\") " pod="openshift-console/console-85df97f456-m66xl"
Jul 01 19:29:10.255042 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:10.255021    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf2583a0-c190-4f5f-8b48-bfef7a3316fa-trusted-ca-bundle\") pod \"console-85df97f456-m66xl\" (UID: \"bf2583a0-c190-4f5f-8b48-bfef7a3316fa\") " pod="openshift-console/console-85df97f456-m66xl"
Jul 01 19:29:10.255130 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:10.255101    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bf2583a0-c190-4f5f-8b48-bfef7a3316fa-service-ca\") pod \"console-85df97f456-m66xl\" (UID: \"bf2583a0-c190-4f5f-8b48-bfef7a3316fa\") " pod="openshift-console/console-85df97f456-m66xl"
Jul 01 19:29:10.255244 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:10.255179    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bf2583a0-c190-4f5f-8b48-bfef7a3316fa-console-config\") pod \"console-85df97f456-m66xl\" (UID: \"bf2583a0-c190-4f5f-8b48-bfef7a3316fa\") " pod="openshift-console/console-85df97f456-m66xl"
Jul 01 19:29:10.257281 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:10.257255    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bf2583a0-c190-4f5f-8b48-bfef7a3316fa-console-serving-cert\") pod \"console-85df97f456-m66xl\" (UID: \"bf2583a0-c190-4f5f-8b48-bfef7a3316fa\") " pod="openshift-console/console-85df97f456-m66xl"
Jul 01 19:29:10.257387 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:10.257283    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bf2583a0-c190-4f5f-8b48-bfef7a3316fa-console-oauth-config\") pod \"console-85df97f456-m66xl\" (UID: \"bf2583a0-c190-4f5f-8b48-bfef7a3316fa\") " pod="openshift-console/console-85df97f456-m66xl"
Jul 01 19:29:10.264419 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:10.264395    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t49ls\" (UniqueName: \"kubernetes.io/projected/bf2583a0-c190-4f5f-8b48-bfef7a3316fa-kube-api-access-t49ls\") pod \"console-85df97f456-m66xl\" (UID: \"bf2583a0-c190-4f5f-8b48-bfef7a3316fa\") " pod="openshift-console/console-85df97f456-m66xl"
Jul 01 19:29:10.433788 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:10.433704    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85df97f456-m66xl"
Jul 01 19:29:10.852814 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:10.852791    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85df97f456-m66xl"]
Jul 01 19:29:10.854908 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:29:10.854866    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf2583a0_c190_4f5f_8b48_bfef7a3316fa.slice/crio-b9f8e96788637023e5d1fa4e12f38c7eaea04f5d4ae8d535f52cdf196b20b728 WatchSource:0}: Error finding container b9f8e96788637023e5d1fa4e12f38c7eaea04f5d4ae8d535f52cdf196b20b728: Status 404 returned error can't find the container with id b9f8e96788637023e5d1fa4e12f38c7eaea04f5d4ae8d535f52cdf196b20b728
Jul 01 19:29:11.407775 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:11.407738    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-h8x54" event={"ID":"afd6ed61-31e8-48a6-b2f3-74e5bf203c19","Type":"ContainerStarted","Data":"4d4d3e4df905007f7bcbe5fb39d28386c80952f43716595d21bc17be9767c267"}
Jul 01 19:29:11.409342 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:11.409318    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85df97f456-m66xl" event={"ID":"bf2583a0-c190-4f5f-8b48-bfef7a3316fa","Type":"ContainerStarted","Data":"50f46c663cf2d03a83b7262e89956db0d5a162cd921f876695e0d018d2264080"}
Jul 01 19:29:11.409484 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:11.409354    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85df97f456-m66xl" event={"ID":"bf2583a0-c190-4f5f-8b48-bfef7a3316fa","Type":"ContainerStarted","Data":"b9f8e96788637023e5d1fa4e12f38c7eaea04f5d4ae8d535f52cdf196b20b728"}
Jul 01 19:29:11.439779 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:11.439732    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-85df97f456-m66xl" podStartSLOduration=1.439712572 podStartE2EDuration="1.439712572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:29:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:29:11.439119809 +0000 UTC m=+457.531119257" watchObservedRunningTime="2026-07-01 19:29:11.439712572 +0000 UTC m=+457.531712021"
Jul 01 19:29:13.420597 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:13.420561    2574 generic.go:356] "Generic (PLEG): container finished" podID="afd6ed61-31e8-48a6-b2f3-74e5bf203c19" containerID="4d4d3e4df905007f7bcbe5fb39d28386c80952f43716595d21bc17be9767c267" exitCode=0
Jul 01 19:29:13.420597 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:13.420600    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-h8x54" event={"ID":"afd6ed61-31e8-48a6-b2f3-74e5bf203c19","Type":"ContainerDied","Data":"4d4d3e4df905007f7bcbe5fb39d28386c80952f43716595d21bc17be9767c267"}
Jul 01 19:29:18.449856 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:18.449822    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-h8x54" event={"ID":"afd6ed61-31e8-48a6-b2f3-74e5bf203c19","Type":"ContainerStarted","Data":"a7acf643f65e9e8aede4e8e2abe060bf5f55717c1b135c2433082a3bc4f2a25c"}
Jul 01 19:29:18.470756 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:18.470702    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-h8x54" podStartSLOduration=1.628899339 podStartE2EDuration="12.470685545s" totalImagesPullingTime="8.720412713s" totalInitContainerRuntime="2.121373493s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:29:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:29:18.469638301 +0000 UTC m=+464.561637750" watchObservedRunningTime="2026-07-01 19:29:18.470685545 +0000 UTC m=+464.562684993"
Jul 01 19:29:20.434856 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:20.434813    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85df97f456-m66xl"
Jul 01 19:29:20.435371 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:20.434886    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-85df97f456-m66xl"
Jul 01 19:29:20.440068 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:20.440041    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-85df97f456-m66xl"
Jul 01 19:29:20.462501 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:20.462469    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-85df97f456-m66xl"
Jul 01 19:29:20.530654 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:20.530609    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c748d989c-rn88n"]
Jul 01 19:29:45.552918 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:45.552864    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6c748d989c-rn88n" podUID="d1522f35-27a3-4a30-9550-2e10b2f98893" containerName="console" containerID="cri-o://9e88c913e664935fc49bd75d9af0a7e1460c39ccd64efd3bfc5ce579987a7e00" gracePeriod=15
Jul 01 19:29:45.796891 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:45.796865    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c748d989c-rn88n"
Jul 01 19:29:45.853754 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:45.853728    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqck8\" (UniqueName: \"kubernetes.io/projected/d1522f35-27a3-4a30-9550-2e10b2f98893-kube-api-access-wqck8\") pod \"d1522f35-27a3-4a30-9550-2e10b2f98893\" (UID: \"d1522f35-27a3-4a30-9550-2e10b2f98893\") "
Jul 01 19:29:45.853931 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:45.853765    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d1522f35-27a3-4a30-9550-2e10b2f98893-console-serving-cert\") pod \"d1522f35-27a3-4a30-9550-2e10b2f98893\" (UID: \"d1522f35-27a3-4a30-9550-2e10b2f98893\") "
Jul 01 19:29:45.853931 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:45.853793    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d1522f35-27a3-4a30-9550-2e10b2f98893-oauth-serving-cert\") pod \"d1522f35-27a3-4a30-9550-2e10b2f98893\" (UID: \"d1522f35-27a3-4a30-9550-2e10b2f98893\") "
Jul 01 19:29:45.853931 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:45.853828    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d1522f35-27a3-4a30-9550-2e10b2f98893-service-ca\") pod \"d1522f35-27a3-4a30-9550-2e10b2f98893\" (UID: \"d1522f35-27a3-4a30-9550-2e10b2f98893\") "
Jul 01 19:29:45.853931 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:45.853861    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d1522f35-27a3-4a30-9550-2e10b2f98893-console-oauth-config\") pod \"d1522f35-27a3-4a30-9550-2e10b2f98893\" (UID: \"d1522f35-27a3-4a30-9550-2e10b2f98893\") "
Jul 01 19:29:45.853931 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:45.853926    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d1522f35-27a3-4a30-9550-2e10b2f98893-console-config\") pod \"d1522f35-27a3-4a30-9550-2e10b2f98893\" (UID: \"d1522f35-27a3-4a30-9550-2e10b2f98893\") "
Jul 01 19:29:45.854200 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:45.853969    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1522f35-27a3-4a30-9550-2e10b2f98893-trusted-ca-bundle\") pod \"d1522f35-27a3-4a30-9550-2e10b2f98893\" (UID: \"d1522f35-27a3-4a30-9550-2e10b2f98893\") "
Jul 01 19:29:45.854257 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:45.854205    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1522f35-27a3-4a30-9550-2e10b2f98893-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d1522f35-27a3-4a30-9550-2e10b2f98893" (UID: "d1522f35-27a3-4a30-9550-2e10b2f98893"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:29:45.854257 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:45.854226    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1522f35-27a3-4a30-9550-2e10b2f98893-service-ca" (OuterVolumeSpecName: "service-ca") pod "d1522f35-27a3-4a30-9550-2e10b2f98893" (UID: "d1522f35-27a3-4a30-9550-2e10b2f98893"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:29:45.854440 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:45.854409    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1522f35-27a3-4a30-9550-2e10b2f98893-console-config" (OuterVolumeSpecName: "console-config") pod "d1522f35-27a3-4a30-9550-2e10b2f98893" (UID: "d1522f35-27a3-4a30-9550-2e10b2f98893"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:29:45.854536 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:45.854512    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1522f35-27a3-4a30-9550-2e10b2f98893-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d1522f35-27a3-4a30-9550-2e10b2f98893" (UID: "d1522f35-27a3-4a30-9550-2e10b2f98893"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:29:45.855928 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:45.855892    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1522f35-27a3-4a30-9550-2e10b2f98893-kube-api-access-wqck8" (OuterVolumeSpecName: "kube-api-access-wqck8") pod "d1522f35-27a3-4a30-9550-2e10b2f98893" (UID: "d1522f35-27a3-4a30-9550-2e10b2f98893"). InnerVolumeSpecName "kube-api-access-wqck8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:29:45.856106 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:45.856080    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1522f35-27a3-4a30-9550-2e10b2f98893-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d1522f35-27a3-4a30-9550-2e10b2f98893" (UID: "d1522f35-27a3-4a30-9550-2e10b2f98893"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:29:45.856155 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:45.856135    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1522f35-27a3-4a30-9550-2e10b2f98893-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d1522f35-27a3-4a30-9550-2e10b2f98893" (UID: "d1522f35-27a3-4a30-9550-2e10b2f98893"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:29:45.955076 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:45.955041    2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d1522f35-27a3-4a30-9550-2e10b2f98893-oauth-serving-cert\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:29:45.955076 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:45.955068    2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d1522f35-27a3-4a30-9550-2e10b2f98893-service-ca\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:29:45.955076 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:45.955077    2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d1522f35-27a3-4a30-9550-2e10b2f98893-console-oauth-config\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:29:45.955076 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:45.955086    2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d1522f35-27a3-4a30-9550-2e10b2f98893-console-config\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:29:45.955322 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:45.955094    2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1522f35-27a3-4a30-9550-2e10b2f98893-trusted-ca-bundle\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:29:45.955322 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:45.955103    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqck8\" (UniqueName: \"kubernetes.io/projected/d1522f35-27a3-4a30-9550-2e10b2f98893-kube-api-access-wqck8\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:29:45.955322 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:45.955111    2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d1522f35-27a3-4a30-9550-2e10b2f98893-console-serving-cert\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:29:46.574451 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:46.574414    2574 generic.go:356] "Generic (PLEG): container finished" podID="d1522f35-27a3-4a30-9550-2e10b2f98893" containerID="9e88c913e664935fc49bd75d9af0a7e1460c39ccd64efd3bfc5ce579987a7e00" exitCode=0
Jul 01 19:29:46.574951 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:46.574461    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c748d989c-rn88n"
Jul 01 19:29:46.574951 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:46.574503    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c748d989c-rn88n" event={"ID":"d1522f35-27a3-4a30-9550-2e10b2f98893","Type":"ContainerDied","Data":"9e88c913e664935fc49bd75d9af0a7e1460c39ccd64efd3bfc5ce579987a7e00"}
Jul 01 19:29:46.574951 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:46.574554    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c748d989c-rn88n" event={"ID":"d1522f35-27a3-4a30-9550-2e10b2f98893","Type":"ContainerDied","Data":"b2d9ea57f8f2147b4d5804dd3c94fbb1decba86502e309de4215921a27b3dc53"}
Jul 01 19:29:46.574951 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:46.574574    2574 scope.go:117] "RemoveContainer" containerID="9e88c913e664935fc49bd75d9af0a7e1460c39ccd64efd3bfc5ce579987a7e00"
Jul 01 19:29:46.583038 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:46.583019    2574 scope.go:117] "RemoveContainer" containerID="9e88c913e664935fc49bd75d9af0a7e1460c39ccd64efd3bfc5ce579987a7e00"
Jul 01 19:29:46.583275 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:29:46.583257    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e88c913e664935fc49bd75d9af0a7e1460c39ccd64efd3bfc5ce579987a7e00\": container with ID starting with 9e88c913e664935fc49bd75d9af0a7e1460c39ccd64efd3bfc5ce579987a7e00 not found: ID does not exist" containerID="9e88c913e664935fc49bd75d9af0a7e1460c39ccd64efd3bfc5ce579987a7e00"
Jul 01 19:29:46.583325 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:46.583281    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e88c913e664935fc49bd75d9af0a7e1460c39ccd64efd3bfc5ce579987a7e00"} err="failed to get container status \"9e88c913e664935fc49bd75d9af0a7e1460c39ccd64efd3bfc5ce579987a7e00\": rpc error: code = NotFound desc = could not find container \"9e88c913e664935fc49bd75d9af0a7e1460c39ccd64efd3bfc5ce579987a7e00\": container with ID starting with 9e88c913e664935fc49bd75d9af0a7e1460c39ccd64efd3bfc5ce579987a7e00 not found: ID does not exist"
Jul 01 19:29:46.591247 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:46.591224    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c748d989c-rn88n"]
Jul 01 19:29:46.595687 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:46.595667    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c748d989c-rn88n"]
Jul 01 19:29:48.356807 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:29:48.356768    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1522f35-27a3-4a30-9550-2e10b2f98893" path="/var/lib/kubelet/pods/d1522f35-27a3-4a30-9550-2e10b2f98893/volumes"
Jul 01 19:30:23.743184 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:23.743151    2574 generic.go:356] "Generic (PLEG): container finished" podID="afd6ed61-31e8-48a6-b2f3-74e5bf203c19" containerID="a7acf643f65e9e8aede4e8e2abe060bf5f55717c1b135c2433082a3bc4f2a25c" exitCode=0
Jul 01 19:30:23.743575 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:23.743221    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-h8x54" event={"ID":"afd6ed61-31e8-48a6-b2f3-74e5bf203c19","Type":"ContainerDied","Data":"a7acf643f65e9e8aede4e8e2abe060bf5f55717c1b135c2433082a3bc4f2a25c"}
Jul 01 19:30:24.873544 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:24.873521    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-h8x54"
Jul 01 19:30:24.985615 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:24.985579    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afd6ed61-31e8-48a6-b2f3-74e5bf203c19-model-cache\") pod \"afd6ed61-31e8-48a6-b2f3-74e5bf203c19\" (UID: \"afd6ed61-31e8-48a6-b2f3-74e5bf203c19\") "
Jul 01 19:30:24.985785 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:24.985623    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mljq\" (UniqueName: \"kubernetes.io/projected/afd6ed61-31e8-48a6-b2f3-74e5bf203c19-kube-api-access-9mljq\") pod \"afd6ed61-31e8-48a6-b2f3-74e5bf203c19\" (UID: \"afd6ed61-31e8-48a6-b2f3-74e5bf203c19\") "
Jul 01 19:30:24.987895 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:24.987870    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd6ed61-31e8-48a6-b2f3-74e5bf203c19-kube-api-access-9mljq" (OuterVolumeSpecName: "kube-api-access-9mljq") pod "afd6ed61-31e8-48a6-b2f3-74e5bf203c19" (UID: "afd6ed61-31e8-48a6-b2f3-74e5bf203c19"). InnerVolumeSpecName "kube-api-access-9mljq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:30:25.004452 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:25.004396    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd6ed61-31e8-48a6-b2f3-74e5bf203c19-model-cache" (OuterVolumeSpecName: "model-cache") pod "afd6ed61-31e8-48a6-b2f3-74e5bf203c19" (UID: "afd6ed61-31e8-48a6-b2f3-74e5bf203c19"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:30:25.086715 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:25.086688    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afd6ed61-31e8-48a6-b2f3-74e5bf203c19-model-cache\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:30:25.086715 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:25.086712    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mljq\" (UniqueName: \"kubernetes.io/projected/afd6ed61-31e8-48a6-b2f3-74e5bf203c19-kube-api-access-9mljq\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:30:25.753881 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:25.753800    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-h8x54" event={"ID":"afd6ed61-31e8-48a6-b2f3-74e5bf203c19","Type":"ContainerDied","Data":"a4620a4643b2020c015c67f23cd1cd0774f8a909924b68c4a3c8721863a13b0a"}
Jul 01 19:30:25.753881 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:25.753821    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-h8x54"
Jul 01 19:30:25.753881 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:25.753839    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4620a4643b2020c015c67f23cd1cd0774f8a909924b68c4a3c8721863a13b0a"
Jul 01 19:30:37.492019 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:37.491980    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7"]
Jul 01 19:30:37.492475 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:37.492386    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afd6ed61-31e8-48a6-b2f3-74e5bf203c19" containerName="download-hf-model"
Jul 01 19:30:37.492475 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:37.492405    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="afd6ed61-31e8-48a6-b2f3-74e5bf203c19" containerName="download-hf-model"
Jul 01 19:30:37.492475 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:37.492420    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afd6ed61-31e8-48a6-b2f3-74e5bf203c19" containerName="s3-init"
Jul 01 19:30:37.492475 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:37.492428    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="afd6ed61-31e8-48a6-b2f3-74e5bf203c19" containerName="s3-init"
Jul 01 19:30:37.492475 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:37.492439    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1522f35-27a3-4a30-9550-2e10b2f98893" containerName="console"
Jul 01 19:30:37.492475 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:37.492445    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1522f35-27a3-4a30-9550-2e10b2f98893" containerName="console"
Jul 01 19:30:37.492710 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:37.492529    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1522f35-27a3-4a30-9550-2e10b2f98893" containerName="console"
Jul 01 19:30:37.492710 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:37.492540    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="afd6ed61-31e8-48a6-b2f3-74e5bf203c19" containerName="s3-init"
Jul 01 19:30:37.496822 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:37.496801    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7"
Jul 01 19:30:37.499279 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:37.499256    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 19:30:37.499394 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:37.499350    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 19:30:37.499394 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:37.499357    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-95l22\""
Jul 01 19:30:37.499394 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:37.499368    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-e27cf-predictor-serving-cert\""
Jul 01 19:30:37.499555 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:37.499443    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 19:30:37.499856 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:37.499842    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 19:30:37.499917 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:37.499874    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-e27cf-kube-rbac-proxy-sar-config\""
Jul 01 19:30:37.508806 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:37.508785    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7"]
Jul 01 19:30:37.596706 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:37.596666    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e27e002a-4ea3-44f6-b1c1-6ac637649400-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7\" (UID: \"e27e002a-4ea3-44f6-b1c1-6ac637649400\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7"
Jul 01 19:30:37.596931 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:37.596718    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e27e002a-4ea3-44f6-b1c1-6ac637649400-proxy-tls\") pod \"isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7\" (UID: \"e27e002a-4ea3-44f6-b1c1-6ac637649400\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7"
Jul 01 19:30:37.596931 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:37.596812    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skkdm\" (UniqueName: \"kubernetes.io/projected/e27e002a-4ea3-44f6-b1c1-6ac637649400-kube-api-access-skkdm\") pod \"isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7\" (UID: \"e27e002a-4ea3-44f6-b1c1-6ac637649400\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7"
Jul 01 19:30:37.596931 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:37.596851    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e27e002a-4ea3-44f6-b1c1-6ac637649400-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7\" (UID: \"e27e002a-4ea3-44f6-b1c1-6ac637649400\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7"
Jul 01 19:30:37.597076 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:37.596968    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e27e002a-4ea3-44f6-b1c1-6ac637649400-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7\" (UID: \"e27e002a-4ea3-44f6-b1c1-6ac637649400\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7"
Jul 01 19:30:37.697727 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:37.697674    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e27e002a-4ea3-44f6-b1c1-6ac637649400-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7\" (UID: \"e27e002a-4ea3-44f6-b1c1-6ac637649400\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7"
Jul 01 19:30:37.697971 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:37.697769    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e27e002a-4ea3-44f6-b1c1-6ac637649400-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7\" (UID: \"e27e002a-4ea3-44f6-b1c1-6ac637649400\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7"
Jul 01 19:30:37.697971 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:37.697856    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e27e002a-4ea3-44f6-b1c1-6ac637649400-proxy-tls\") pod \"isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7\" (UID: \"e27e002a-4ea3-44f6-b1c1-6ac637649400\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7"
Jul 01 19:30:37.697971 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:37.697949    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skkdm\" (UniqueName: \"kubernetes.io/projected/e27e002a-4ea3-44f6-b1c1-6ac637649400-kube-api-access-skkdm\") pod \"isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7\" (UID: \"e27e002a-4ea3-44f6-b1c1-6ac637649400\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7"
Jul 01 19:30:37.698165 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:37.697982    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e27e002a-4ea3-44f6-b1c1-6ac637649400-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7\" (UID: \"e27e002a-4ea3-44f6-b1c1-6ac637649400\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7"
Jul 01 19:30:37.698259 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:37.698234    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e27e002a-4ea3-44f6-b1c1-6ac637649400-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7\" (UID: \"e27e002a-4ea3-44f6-b1c1-6ac637649400\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7"
Jul 01 19:30:37.698514 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:37.698491    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e27e002a-4ea3-44f6-b1c1-6ac637649400-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7\" (UID: \"e27e002a-4ea3-44f6-b1c1-6ac637649400\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7"
Jul 01 19:30:37.698640 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:37.698624    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e27e002a-4ea3-44f6-b1c1-6ac637649400-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7\" (UID: \"e27e002a-4ea3-44f6-b1c1-6ac637649400\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7"
Jul 01 19:30:37.700341 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:37.700324    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e27e002a-4ea3-44f6-b1c1-6ac637649400-proxy-tls\") pod \"isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7\" (UID: \"e27e002a-4ea3-44f6-b1c1-6ac637649400\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7"
Jul 01 19:30:37.707273 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:37.707251    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skkdm\" (UniqueName: \"kubernetes.io/projected/e27e002a-4ea3-44f6-b1c1-6ac637649400-kube-api-access-skkdm\") pod \"isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7\" (UID: \"e27e002a-4ea3-44f6-b1c1-6ac637649400\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7"
Jul 01 19:30:37.808543 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:37.808462    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7"
Jul 01 19:30:37.948972 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:37.948944    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7"]
Jul 01 19:30:37.951108 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:30:37.951078    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode27e002a_4ea3_44f6_b1c1_6ac637649400.slice/crio-86f423b50fd7c32e69ce6a6f11153cf9dbbef733face14ec20546802af70d5cb WatchSource:0}: Error finding container 86f423b50fd7c32e69ce6a6f11153cf9dbbef733face14ec20546802af70d5cb: Status 404 returned error can't find the container with id 86f423b50fd7c32e69ce6a6f11153cf9dbbef733face14ec20546802af70d5cb
Jul 01 19:30:38.817649 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:38.817618    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" event={"ID":"e27e002a-4ea3-44f6-b1c1-6ac637649400","Type":"ContainerStarted","Data":"9cdd6a01d4963767f184bffc68a34cbad2d599045d46d2c6690ed61355460677"}
Jul 01 19:30:38.818053 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:38.817684    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" event={"ID":"e27e002a-4ea3-44f6-b1c1-6ac637649400","Type":"ContainerStarted","Data":"86f423b50fd7c32e69ce6a6f11153cf9dbbef733face14ec20546802af70d5cb"}
Jul 01 19:30:41.833097 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:41.833060    2574 generic.go:356] "Generic (PLEG): container finished" podID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerID="9cdd6a01d4963767f184bffc68a34cbad2d599045d46d2c6690ed61355460677" exitCode=0
Jul 01 19:30:41.833501 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:41.833131    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" event={"ID":"e27e002a-4ea3-44f6-b1c1-6ac637649400","Type":"ContainerDied","Data":"9cdd6a01d4963767f184bffc68a34cbad2d599045d46d2c6690ed61355460677"}
Jul 01 19:30:57.923404 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:57.923373    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" event={"ID":"e27e002a-4ea3-44f6-b1c1-6ac637649400","Type":"ContainerStarted","Data":"b3f585c63504fd1aab3f660b6056288b66d7d16d6b4081fd9214d61e118e15b1"}
Jul 01 19:30:59.944281 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:30:59.944249    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" event={"ID":"e27e002a-4ea3-44f6-b1c1-6ac637649400","Type":"ContainerStarted","Data":"c06d40fcab465854c5ca457f65b4eba63a63ef3102b14320e22bec90bbe8a0be"}
Jul 01 19:31:03.966257 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:31:03.966222    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" event={"ID":"e27e002a-4ea3-44f6-b1c1-6ac637649400","Type":"ContainerStarted","Data":"be91c9dd3f5e46e84b032da613f56b485b1fd514bedf307f14b97101b6eb079b"}
Jul 01 19:31:03.966671 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:31:03.966354    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7"
Jul 01 19:31:03.966671 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:31:03.966415    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7"
Jul 01 19:31:03.967683 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:31:03.967655    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 19:31:03.991210 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:31:03.991160    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" podStartSLOduration=2.449225954 podStartE2EDuration="26.991145989s" totalImagesPullingTime="20.950893563s" totalInitContainerRuntime="3.591026472s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:30:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:31:03.988649863 +0000 UTC m=+570.080649323" watchObservedRunningTime="2026-07-01 19:31:03.991145989 +0000 UTC m=+570.083145437"
Jul 01 19:31:04.971646 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:31:04.971606    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7"
Jul 01 19:31:04.972129 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:31:04.971686    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 19:31:04.972664 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:31:04.972630    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:31:04.975433 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:31:04.975414    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7"
Jul 01 19:31:05.976134 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:31:05.976091    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 19:31:05.976601 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:31:05.976402    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:31:06.980236 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:31:06.980194    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 19:31:06.980710 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:31:06.980548    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:31:16.980531 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:31:16.980477    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 19:31:16.981048 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:31:16.980975    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:31:26.980982 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:31:26.980933    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 19:31:26.981494 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:31:26.981378    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:31:34.345084 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:31:34.345054    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bplsm_bfb845e6-341d-492e-91c9-9c698a0b8d29/ovn-acl-logging/0.log"
Jul 01 19:31:34.345084 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:31:34.345070    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bplsm_bfb845e6-341d-492e-91c9-9c698a0b8d29/ovn-acl-logging/0.log"
Jul 01 19:31:36.980813 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:31:36.980771    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 19:31:36.982326 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:31:36.981241    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:31:46.981056 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:31:46.981011    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 19:31:46.981494 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:31:46.981467    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:31:56.980852 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:31:56.980746    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 19:31:56.981285 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:31:56.981166    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:32:06.980424 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:06.980370    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 19:32:06.980945 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:06.980782    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:32:16.981716 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:16.981685    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7"
Jul 01 19:32:16.982217 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:16.982160    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7"
Jul 01 19:32:22.642894 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:22.642862    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7"]
Jul 01 19:32:22.643300 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:22.643154    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="kserve-container" containerID="cri-o://b3f585c63504fd1aab3f660b6056288b66d7d16d6b4081fd9214d61e118e15b1" gracePeriod=30
Jul 01 19:32:22.643300 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:22.643191    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="kube-rbac-proxy" containerID="cri-o://c06d40fcab465854c5ca457f65b4eba63a63ef3102b14320e22bec90bbe8a0be" gracePeriod=30
Jul 01 19:32:22.643300 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:22.643233    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="agent" containerID="cri-o://be91c9dd3f5e46e84b032da613f56b485b1fd514bedf307f14b97101b6eb079b" gracePeriod=30
Jul 01 19:32:22.673643 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:22.673609    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j"]
Jul 01 19:32:22.679316 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:22.678163    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j"
Jul 01 19:32:22.681091 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:22.681066    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-66e42-kube-rbac-proxy-sar-config\""
Jul 01 19:32:22.681239 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:22.681208    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-66e42-predictor-serving-cert\""
Jul 01 19:32:22.689560 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:22.689534    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j"]
Jul 01 19:32:22.750127 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:22.750091    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc"]
Jul 01 19:32:22.754039 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:22.754017    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc"
Jul 01 19:32:22.758394 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:22.758287    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-66e42-predictor-serving-cert\""
Jul 01 19:32:22.759011 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:22.758993    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-66e42-kube-rbac-proxy-sar-config\""
Jul 01 19:32:22.764758 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:22.764738    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc"]
Jul 01 19:32:22.795634 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:22.795606    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cvcs\" (UniqueName: \"kubernetes.io/projected/835093bf-c5c0-4835-96c5-bd03e5097839-kube-api-access-6cvcs\") pod \"isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j\" (UID: \"835093bf-c5c0-4835-96c5-bd03e5097839\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j"
Jul 01 19:32:22.795634 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:22.795641    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/835093bf-c5c0-4835-96c5-bd03e5097839-proxy-tls\") pod \"isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j\" (UID: \"835093bf-c5c0-4835-96c5-bd03e5097839\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j"
Jul 01 19:32:22.795819 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:22.795675    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/835093bf-c5c0-4835-96c5-bd03e5097839-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j\" (UID: \"835093bf-c5c0-4835-96c5-bd03e5097839\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j"
Jul 01 19:32:22.795819 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:22.795699    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/835093bf-c5c0-4835-96c5-bd03e5097839-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j\" (UID: \"835093bf-c5c0-4835-96c5-bd03e5097839\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j"
Jul 01 19:32:22.795819 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:22.795745    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/835093bf-c5c0-4835-96c5-bd03e5097839-cabundle-cert\") pod \"isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j\" (UID: \"835093bf-c5c0-4835-96c5-bd03e5097839\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j"
Jul 01 19:32:22.897253 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:22.897183    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1ca2705-0c5d-45cb-9f54-5f82238eabec-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc\" (UID: \"e1ca2705-0c5d-45cb-9f54-5f82238eabec\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc"
Jul 01 19:32:22.897253 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:22.897221    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1ca2705-0c5d-45cb-9f54-5f82238eabec-proxy-tls\") pod \"isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc\" (UID: \"e1ca2705-0c5d-45cb-9f54-5f82238eabec\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc"
Jul 01 19:32:22.897253 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:22.897250    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1ca2705-0c5d-45cb-9f54-5f82238eabec-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc\" (UID: \"e1ca2705-0c5d-45cb-9f54-5f82238eabec\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc"
Jul 01 19:32:22.897516 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:22.897355    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cvcs\" (UniqueName: \"kubernetes.io/projected/835093bf-c5c0-4835-96c5-bd03e5097839-kube-api-access-6cvcs\") pod \"isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j\" (UID: \"835093bf-c5c0-4835-96c5-bd03e5097839\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j"
Jul 01 19:32:22.897516 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:22.897390    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/835093bf-c5c0-4835-96c5-bd03e5097839-proxy-tls\") pod \"isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j\" (UID: \"835093bf-c5c0-4835-96c5-bd03e5097839\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j"
Jul 01 19:32:22.897516 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:22.897417    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqq55\" (UniqueName: \"kubernetes.io/projected/e1ca2705-0c5d-45cb-9f54-5f82238eabec-kube-api-access-zqq55\") pod \"isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc\" (UID: \"e1ca2705-0c5d-45cb-9f54-5f82238eabec\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc"
Jul 01 19:32:22.897516 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:22.897467    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/835093bf-c5c0-4835-96c5-bd03e5097839-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j\" (UID: \"835093bf-c5c0-4835-96c5-bd03e5097839\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j"
Jul 01 19:32:22.897516 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:22.897499    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/835093bf-c5c0-4835-96c5-bd03e5097839-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j\" (UID: \"835093bf-c5c0-4835-96c5-bd03e5097839\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j"
Jul 01 19:32:22.897750 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:22.897534    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1ca2705-0c5d-45cb-9f54-5f82238eabec-cabundle-cert\") pod \"isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc\" (UID: \"e1ca2705-0c5d-45cb-9f54-5f82238eabec\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc"
Jul 01 19:32:22.897750 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:22.897596    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/835093bf-c5c0-4835-96c5-bd03e5097839-cabundle-cert\") pod \"isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j\" (UID: \"835093bf-c5c0-4835-96c5-bd03e5097839\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j"
Jul 01 19:32:22.897946 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:22.897922    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/835093bf-c5c0-4835-96c5-bd03e5097839-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j\" (UID: \"835093bf-c5c0-4835-96c5-bd03e5097839\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j"
Jul 01 19:32:22.898222 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:22.898205    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/835093bf-c5c0-4835-96c5-bd03e5097839-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j\" (UID: \"835093bf-c5c0-4835-96c5-bd03e5097839\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j"
Jul 01 19:32:22.898270 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:22.898250    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/835093bf-c5c0-4835-96c5-bd03e5097839-cabundle-cert\") pod \"isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j\" (UID: \"835093bf-c5c0-4835-96c5-bd03e5097839\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j"
Jul 01 19:32:22.899760 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:22.899740    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/835093bf-c5c0-4835-96c5-bd03e5097839-proxy-tls\") pod \"isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j\" (UID: \"835093bf-c5c0-4835-96c5-bd03e5097839\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j"
Jul 01 19:32:22.905460 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:22.905436    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cvcs\" (UniqueName: \"kubernetes.io/projected/835093bf-c5c0-4835-96c5-bd03e5097839-kube-api-access-6cvcs\") pod \"isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j\" (UID: \"835093bf-c5c0-4835-96c5-bd03e5097839\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j"
Jul 01 19:32:22.991431 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:22.991407    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j"
Jul 01 19:32:22.998675 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:22.998653    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1ca2705-0c5d-45cb-9f54-5f82238eabec-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc\" (UID: \"e1ca2705-0c5d-45cb-9f54-5f82238eabec\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc"
Jul 01 19:32:22.998785 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:22.998690    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1ca2705-0c5d-45cb-9f54-5f82238eabec-proxy-tls\") pod \"isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc\" (UID: \"e1ca2705-0c5d-45cb-9f54-5f82238eabec\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc"
Jul 01 19:32:22.998785 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:22.998717    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1ca2705-0c5d-45cb-9f54-5f82238eabec-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc\" (UID: \"e1ca2705-0c5d-45cb-9f54-5f82238eabec\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc"
Jul 01 19:32:22.998785 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:22.998751    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqq55\" (UniqueName: \"kubernetes.io/projected/e1ca2705-0c5d-45cb-9f54-5f82238eabec-kube-api-access-zqq55\") pod \"isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc\" (UID: \"e1ca2705-0c5d-45cb-9f54-5f82238eabec\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc"
Jul 01 19:32:22.998785 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:22.998782    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1ca2705-0c5d-45cb-9f54-5f82238eabec-cabundle-cert\") pod \"isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc\" (UID: \"e1ca2705-0c5d-45cb-9f54-5f82238eabec\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc"
Jul 01 19:32:22.999028 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:32:22.998816    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-serving-cert: secret "isvc-xgboost-graph-raw-66e42-predictor-serving-cert" not found
Jul 01 19:32:22.999028 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:32:22.998887    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1ca2705-0c5d-45cb-9f54-5f82238eabec-proxy-tls podName:e1ca2705-0c5d-45cb-9f54-5f82238eabec nodeName:}" failed. No retries permitted until 2026-07-01 19:32:23.498865215 +0000 UTC m=+649.590864645 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e1ca2705-0c5d-45cb-9f54-5f82238eabec-proxy-tls") pod "isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc" (UID: "e1ca2705-0c5d-45cb-9f54-5f82238eabec") : secret "isvc-xgboost-graph-raw-66e42-predictor-serving-cert" not found
Jul 01 19:32:22.999147 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:22.999133    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1ca2705-0c5d-45cb-9f54-5f82238eabec-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc\" (UID: \"e1ca2705-0c5d-45cb-9f54-5f82238eabec\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc"
Jul 01 19:32:22.999437 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:22.999418    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1ca2705-0c5d-45cb-9f54-5f82238eabec-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc\" (UID: \"e1ca2705-0c5d-45cb-9f54-5f82238eabec\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc"
Jul 01 19:32:22.999599 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:22.999577    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1ca2705-0c5d-45cb-9f54-5f82238eabec-cabundle-cert\") pod \"isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc\" (UID: \"e1ca2705-0c5d-45cb-9f54-5f82238eabec\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc"
Jul 01 19:32:23.008411 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:23.008387    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqq55\" (UniqueName: \"kubernetes.io/projected/e1ca2705-0c5d-45cb-9f54-5f82238eabec-kube-api-access-zqq55\") pod \"isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc\" (UID: \"e1ca2705-0c5d-45cb-9f54-5f82238eabec\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc"
Jul 01 19:32:23.122406 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:23.122379    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j"]
Jul 01 19:32:23.124445 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:32:23.124419    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod835093bf_c5c0_4835_96c5_bd03e5097839.slice/crio-edbc676907a93baf81c94da9de1ffe05d115178da00affcfe596f55954915c4e WatchSource:0}: Error finding container edbc676907a93baf81c94da9de1ffe05d115178da00affcfe596f55954915c4e: Status 404 returned error can't find the container with id edbc676907a93baf81c94da9de1ffe05d115178da00affcfe596f55954915c4e
Jul 01 19:32:23.328826 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:23.328793    2574 generic.go:356] "Generic (PLEG): container finished" podID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerID="c06d40fcab465854c5ca457f65b4eba63a63ef3102b14320e22bec90bbe8a0be" exitCode=2
Jul 01 19:32:23.329004 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:23.328868    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" event={"ID":"e27e002a-4ea3-44f6-b1c1-6ac637649400","Type":"ContainerDied","Data":"c06d40fcab465854c5ca457f65b4eba63a63ef3102b14320e22bec90bbe8a0be"}
Jul 01 19:32:23.330422 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:23.330397    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j" event={"ID":"835093bf-c5c0-4835-96c5-bd03e5097839","Type":"ContainerStarted","Data":"f3e8cd9f9358ead8597bc5643888871b2a35b166bbbd8c6a179c1ee8872fa1e8"}
Jul 01 19:32:23.330528 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:23.330450    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j" event={"ID":"835093bf-c5c0-4835-96c5-bd03e5097839","Type":"ContainerStarted","Data":"edbc676907a93baf81c94da9de1ffe05d115178da00affcfe596f55954915c4e"}
Jul 01 19:32:23.503562 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:23.503463    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1ca2705-0c5d-45cb-9f54-5f82238eabec-proxy-tls\") pod \"isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc\" (UID: \"e1ca2705-0c5d-45cb-9f54-5f82238eabec\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc"
Jul 01 19:32:23.505635 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:23.505613    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1ca2705-0c5d-45cb-9f54-5f82238eabec-proxy-tls\") pod \"isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc\" (UID: \"e1ca2705-0c5d-45cb-9f54-5f82238eabec\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc"
Jul 01 19:32:23.664304 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:23.664276    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc"
Jul 01 19:32:23.791989 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:23.791960    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc"]
Jul 01 19:32:23.794105 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:32:23.794074    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1ca2705_0c5d_45cb_9f54_5f82238eabec.slice/crio-bbf83234cd6c328206cef0143594e59b5bf92446b070ee537aa9a79898e4dd35 WatchSource:0}: Error finding container bbf83234cd6c328206cef0143594e59b5bf92446b070ee537aa9a79898e4dd35: Status 404 returned error can't find the container with id bbf83234cd6c328206cef0143594e59b5bf92446b070ee537aa9a79898e4dd35
Jul 01 19:32:24.338040 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:24.338004    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc" event={"ID":"e1ca2705-0c5d-45cb-9f54-5f82238eabec","Type":"ContainerStarted","Data":"3c31a052c7effa3a6eee14eaab1d2df723838dbe7e93fd9b52700202fade41eb"}
Jul 01 19:32:24.338224 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:24.338070    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc" event={"ID":"e1ca2705-0c5d-45cb-9f54-5f82238eabec","Type":"ContainerStarted","Data":"bbf83234cd6c328206cef0143594e59b5bf92446b070ee537aa9a79898e4dd35"}
Jul 01 19:32:24.972052 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:24.972004    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 01 19:32:26.981211 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:26.981175    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:32:26.982442 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:26.982410    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 19:32:27.353395 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:27.353367    2574 generic.go:356] "Generic (PLEG): container finished" podID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerID="b3f585c63504fd1aab3f660b6056288b66d7d16d6b4081fd9214d61e118e15b1" exitCode=0
Jul 01 19:32:27.353545 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:27.353445    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" event={"ID":"e27e002a-4ea3-44f6-b1c1-6ac637649400","Type":"ContainerDied","Data":"b3f585c63504fd1aab3f660b6056288b66d7d16d6b4081fd9214d61e118e15b1"}
Jul 01 19:32:27.354734 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:27.354708    2574 generic.go:356] "Generic (PLEG): container finished" podID="835093bf-c5c0-4835-96c5-bd03e5097839" containerID="f3e8cd9f9358ead8597bc5643888871b2a35b166bbbd8c6a179c1ee8872fa1e8" exitCode=0
Jul 01 19:32:27.354833 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:27.354779    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j" event={"ID":"835093bf-c5c0-4835-96c5-bd03e5097839","Type":"ContainerDied","Data":"f3e8cd9f9358ead8597bc5643888871b2a35b166bbbd8c6a179c1ee8872fa1e8"}
Jul 01 19:32:28.361944 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:28.361907    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j" event={"ID":"835093bf-c5c0-4835-96c5-bd03e5097839","Type":"ContainerStarted","Data":"3fa3e1fc96205099e91b7c7aa0df7abe266409deb40f7d23e5178068b513a599"}
Jul 01 19:32:28.362375 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:28.361987    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j" event={"ID":"835093bf-c5c0-4835-96c5-bd03e5097839","Type":"ContainerStarted","Data":"2bd27b81b1402cc33a1ba8ce10d6accec4545b46a5603c31ccd8a7e9acf1b9de"}
Jul 01 19:32:28.362375 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:28.362231    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j"
Jul 01 19:32:28.363454 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:28.363434    2574 generic.go:356] "Generic (PLEG): container finished" podID="e1ca2705-0c5d-45cb-9f54-5f82238eabec" containerID="3c31a052c7effa3a6eee14eaab1d2df723838dbe7e93fd9b52700202fade41eb" exitCode=0
Jul 01 19:32:28.363539 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:28.363504    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc" event={"ID":"e1ca2705-0c5d-45cb-9f54-5f82238eabec","Type":"ContainerDied","Data":"3c31a052c7effa3a6eee14eaab1d2df723838dbe7e93fd9b52700202fade41eb"}
Jul 01 19:32:28.384349 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:28.384306    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j" podStartSLOduration=2.775617729 podStartE2EDuration="6.384292499s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.60867477s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:32:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:32:28.382639286 +0000 UTC m=+654.474638754" watchObservedRunningTime="2026-07-01 19:32:28.384292499 +0000 UTC m=+654.476291946"
Jul 01 19:32:29.370191 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:29.370150    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j"
Jul 01 19:32:29.371778 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:29.371747    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j" podUID="835093bf-c5c0-4835-96c5-bd03e5097839" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 01 19:32:29.972372 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:29.972336    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 01 19:32:30.378174 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:30.377673    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j" podUID="835093bf-c5c0-4835-96c5-bd03e5097839" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 01 19:32:34.972838 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:34.972792    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 01 19:32:34.973291 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:34.972964    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7"
Jul 01 19:32:35.382984 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:35.382955    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j"
Jul 01 19:32:35.383671 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:35.383642    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j" podUID="835093bf-c5c0-4835-96c5-bd03e5097839" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 01 19:32:36.981425 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:36.981377    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:32:36.982644 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:36.982605    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 19:32:39.972701 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:39.972661    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 01 19:32:44.972733 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:44.972687    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 01 19:32:45.384335 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:45.384296    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j" podUID="835093bf-c5c0-4835-96c5-bd03e5097839" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 01 19:32:46.981215 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:46.981171    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:32:46.981653 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:46.981303    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7"
Jul 01 19:32:46.982373 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:46.982344    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 19:32:46.982481 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:46.982469    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7"
Jul 01 19:32:49.972708 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:49.972662    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 01 19:32:52.498534 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:52.498495    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc" event={"ID":"e1ca2705-0c5d-45cb-9f54-5f82238eabec","Type":"ContainerStarted","Data":"af0e88b541ec49f1a7bdc49d5014c3f766f5d385ab58883e3de261bf0c71cb79"}
Jul 01 19:32:52.498938 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:52.498576    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc" event={"ID":"e1ca2705-0c5d-45cb-9f54-5f82238eabec","Type":"ContainerStarted","Data":"fe8c1ccf408600e0925ef9f67274321f4f72020a4d91cca0b7153c6661ef3648"}
Jul 01 19:32:52.498938 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:52.498773    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc"
Jul 01 19:32:52.518236 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:52.518191    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc" podStartSLOduration=3.786898838 podStartE2EDuration="30.518174851s" totalImagesPullingTime="23.131661528s" totalInitContainerRuntime="3.599614485s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:32:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:32:52.516573838 +0000 UTC m=+678.608573287" watchObservedRunningTime="2026-07-01 19:32:52.518174851 +0000 UTC m=+678.610174299"
Jul 01 19:32:52.793132 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:52.793109    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7"
Jul 01 19:32:52.872675 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:52.872640    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e27e002a-4ea3-44f6-b1c1-6ac637649400-kserve-provision-location\") pod \"e27e002a-4ea3-44f6-b1c1-6ac637649400\" (UID: \"e27e002a-4ea3-44f6-b1c1-6ac637649400\") "
Jul 01 19:32:52.872675 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:52.872675    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-skkdm\" (UniqueName: \"kubernetes.io/projected/e27e002a-4ea3-44f6-b1c1-6ac637649400-kube-api-access-skkdm\") pod \"e27e002a-4ea3-44f6-b1c1-6ac637649400\" (UID: \"e27e002a-4ea3-44f6-b1c1-6ac637649400\") "
Jul 01 19:32:52.872957 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:52.872707    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e27e002a-4ea3-44f6-b1c1-6ac637649400-proxy-tls\") pod \"e27e002a-4ea3-44f6-b1c1-6ac637649400\" (UID: \"e27e002a-4ea3-44f6-b1c1-6ac637649400\") "
Jul 01 19:32:52.872957 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:52.872725    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e27e002a-4ea3-44f6-b1c1-6ac637649400-cabundle-cert\") pod \"e27e002a-4ea3-44f6-b1c1-6ac637649400\" (UID: \"e27e002a-4ea3-44f6-b1c1-6ac637649400\") "
Jul 01 19:32:52.872957 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:52.872857    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e27e002a-4ea3-44f6-b1c1-6ac637649400-kube-rbac-proxy-sar-config\") pod \"e27e002a-4ea3-44f6-b1c1-6ac637649400\" (UID: \"e27e002a-4ea3-44f6-b1c1-6ac637649400\") "
Jul 01 19:32:52.873123 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:52.872991    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27e002a-4ea3-44f6-b1c1-6ac637649400-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e27e002a-4ea3-44f6-b1c1-6ac637649400" (UID: "e27e002a-4ea3-44f6-b1c1-6ac637649400"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:32:52.873123 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:52.873112    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e27e002a-4ea3-44f6-b1c1-6ac637649400-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e27e002a-4ea3-44f6-b1c1-6ac637649400" (UID: "e27e002a-4ea3-44f6-b1c1-6ac637649400"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:32:52.873193 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:52.873161    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e27e002a-4ea3-44f6-b1c1-6ac637649400-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e27e002a-4ea3-44f6-b1c1-6ac637649400" (UID: "e27e002a-4ea3-44f6-b1c1-6ac637649400"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:32:52.873252 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:52.873238    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e27e002a-4ea3-44f6-b1c1-6ac637649400-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:32:52.873288 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:52.873254    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e27e002a-4ea3-44f6-b1c1-6ac637649400-kserve-provision-location\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:32:52.873288 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:52.873263    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e27e002a-4ea3-44f6-b1c1-6ac637649400-cabundle-cert\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:32:52.874880 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:52.874861    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e27e002a-4ea3-44f6-b1c1-6ac637649400-kube-api-access-skkdm" (OuterVolumeSpecName: "kube-api-access-skkdm") pod "e27e002a-4ea3-44f6-b1c1-6ac637649400" (UID: "e27e002a-4ea3-44f6-b1c1-6ac637649400"). InnerVolumeSpecName "kube-api-access-skkdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:32:52.874880 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:52.874861    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e27e002a-4ea3-44f6-b1c1-6ac637649400-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e27e002a-4ea3-44f6-b1c1-6ac637649400" (UID: "e27e002a-4ea3-44f6-b1c1-6ac637649400"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:32:52.974457 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:52.974420    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skkdm\" (UniqueName: \"kubernetes.io/projected/e27e002a-4ea3-44f6-b1c1-6ac637649400-kube-api-access-skkdm\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:32:52.974457 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:52.974450    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e27e002a-4ea3-44f6-b1c1-6ac637649400-proxy-tls\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:32:53.505533 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:53.505501    2574 generic.go:356] "Generic (PLEG): container finished" podID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerID="be91c9dd3f5e46e84b032da613f56b485b1fd514bedf307f14b97101b6eb079b" exitCode=0
Jul 01 19:32:53.506030 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:53.505555    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7"
Jul 01 19:32:53.506030 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:53.505583    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" event={"ID":"e27e002a-4ea3-44f6-b1c1-6ac637649400","Type":"ContainerDied","Data":"be91c9dd3f5e46e84b032da613f56b485b1fd514bedf307f14b97101b6eb079b"}
Jul 01 19:32:53.506030 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:53.505645    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7" event={"ID":"e27e002a-4ea3-44f6-b1c1-6ac637649400","Type":"ContainerDied","Data":"86f423b50fd7c32e69ce6a6f11153cf9dbbef733face14ec20546802af70d5cb"}
Jul 01 19:32:53.506030 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:53.505674    2574 scope.go:117] "RemoveContainer" containerID="be91c9dd3f5e46e84b032da613f56b485b1fd514bedf307f14b97101b6eb079b"
Jul 01 19:32:53.506280 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:53.506096    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc"
Jul 01 19:32:53.507690 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:53.507664    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc" podUID="e1ca2705-0c5d-45cb-9f54-5f82238eabec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 01 19:32:53.515179 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:53.514873    2574 scope.go:117] "RemoveContainer" containerID="c06d40fcab465854c5ca457f65b4eba63a63ef3102b14320e22bec90bbe8a0be"
Jul 01 19:32:53.522401 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:53.522386    2574 scope.go:117] "RemoveContainer" containerID="b3f585c63504fd1aab3f660b6056288b66d7d16d6b4081fd9214d61e118e15b1"
Jul 01 19:32:53.530121 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:53.530105    2574 scope.go:117] "RemoveContainer" containerID="9cdd6a01d4963767f184bffc68a34cbad2d599045d46d2c6690ed61355460677"
Jul 01 19:32:53.530375 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:53.530355    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7"]
Jul 01 19:32:53.534333 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:53.534311    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e27cf-predictor-96b47f887-bhhk7"]
Jul 01 19:32:53.537541 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:53.537527    2574 scope.go:117] "RemoveContainer" containerID="be91c9dd3f5e46e84b032da613f56b485b1fd514bedf307f14b97101b6eb079b"
Jul 01 19:32:53.537794 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:32:53.537774    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be91c9dd3f5e46e84b032da613f56b485b1fd514bedf307f14b97101b6eb079b\": container with ID starting with be91c9dd3f5e46e84b032da613f56b485b1fd514bedf307f14b97101b6eb079b not found: ID does not exist" containerID="be91c9dd3f5e46e84b032da613f56b485b1fd514bedf307f14b97101b6eb079b"
Jul 01 19:32:53.537876 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:53.537807    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be91c9dd3f5e46e84b032da613f56b485b1fd514bedf307f14b97101b6eb079b"} err="failed to get container status \"be91c9dd3f5e46e84b032da613f56b485b1fd514bedf307f14b97101b6eb079b\": rpc error: code = NotFound desc = could not find container \"be91c9dd3f5e46e84b032da613f56b485b1fd514bedf307f14b97101b6eb079b\": container with ID starting with be91c9dd3f5e46e84b032da613f56b485b1fd514bedf307f14b97101b6eb079b not found: ID does not exist"
Jul 01 19:32:53.537876 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:53.537833    2574 scope.go:117] "RemoveContainer" containerID="c06d40fcab465854c5ca457f65b4eba63a63ef3102b14320e22bec90bbe8a0be"
Jul 01 19:32:53.538222 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:32:53.538201    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c06d40fcab465854c5ca457f65b4eba63a63ef3102b14320e22bec90bbe8a0be\": container with ID starting with c06d40fcab465854c5ca457f65b4eba63a63ef3102b14320e22bec90bbe8a0be not found: ID does not exist" containerID="c06d40fcab465854c5ca457f65b4eba63a63ef3102b14320e22bec90bbe8a0be"
Jul 01 19:32:53.538305 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:53.538227    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c06d40fcab465854c5ca457f65b4eba63a63ef3102b14320e22bec90bbe8a0be"} err="failed to get container status \"c06d40fcab465854c5ca457f65b4eba63a63ef3102b14320e22bec90bbe8a0be\": rpc error: code = NotFound desc = could not find container \"c06d40fcab465854c5ca457f65b4eba63a63ef3102b14320e22bec90bbe8a0be\": container with ID starting with c06d40fcab465854c5ca457f65b4eba63a63ef3102b14320e22bec90bbe8a0be not found: ID does not exist"
Jul 01 19:32:53.538305 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:53.538249    2574 scope.go:117] "RemoveContainer" containerID="b3f585c63504fd1aab3f660b6056288b66d7d16d6b4081fd9214d61e118e15b1"
Jul 01 19:32:53.538518 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:32:53.538503    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3f585c63504fd1aab3f660b6056288b66d7d16d6b4081fd9214d61e118e15b1\": container with ID starting with b3f585c63504fd1aab3f660b6056288b66d7d16d6b4081fd9214d61e118e15b1 not found: ID does not exist" containerID="b3f585c63504fd1aab3f660b6056288b66d7d16d6b4081fd9214d61e118e15b1"
Jul 01 19:32:53.538583 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:53.538525    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3f585c63504fd1aab3f660b6056288b66d7d16d6b4081fd9214d61e118e15b1"} err="failed to get container status \"b3f585c63504fd1aab3f660b6056288b66d7d16d6b4081fd9214d61e118e15b1\": rpc error: code = NotFound desc = could not find container \"b3f585c63504fd1aab3f660b6056288b66d7d16d6b4081fd9214d61e118e15b1\": container with ID starting with b3f585c63504fd1aab3f660b6056288b66d7d16d6b4081fd9214d61e118e15b1 not found: ID does not exist"
Jul 01 19:32:53.538583 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:53.538546    2574 scope.go:117] "RemoveContainer" containerID="9cdd6a01d4963767f184bffc68a34cbad2d599045d46d2c6690ed61355460677"
Jul 01 19:32:53.538814 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:32:53.538795    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cdd6a01d4963767f184bffc68a34cbad2d599045d46d2c6690ed61355460677\": container with ID starting with 9cdd6a01d4963767f184bffc68a34cbad2d599045d46d2c6690ed61355460677 not found: ID does not exist" containerID="9cdd6a01d4963767f184bffc68a34cbad2d599045d46d2c6690ed61355460677"
Jul 01 19:32:53.538853 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:53.538821    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cdd6a01d4963767f184bffc68a34cbad2d599045d46d2c6690ed61355460677"} err="failed to get container status \"9cdd6a01d4963767f184bffc68a34cbad2d599045d46d2c6690ed61355460677\": rpc error: code = NotFound desc = could not find container \"9cdd6a01d4963767f184bffc68a34cbad2d599045d46d2c6690ed61355460677\": container with ID starting with 9cdd6a01d4963767f184bffc68a34cbad2d599045d46d2c6690ed61355460677 not found: ID does not exist"
Jul 01 19:32:54.359050 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:54.359011    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" path="/var/lib/kubelet/pods/e27e002a-4ea3-44f6-b1c1-6ac637649400/volumes"
Jul 01 19:32:54.511665 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:54.511627    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc" podUID="e1ca2705-0c5d-45cb-9f54-5f82238eabec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 01 19:32:55.384178 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:55.384142    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j" podUID="835093bf-c5c0-4835-96c5-bd03e5097839" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 01 19:32:59.516089 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:59.516057    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc"
Jul 01 19:32:59.516548 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:32:59.516516    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc" podUID="e1ca2705-0c5d-45cb-9f54-5f82238eabec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 01 19:33:05.383680 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:33:05.383642    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j" podUID="835093bf-c5c0-4835-96c5-bd03e5097839" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 01 19:33:09.516876 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:33:09.516839    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc" podUID="e1ca2705-0c5d-45cb-9f54-5f82238eabec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 01 19:33:15.383824 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:33:15.383783    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j" podUID="835093bf-c5c0-4835-96c5-bd03e5097839" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 01 19:33:19.516561 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:33:19.516519    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc" podUID="e1ca2705-0c5d-45cb-9f54-5f82238eabec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 01 19:33:25.384489 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:33:25.384448    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j" podUID="835093bf-c5c0-4835-96c5-bd03e5097839" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 01 19:33:29.516826 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:33:29.516788    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc" podUID="e1ca2705-0c5d-45cb-9f54-5f82238eabec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 01 19:33:35.383609 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:33:35.383566    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j" podUID="835093bf-c5c0-4835-96c5-bd03e5097839" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 01 19:33:39.516703 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:33:39.516665    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc" podUID="e1ca2705-0c5d-45cb-9f54-5f82238eabec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 01 19:33:45.384485 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:33:45.384449    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j"
Jul 01 19:33:49.516500 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:33:49.516453    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc" podUID="e1ca2705-0c5d-45cb-9f54-5f82238eabec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 01 19:33:59.518021 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:33:59.517993    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc"
Jul 01 19:34:12.976701 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:12.976659    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx"]
Jul 01 19:34:12.977213 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:12.977076    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="kube-rbac-proxy"
Jul 01 19:34:12.977213 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:12.977093    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="kube-rbac-proxy"
Jul 01 19:34:12.977213 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:12.977119    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="agent"
Jul 01 19:34:12.977213 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:12.977125    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="agent"
Jul 01 19:34:12.977213 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:12.977136    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="storage-initializer"
Jul 01 19:34:12.977213 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:12.977143    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="storage-initializer"
Jul 01 19:34:12.977213 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:12.977148    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="kserve-container"
Jul 01 19:34:12.977213 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:12.977154    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="kserve-container"
Jul 01 19:34:12.977213 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:12.977212    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="kserve-container"
Jul 01 19:34:12.977612 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:12.977220    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="kube-rbac-proxy"
Jul 01 19:34:12.977612 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:12.977229    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e27e002a-4ea3-44f6-b1c1-6ac637649400" containerName="agent"
Jul 01 19:34:12.981104 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:12.981085    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx"
Jul 01 19:34:12.983501 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:12.983479    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-6e49a-predictor-serving-cert\""
Jul 01 19:34:12.983717 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:12.983701    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-6e49a-kube-rbac-proxy-sar-config\""
Jul 01 19:34:12.993462 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:12.993435    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx"]
Jul 01 19:34:13.015843 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.015810    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j"]
Jul 01 19:34:13.016197 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.016143    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j" podUID="835093bf-c5c0-4835-96c5-bd03e5097839" containerName="kserve-container" containerID="cri-o://2bd27b81b1402cc33a1ba8ce10d6accec4545b46a5603c31ccd8a7e9acf1b9de" gracePeriod=30
Jul 01 19:34:13.016341 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.016218    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j" podUID="835093bf-c5c0-4835-96c5-bd03e5097839" containerName="kube-rbac-proxy" containerID="cri-o://3fa3e1fc96205099e91b7c7aa0df7abe266409deb40f7d23e5178068b513a599" gracePeriod=30
Jul 01 19:34:13.073874 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.073837    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d770e95-639f-423c-8584-8ec945730bd8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx\" (UID: \"4d770e95-639f-423c-8584-8ec945730bd8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx"
Jul 01 19:34:13.074061 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.073893    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjpqh\" (UniqueName: \"kubernetes.io/projected/4d770e95-639f-423c-8584-8ec945730bd8-kube-api-access-pjpqh\") pod \"isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx\" (UID: \"4d770e95-639f-423c-8584-8ec945730bd8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx"
Jul 01 19:34:13.074061 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.073973    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d770e95-639f-423c-8584-8ec945730bd8-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx\" (UID: \"4d770e95-639f-423c-8584-8ec945730bd8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx"
Jul 01 19:34:13.074155 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.074069    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d770e95-639f-423c-8584-8ec945730bd8-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx\" (UID: \"4d770e95-639f-423c-8584-8ec945730bd8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx"
Jul 01 19:34:13.074155 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.074110    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d770e95-639f-423c-8584-8ec945730bd8-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx\" (UID: \"4d770e95-639f-423c-8584-8ec945730bd8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx"
Jul 01 19:34:13.074550 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.074531    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc"]
Jul 01 19:34:13.078223 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.078207    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc"
Jul 01 19:34:13.080289 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.080265    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-6e49a-predictor-serving-cert\""
Jul 01 19:34:13.080632 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.080603    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-6e49a-kube-rbac-proxy-sar-config\""
Jul 01 19:34:13.093449 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.093424    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc"]
Jul 01 19:34:13.126669 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.126637    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc"]
Jul 01 19:34:13.126924 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.126876    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc" podUID="e1ca2705-0c5d-45cb-9f54-5f82238eabec" containerName="kserve-container" containerID="cri-o://fe8c1ccf408600e0925ef9f67274321f4f72020a4d91cca0b7153c6661ef3648" gracePeriod=30
Jul 01 19:34:13.127000 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.126948    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc" podUID="e1ca2705-0c5d-45cb-9f54-5f82238eabec" containerName="kube-rbac-proxy" containerID="cri-o://af0e88b541ec49f1a7bdc49d5014c3f766f5d385ab58883e3de261bf0c71cb79" gracePeriod=30
Jul 01 19:34:13.174780 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.174747    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d770e95-639f-423c-8584-8ec945730bd8-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx\" (UID: \"4d770e95-639f-423c-8584-8ec945730bd8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx"
Jul 01 19:34:13.174995 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.174788    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d770e95-639f-423c-8584-8ec945730bd8-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx\" (UID: \"4d770e95-639f-423c-8584-8ec945730bd8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx"
Jul 01 19:34:13.174995 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.174827    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9ff26bb-9658-4d8b-b10a-eddf3932c62c-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc\" (UID: \"d9ff26bb-9658-4d8b-b10a-eddf3932c62c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc"
Jul 01 19:34:13.174995 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.174867    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d770e95-639f-423c-8584-8ec945730bd8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx\" (UID: \"4d770e95-639f-423c-8584-8ec945730bd8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx"
Jul 01 19:34:13.175230 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.175203    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjpqh\" (UniqueName: \"kubernetes.io/projected/4d770e95-639f-423c-8584-8ec945730bd8-kube-api-access-pjpqh\") pod \"isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx\" (UID: \"4d770e95-639f-423c-8584-8ec945730bd8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx"
Jul 01 19:34:13.175312 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.175256    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49m68\" (UniqueName: \"kubernetes.io/projected/d9ff26bb-9658-4d8b-b10a-eddf3932c62c-kube-api-access-49m68\") pod \"isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc\" (UID: \"d9ff26bb-9658-4d8b-b10a-eddf3932c62c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc"
Jul 01 19:34:13.175373 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.175309    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9ff26bb-9658-4d8b-b10a-eddf3932c62c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc\" (UID: \"d9ff26bb-9658-4d8b-b10a-eddf3932c62c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc"
Jul 01 19:34:13.175373 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.175342    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9ff26bb-9658-4d8b-b10a-eddf3932c62c-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc\" (UID: \"d9ff26bb-9658-4d8b-b10a-eddf3932c62c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc"
Jul 01 19:34:13.175467 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.175374    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d770e95-639f-423c-8584-8ec945730bd8-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx\" (UID: \"4d770e95-639f-423c-8584-8ec945730bd8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx"
Jul 01 19:34:13.175467 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.175401    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9ff26bb-9658-4d8b-b10a-eddf3932c62c-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc\" (UID: \"d9ff26bb-9658-4d8b-b10a-eddf3932c62c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc"
Jul 01 19:34:13.175586 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:34:13.175528    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-serving-cert: secret "isvc-sklearn-graph-raw-hpa-6e49a-predictor-serving-cert" not found
Jul 01 19:34:13.175644 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:34:13.175592    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d770e95-639f-423c-8584-8ec945730bd8-proxy-tls podName:4d770e95-639f-423c-8584-8ec945730bd8 nodeName:}" failed. No retries permitted until 2026-07-01 19:34:13.675576666 +0000 UTC m=+759.767576097 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4d770e95-639f-423c-8584-8ec945730bd8-proxy-tls") pod "isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx" (UID: "4d770e95-639f-423c-8584-8ec945730bd8") : secret "isvc-sklearn-graph-raw-hpa-6e49a-predictor-serving-cert" not found
Jul 01 19:34:13.175712 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.175673    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d770e95-639f-423c-8584-8ec945730bd8-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx\" (UID: \"4d770e95-639f-423c-8584-8ec945730bd8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx"
Jul 01 19:34:13.176077 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.176056    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d770e95-639f-423c-8584-8ec945730bd8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx\" (UID: \"4d770e95-639f-423c-8584-8ec945730bd8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx"
Jul 01 19:34:13.176271 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.176250    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d770e95-639f-423c-8584-8ec945730bd8-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx\" (UID: \"4d770e95-639f-423c-8584-8ec945730bd8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx"
Jul 01 19:34:13.185980 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.185957    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjpqh\" (UniqueName: \"kubernetes.io/projected/4d770e95-639f-423c-8584-8ec945730bd8-kube-api-access-pjpqh\") pod \"isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx\" (UID: \"4d770e95-639f-423c-8584-8ec945730bd8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx"
Jul 01 19:34:13.276300 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.276269    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49m68\" (UniqueName: \"kubernetes.io/projected/d9ff26bb-9658-4d8b-b10a-eddf3932c62c-kube-api-access-49m68\") pod \"isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc\" (UID: \"d9ff26bb-9658-4d8b-b10a-eddf3932c62c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc"
Jul 01 19:34:13.276434 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.276313    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9ff26bb-9658-4d8b-b10a-eddf3932c62c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc\" (UID: \"d9ff26bb-9658-4d8b-b10a-eddf3932c62c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc"
Jul 01 19:34:13.276434 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.276337    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9ff26bb-9658-4d8b-b10a-eddf3932c62c-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc\" (UID: \"d9ff26bb-9658-4d8b-b10a-eddf3932c62c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc"
Jul 01 19:34:13.276434 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.276367    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9ff26bb-9658-4d8b-b10a-eddf3932c62c-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc\" (UID: \"d9ff26bb-9658-4d8b-b10a-eddf3932c62c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc"
Jul 01 19:34:13.276603 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.276443    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9ff26bb-9658-4d8b-b10a-eddf3932c62c-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc\" (UID: \"d9ff26bb-9658-4d8b-b10a-eddf3932c62c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc"
Jul 01 19:34:13.276826 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.276804    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9ff26bb-9658-4d8b-b10a-eddf3932c62c-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc\" (UID: \"d9ff26bb-9658-4d8b-b10a-eddf3932c62c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc"
Jul 01 19:34:13.277083 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.277062    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9ff26bb-9658-4d8b-b10a-eddf3932c62c-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc\" (UID: \"d9ff26bb-9658-4d8b-b10a-eddf3932c62c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc"
Jul 01 19:34:13.277083 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.277075    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9ff26bb-9658-4d8b-b10a-eddf3932c62c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc\" (UID: \"d9ff26bb-9658-4d8b-b10a-eddf3932c62c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc"
Jul 01 19:34:13.278632 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.278615    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9ff26bb-9658-4d8b-b10a-eddf3932c62c-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc\" (UID: \"d9ff26bb-9658-4d8b-b10a-eddf3932c62c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc"
Jul 01 19:34:13.286700 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.286676    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49m68\" (UniqueName: \"kubernetes.io/projected/d9ff26bb-9658-4d8b-b10a-eddf3932c62c-kube-api-access-49m68\") pod \"isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc\" (UID: \"d9ff26bb-9658-4d8b-b10a-eddf3932c62c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc"
Jul 01 19:34:13.390720 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.390692    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc"
Jul 01 19:34:13.522184 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.522160    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc"]
Jul 01 19:34:13.524464 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:34:13.524442    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9ff26bb_9658_4d8b_b10a_eddf3932c62c.slice/crio-cb0e9a712a8f4311e4f7b9ffaf0cc1e1c1699a5e9e967c32de5233e1f79a30fa WatchSource:0}: Error finding container cb0e9a712a8f4311e4f7b9ffaf0cc1e1c1699a5e9e967c32de5233e1f79a30fa: Status 404 returned error can't find the container with id cb0e9a712a8f4311e4f7b9ffaf0cc1e1c1699a5e9e967c32de5233e1f79a30fa
Jul 01 19:34:13.681088 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.681051    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d770e95-639f-423c-8584-8ec945730bd8-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx\" (UID: \"4d770e95-639f-423c-8584-8ec945730bd8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx"
Jul 01 19:34:13.683361 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.683334    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d770e95-639f-423c-8584-8ec945730bd8-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx\" (UID: \"4d770e95-639f-423c-8584-8ec945730bd8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx"
Jul 01 19:34:13.874773 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.874739    2574 generic.go:356] "Generic (PLEG): container finished" podID="e1ca2705-0c5d-45cb-9f54-5f82238eabec" containerID="af0e88b541ec49f1a7bdc49d5014c3f766f5d385ab58883e3de261bf0c71cb79" exitCode=2
Jul 01 19:34:13.874996 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.874803    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc" event={"ID":"e1ca2705-0c5d-45cb-9f54-5f82238eabec","Type":"ContainerDied","Data":"af0e88b541ec49f1a7bdc49d5014c3f766f5d385ab58883e3de261bf0c71cb79"}
Jul 01 19:34:13.876300 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.876277    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc" event={"ID":"d9ff26bb-9658-4d8b-b10a-eddf3932c62c","Type":"ContainerStarted","Data":"feca1ce156cc2c8977e238f83b59c93d5d01be778e9278948f932d3db1dc4faa"}
Jul 01 19:34:13.876420 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.876332    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc" event={"ID":"d9ff26bb-9658-4d8b-b10a-eddf3932c62c","Type":"ContainerStarted","Data":"cb0e9a712a8f4311e4f7b9ffaf0cc1e1c1699a5e9e967c32de5233e1f79a30fa"}
Jul 01 19:34:13.878318 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.878299    2574 generic.go:356] "Generic (PLEG): container finished" podID="835093bf-c5c0-4835-96c5-bd03e5097839" containerID="3fa3e1fc96205099e91b7c7aa0df7abe266409deb40f7d23e5178068b513a599" exitCode=2
Jul 01 19:34:13.878420 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.878346    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j" event={"ID":"835093bf-c5c0-4835-96c5-bd03e5097839","Type":"ContainerDied","Data":"3fa3e1fc96205099e91b7c7aa0df7abe266409deb40f7d23e5178068b513a599"}
Jul 01 19:34:13.891927 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:13.891885    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx"
Jul 01 19:34:14.028537 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:14.028511    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx"]
Jul 01 19:34:14.030819 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:34:14.030795    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d770e95_639f_423c_8584_8ec945730bd8.slice/crio-ccd2b38d87fb79ba5a3bc6cc37abc5a054fbe62435be885a418e2bc26cbee4c2 WatchSource:0}: Error finding container ccd2b38d87fb79ba5a3bc6cc37abc5a054fbe62435be885a418e2bc26cbee4c2: Status 404 returned error can't find the container with id ccd2b38d87fb79ba5a3bc6cc37abc5a054fbe62435be885a418e2bc26cbee4c2
Jul 01 19:34:14.511991 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:14.511945    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc" podUID="e1ca2705-0c5d-45cb-9f54-5f82238eabec" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 01 19:34:14.885391 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:14.885349    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx" event={"ID":"4d770e95-639f-423c-8584-8ec945730bd8","Type":"ContainerStarted","Data":"86b3a37eff22993260d050b41737673feee701fe79fc595aad72d56af7048642"}
Jul 01 19:34:14.885561 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:14.885427    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx" event={"ID":"4d770e95-639f-423c-8584-8ec945730bd8","Type":"ContainerStarted","Data":"ccd2b38d87fb79ba5a3bc6cc37abc5a054fbe62435be885a418e2bc26cbee4c2"}
Jul 01 19:34:15.378268 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:15.378224    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j" podUID="835093bf-c5c0-4835-96c5-bd03e5097839" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 01 19:34:15.384480 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:15.384453    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j" podUID="835093bf-c5c0-4835-96c5-bd03e5097839" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 01 19:34:16.898009 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:16.897980    2574 generic.go:356] "Generic (PLEG): container finished" podID="e1ca2705-0c5d-45cb-9f54-5f82238eabec" containerID="fe8c1ccf408600e0925ef9f67274321f4f72020a4d91cca0b7153c6661ef3648" exitCode=0
Jul 01 19:34:16.898474 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:16.898021    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc" event={"ID":"e1ca2705-0c5d-45cb-9f54-5f82238eabec","Type":"ContainerDied","Data":"fe8c1ccf408600e0925ef9f67274321f4f72020a4d91cca0b7153c6661ef3648"}
Jul 01 19:34:16.981232 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:16.981210    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc"
Jul 01 19:34:17.010986 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.010921    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1ca2705-0c5d-45cb-9f54-5f82238eabec-kserve-provision-location\") pod \"e1ca2705-0c5d-45cb-9f54-5f82238eabec\" (UID: \"e1ca2705-0c5d-45cb-9f54-5f82238eabec\") "
Jul 01 19:34:17.011092 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.011010    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1ca2705-0c5d-45cb-9f54-5f82238eabec-proxy-tls\") pod \"e1ca2705-0c5d-45cb-9f54-5f82238eabec\" (UID: \"e1ca2705-0c5d-45cb-9f54-5f82238eabec\") "
Jul 01 19:34:17.011092 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.011048    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1ca2705-0c5d-45cb-9f54-5f82238eabec-kube-rbac-proxy-sar-config\") pod \"e1ca2705-0c5d-45cb-9f54-5f82238eabec\" (UID: \"e1ca2705-0c5d-45cb-9f54-5f82238eabec\") "
Jul 01 19:34:17.011092 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.011067    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1ca2705-0c5d-45cb-9f54-5f82238eabec-cabundle-cert\") pod \"e1ca2705-0c5d-45cb-9f54-5f82238eabec\" (UID: \"e1ca2705-0c5d-45cb-9f54-5f82238eabec\") "
Jul 01 19:34:17.011231 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.011128    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqq55\" (UniqueName: \"kubernetes.io/projected/e1ca2705-0c5d-45cb-9f54-5f82238eabec-kube-api-access-zqq55\") pod \"e1ca2705-0c5d-45cb-9f54-5f82238eabec\" (UID: \"e1ca2705-0c5d-45cb-9f54-5f82238eabec\") "
Jul 01 19:34:17.011331 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.011302    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1ca2705-0c5d-45cb-9f54-5f82238eabec-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e1ca2705-0c5d-45cb-9f54-5f82238eabec" (UID: "e1ca2705-0c5d-45cb-9f54-5f82238eabec"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:34:17.011422 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.011371    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1ca2705-0c5d-45cb-9f54-5f82238eabec-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e1ca2705-0c5d-45cb-9f54-5f82238eabec" (UID: "e1ca2705-0c5d-45cb-9f54-5f82238eabec"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:34:17.011508 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.011488    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1ca2705-0c5d-45cb-9f54-5f82238eabec-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e1ca2705-0c5d-45cb-9f54-5f82238eabec" (UID: "e1ca2705-0c5d-45cb-9f54-5f82238eabec"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:34:17.013147 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.013117    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1ca2705-0c5d-45cb-9f54-5f82238eabec-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e1ca2705-0c5d-45cb-9f54-5f82238eabec" (UID: "e1ca2705-0c5d-45cb-9f54-5f82238eabec"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:34:17.013147 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.013124    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1ca2705-0c5d-45cb-9f54-5f82238eabec-kube-api-access-zqq55" (OuterVolumeSpecName: "kube-api-access-zqq55") pod "e1ca2705-0c5d-45cb-9f54-5f82238eabec" (UID: "e1ca2705-0c5d-45cb-9f54-5f82238eabec"). InnerVolumeSpecName "kube-api-access-zqq55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:34:17.112422 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.112385    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1ca2705-0c5d-45cb-9f54-5f82238eabec-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:34:17.112422 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.112410    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1ca2705-0c5d-45cb-9f54-5f82238eabec-cabundle-cert\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:34:17.112422 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.112419    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqq55\" (UniqueName: \"kubernetes.io/projected/e1ca2705-0c5d-45cb-9f54-5f82238eabec-kube-api-access-zqq55\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:34:17.112422 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.112429    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1ca2705-0c5d-45cb-9f54-5f82238eabec-kserve-provision-location\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:34:17.112664 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.112437    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1ca2705-0c5d-45cb-9f54-5f82238eabec-proxy-tls\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:34:17.860019 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.859995    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j"
Jul 01 19:34:17.905484 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.905449    2574 generic.go:356] "Generic (PLEG): container finished" podID="4d770e95-639f-423c-8584-8ec945730bd8" containerID="86b3a37eff22993260d050b41737673feee701fe79fc595aad72d56af7048642" exitCode=0
Jul 01 19:34:17.905855 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.905532    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx" event={"ID":"4d770e95-639f-423c-8584-8ec945730bd8","Type":"ContainerDied","Data":"86b3a37eff22993260d050b41737673feee701fe79fc595aad72d56af7048642"}
Jul 01 19:34:17.907571 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.907548    2574 generic.go:356] "Generic (PLEG): container finished" podID="835093bf-c5c0-4835-96c5-bd03e5097839" containerID="2bd27b81b1402cc33a1ba8ce10d6accec4545b46a5603c31ccd8a7e9acf1b9de" exitCode=0
Jul 01 19:34:17.907673 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.907597    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j"
Jul 01 19:34:17.907673 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.907641    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j" event={"ID":"835093bf-c5c0-4835-96c5-bd03e5097839","Type":"ContainerDied","Data":"2bd27b81b1402cc33a1ba8ce10d6accec4545b46a5603c31ccd8a7e9acf1b9de"}
Jul 01 19:34:17.907770 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.907698    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j" event={"ID":"835093bf-c5c0-4835-96c5-bd03e5097839","Type":"ContainerDied","Data":"edbc676907a93baf81c94da9de1ffe05d115178da00affcfe596f55954915c4e"}
Jul 01 19:34:17.907770 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.907763    2574 scope.go:117] "RemoveContainer" containerID="3fa3e1fc96205099e91b7c7aa0df7abe266409deb40f7d23e5178068b513a599"
Jul 01 19:34:17.909955 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.909934    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc" event={"ID":"e1ca2705-0c5d-45cb-9f54-5f82238eabec","Type":"ContainerDied","Data":"bbf83234cd6c328206cef0143594e59b5bf92446b070ee537aa9a79898e4dd35"}
Jul 01 19:34:17.910081 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.909975    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc"
Jul 01 19:34:17.911735 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.911674    2574 generic.go:356] "Generic (PLEG): container finished" podID="d9ff26bb-9658-4d8b-b10a-eddf3932c62c" containerID="feca1ce156cc2c8977e238f83b59c93d5d01be778e9278948f932d3db1dc4faa" exitCode=0
Jul 01 19:34:17.911735 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.911722    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc" event={"ID":"d9ff26bb-9658-4d8b-b10a-eddf3932c62c","Type":"ContainerDied","Data":"feca1ce156cc2c8977e238f83b59c93d5d01be778e9278948f932d3db1dc4faa"}
Jul 01 19:34:17.919486 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.919468    2574 scope.go:117] "RemoveContainer" containerID="2bd27b81b1402cc33a1ba8ce10d6accec4545b46a5603c31ccd8a7e9acf1b9de"
Jul 01 19:34:17.919753 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.919734    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/835093bf-c5c0-4835-96c5-bd03e5097839-proxy-tls\") pod \"835093bf-c5c0-4835-96c5-bd03e5097839\" (UID: \"835093bf-c5c0-4835-96c5-bd03e5097839\") "
Jul 01 19:34:17.919795 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.919777    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/835093bf-c5c0-4835-96c5-bd03e5097839-kserve-provision-location\") pod \"835093bf-c5c0-4835-96c5-bd03e5097839\" (UID: \"835093bf-c5c0-4835-96c5-bd03e5097839\") "
Jul 01 19:34:17.919826 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.919813    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/835093bf-c5c0-4835-96c5-bd03e5097839-cabundle-cert\") pod \"835093bf-c5c0-4835-96c5-bd03e5097839\" (UID: \"835093bf-c5c0-4835-96c5-bd03e5097839\") "
Jul 01 19:34:17.919866 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.919843    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/835093bf-c5c0-4835-96c5-bd03e5097839-kube-rbac-proxy-sar-config\") pod \"835093bf-c5c0-4835-96c5-bd03e5097839\" (UID: \"835093bf-c5c0-4835-96c5-bd03e5097839\") "
Jul 01 19:34:17.919944 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.919916    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cvcs\" (UniqueName: \"kubernetes.io/projected/835093bf-c5c0-4835-96c5-bd03e5097839-kube-api-access-6cvcs\") pod \"835093bf-c5c0-4835-96c5-bd03e5097839\" (UID: \"835093bf-c5c0-4835-96c5-bd03e5097839\") "
Jul 01 19:34:17.920220 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.920193    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835093bf-c5c0-4835-96c5-bd03e5097839-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "835093bf-c5c0-4835-96c5-bd03e5097839" (UID: "835093bf-c5c0-4835-96c5-bd03e5097839"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:34:17.920322 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.920217    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/835093bf-c5c0-4835-96c5-bd03e5097839-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "835093bf-c5c0-4835-96c5-bd03e5097839" (UID: "835093bf-c5c0-4835-96c5-bd03e5097839"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:34:17.920322 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.920261    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/835093bf-c5c0-4835-96c5-bd03e5097839-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "835093bf-c5c0-4835-96c5-bd03e5097839" (UID: "835093bf-c5c0-4835-96c5-bd03e5097839"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:34:17.920322 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.920284    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/835093bf-c5c0-4835-96c5-bd03e5097839-kserve-provision-location\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:34:17.920322 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.920303    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/835093bf-c5c0-4835-96c5-bd03e5097839-cabundle-cert\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:34:17.921974 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.921947    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/835093bf-c5c0-4835-96c5-bd03e5097839-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "835093bf-c5c0-4835-96c5-bd03e5097839" (UID: "835093bf-c5c0-4835-96c5-bd03e5097839"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:34:17.922629 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.922509    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/835093bf-c5c0-4835-96c5-bd03e5097839-kube-api-access-6cvcs" (OuterVolumeSpecName: "kube-api-access-6cvcs") pod "835093bf-c5c0-4835-96c5-bd03e5097839" (UID: "835093bf-c5c0-4835-96c5-bd03e5097839"). InnerVolumeSpecName "kube-api-access-6cvcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:34:17.935311 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.935287    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc"]
Jul 01 19:34:17.939618 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.939592    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-66e42-predictor-5b8c4d868f-dv8kc"]
Jul 01 19:34:17.940629 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.940615    2574 scope.go:117] "RemoveContainer" containerID="f3e8cd9f9358ead8597bc5643888871b2a35b166bbbd8c6a179c1ee8872fa1e8"
Jul 01 19:34:17.962130 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.962105    2574 scope.go:117] "RemoveContainer" containerID="3fa3e1fc96205099e91b7c7aa0df7abe266409deb40f7d23e5178068b513a599"
Jul 01 19:34:17.962438 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:34:17.962409    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fa3e1fc96205099e91b7c7aa0df7abe266409deb40f7d23e5178068b513a599\": container with ID starting with 3fa3e1fc96205099e91b7c7aa0df7abe266409deb40f7d23e5178068b513a599 not found: ID does not exist" containerID="3fa3e1fc96205099e91b7c7aa0df7abe266409deb40f7d23e5178068b513a599"
Jul 01 19:34:17.962513 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.962448    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fa3e1fc96205099e91b7c7aa0df7abe266409deb40f7d23e5178068b513a599"} err="failed to get container status \"3fa3e1fc96205099e91b7c7aa0df7abe266409deb40f7d23e5178068b513a599\": rpc error: code = NotFound desc = could not find container \"3fa3e1fc96205099e91b7c7aa0df7abe266409deb40f7d23e5178068b513a599\": container with ID starting with 3fa3e1fc96205099e91b7c7aa0df7abe266409deb40f7d23e5178068b513a599 not found: ID does not exist"
Jul 01 19:34:17.962513 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.962475    2574 scope.go:117] "RemoveContainer" containerID="2bd27b81b1402cc33a1ba8ce10d6accec4545b46a5603c31ccd8a7e9acf1b9de"
Jul 01 19:34:17.962749 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:34:17.962731    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bd27b81b1402cc33a1ba8ce10d6accec4545b46a5603c31ccd8a7e9acf1b9de\": container with ID starting with 2bd27b81b1402cc33a1ba8ce10d6accec4545b46a5603c31ccd8a7e9acf1b9de not found: ID does not exist" containerID="2bd27b81b1402cc33a1ba8ce10d6accec4545b46a5603c31ccd8a7e9acf1b9de"
Jul 01 19:34:17.962792 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.962758    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bd27b81b1402cc33a1ba8ce10d6accec4545b46a5603c31ccd8a7e9acf1b9de"} err="failed to get container status \"2bd27b81b1402cc33a1ba8ce10d6accec4545b46a5603c31ccd8a7e9acf1b9de\": rpc error: code = NotFound desc = could not find container \"2bd27b81b1402cc33a1ba8ce10d6accec4545b46a5603c31ccd8a7e9acf1b9de\": container with ID starting with 2bd27b81b1402cc33a1ba8ce10d6accec4545b46a5603c31ccd8a7e9acf1b9de not found: ID does not exist"
Jul 01 19:34:17.962792 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.962773    2574 scope.go:117] "RemoveContainer" containerID="f3e8cd9f9358ead8597bc5643888871b2a35b166bbbd8c6a179c1ee8872fa1e8"
Jul 01 19:34:17.963042 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:34:17.963024    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3e8cd9f9358ead8597bc5643888871b2a35b166bbbd8c6a179c1ee8872fa1e8\": container with ID starting with f3e8cd9f9358ead8597bc5643888871b2a35b166bbbd8c6a179c1ee8872fa1e8 not found: ID does not exist" containerID="f3e8cd9f9358ead8597bc5643888871b2a35b166bbbd8c6a179c1ee8872fa1e8"
Jul 01 19:34:17.963093 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.963045    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3e8cd9f9358ead8597bc5643888871b2a35b166bbbd8c6a179c1ee8872fa1e8"} err="failed to get container status \"f3e8cd9f9358ead8597bc5643888871b2a35b166bbbd8c6a179c1ee8872fa1e8\": rpc error: code = NotFound desc = could not find container \"f3e8cd9f9358ead8597bc5643888871b2a35b166bbbd8c6a179c1ee8872fa1e8\": container with ID starting with f3e8cd9f9358ead8597bc5643888871b2a35b166bbbd8c6a179c1ee8872fa1e8 not found: ID does not exist"
Jul 01 19:34:17.963093 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.963058    2574 scope.go:117] "RemoveContainer" containerID="af0e88b541ec49f1a7bdc49d5014c3f766f5d385ab58883e3de261bf0c71cb79"
Jul 01 19:34:17.971789 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.971772    2574 scope.go:117] "RemoveContainer" containerID="fe8c1ccf408600e0925ef9f67274321f4f72020a4d91cca0b7153c6661ef3648"
Jul 01 19:34:17.979230 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:17.979210    2574 scope.go:117] "RemoveContainer" containerID="3c31a052c7effa3a6eee14eaab1d2df723838dbe7e93fd9b52700202fade41eb"
Jul 01 19:34:18.021710 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:18.021686    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/835093bf-c5c0-4835-96c5-bd03e5097839-proxy-tls\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:34:18.021822 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:18.021713    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/835093bf-c5c0-4835-96c5-bd03e5097839-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:34:18.021822 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:18.021727    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cvcs\" (UniqueName: \"kubernetes.io/projected/835093bf-c5c0-4835-96c5-bd03e5097839-kube-api-access-6cvcs\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:34:18.228877 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:18.228848    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j"]
Jul 01 19:34:18.232581 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:18.232559    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-66e42-predictor-6b5dc65cc5-kcg6j"]
Jul 01 19:34:18.358929 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:18.358883    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="835093bf-c5c0-4835-96c5-bd03e5097839" path="/var/lib/kubelet/pods/835093bf-c5c0-4835-96c5-bd03e5097839/volumes"
Jul 01 19:34:18.359410 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:18.359396    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1ca2705-0c5d-45cb-9f54-5f82238eabec" path="/var/lib/kubelet/pods/e1ca2705-0c5d-45cb-9f54-5f82238eabec/volumes"
Jul 01 19:34:18.918196 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:18.918160    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx" event={"ID":"4d770e95-639f-423c-8584-8ec945730bd8","Type":"ContainerStarted","Data":"b05b67849f48f32a74f0113fed0c4f3273cb05330d08bbc3f5bd822c47372586"}
Jul 01 19:34:18.918629 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:18.918276    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx" event={"ID":"4d770e95-639f-423c-8584-8ec945730bd8","Type":"ContainerStarted","Data":"4aef491a37a2fd92cb55d8afaeb4098a839b309339c66776b947e705a0022007"}
Jul 01 19:34:18.918629 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:18.918545    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx"
Jul 01 19:34:18.918768 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:18.918747    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx"
Jul 01 19:34:18.920019 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:18.919989    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx" podUID="4d770e95-639f-423c-8584-8ec945730bd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 19:34:18.922318 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:18.922290    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc" event={"ID":"d9ff26bb-9658-4d8b-b10a-eddf3932c62c","Type":"ContainerStarted","Data":"89c56127dc9aecf30dbfbc3f47469d7bed912794bdb9c5577cb23298b2fc7c16"}
Jul 01 19:34:18.922406 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:18.922348    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc" event={"ID":"d9ff26bb-9658-4d8b-b10a-eddf3932c62c","Type":"ContainerStarted","Data":"84b614285eb84d913d63df7b5378ac2183ed8d9e2e05be3301cc3c44d0a59729"}
Jul 01 19:34:18.922637 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:18.922616    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc"
Jul 01 19:34:18.922758 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:18.922663    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc"
Jul 01 19:34:18.923732 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:18.923712    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc" podUID="d9ff26bb-9658-4d8b-b10a-eddf3932c62c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 01 19:34:18.938857 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:18.938809    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx" podStartSLOduration=3.525854572 podStartE2EDuration="6.938792526s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.412937954s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:34:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:34:18.937052978 +0000 UTC m=+765.029052426" watchObservedRunningTime="2026-07-01 19:34:18.938792526 +0000 UTC m=+765.030791975"
Jul 01 19:34:18.958225 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:18.958182    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc" podStartSLOduration=2.242023696 podStartE2EDuration="5.958169699s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.716146003s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:34:18.956848861 +0000 UTC m=+765.048848310" watchObservedRunningTime="2026-07-01 19:34:18.958169699 +0000 UTC m=+765.050169146"
Jul 01 19:34:19.927044 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:19.926996    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc" podUID="d9ff26bb-9658-4d8b-b10a-eddf3932c62c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 01 19:34:19.927453 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:19.927112    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx" podUID="4d770e95-639f-423c-8584-8ec945730bd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 19:34:24.931175 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:24.931145    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc"
Jul 01 19:34:24.931645 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:24.931611    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx"
Jul 01 19:34:24.931707 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:24.931662    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc" podUID="d9ff26bb-9658-4d8b-b10a-eddf3932c62c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 01 19:34:24.932085 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:24.932061    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx" podUID="4d770e95-639f-423c-8584-8ec945730bd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 19:34:34.931716 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:34.931675    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc" podUID="d9ff26bb-9658-4d8b-b10a-eddf3932c62c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 01 19:34:34.932189 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:34.932156    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx" podUID="4d770e95-639f-423c-8584-8ec945730bd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 19:34:44.931695 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:44.931647    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc" podUID="d9ff26bb-9658-4d8b-b10a-eddf3932c62c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 01 19:34:44.932215 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:44.932137    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx" podUID="4d770e95-639f-423c-8584-8ec945730bd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 19:34:54.931767 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:54.931721    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc" podUID="d9ff26bb-9658-4d8b-b10a-eddf3932c62c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 01 19:34:54.932321 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:34:54.932077    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx" podUID="4d770e95-639f-423c-8584-8ec945730bd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 19:35:04.932552 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:35:04.932458    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx" podUID="4d770e95-639f-423c-8584-8ec945730bd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 19:35:04.932552 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:35:04.932479    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc" podUID="d9ff26bb-9658-4d8b-b10a-eddf3932c62c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 01 19:35:14.932453 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:35:14.932418    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc" podUID="d9ff26bb-9658-4d8b-b10a-eddf3932c62c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 01 19:35:14.932929 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:35:14.932419    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx" podUID="4d770e95-639f-423c-8584-8ec945730bd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 19:35:24.932447 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:35:24.932406    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx" podUID="4d770e95-639f-423c-8584-8ec945730bd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 19:35:24.933062 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:35:24.933027    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc"
Jul 01 19:35:34.932647 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:35:34.932621    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx"
Jul 01 19:36:03.264986 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:03.264949    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-553af-predictor-fdf784bb4-h4pqq"]
Jul 01 19:36:03.265443 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:03.265367    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1ca2705-0c5d-45cb-9f54-5f82238eabec" containerName="storage-initializer"
Jul 01 19:36:03.265443 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:03.265379    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1ca2705-0c5d-45cb-9f54-5f82238eabec" containerName="storage-initializer"
Jul 01 19:36:03.265443 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:03.265392    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="835093bf-c5c0-4835-96c5-bd03e5097839" containerName="kserve-container"
Jul 01 19:36:03.265443 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:03.265398    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="835093bf-c5c0-4835-96c5-bd03e5097839" containerName="kserve-container"
Jul 01 19:36:03.265443 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:03.265408    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1ca2705-0c5d-45cb-9f54-5f82238eabec" containerName="kube-rbac-proxy"
Jul 01 19:36:03.265443 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:03.265417    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1ca2705-0c5d-45cb-9f54-5f82238eabec" containerName="kube-rbac-proxy"
Jul 01 19:36:03.265443 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:03.265427    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="835093bf-c5c0-4835-96c5-bd03e5097839" containerName="storage-initializer"
Jul 01 19:36:03.265443 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:03.265432    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="835093bf-c5c0-4835-96c5-bd03e5097839" containerName="storage-initializer"
Jul 01 19:36:03.265443 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:03.265444    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1ca2705-0c5d-45cb-9f54-5f82238eabec" containerName="kserve-container"
Jul 01 19:36:03.265443 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:03.265449    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1ca2705-0c5d-45cb-9f54-5f82238eabec" containerName="kserve-container"
Jul 01 19:36:03.265932 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:03.265457    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="835093bf-c5c0-4835-96c5-bd03e5097839" containerName="kube-rbac-proxy"
Jul 01 19:36:03.265932 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:03.265462    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="835093bf-c5c0-4835-96c5-bd03e5097839" containerName="kube-rbac-proxy"
Jul 01 19:36:03.265932 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:03.265522    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="835093bf-c5c0-4835-96c5-bd03e5097839" containerName="kserve-container"
Jul 01 19:36:03.265932 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:03.265531    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1ca2705-0c5d-45cb-9f54-5f82238eabec" containerName="kserve-container"
Jul 01 19:36:03.265932 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:03.265537    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="835093bf-c5c0-4835-96c5-bd03e5097839" containerName="kube-rbac-proxy"
Jul 01 19:36:03.265932 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:03.265548    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1ca2705-0c5d-45cb-9f54-5f82238eabec" containerName="kube-rbac-proxy"
Jul 01 19:36:03.268769 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:03.268753    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-553af-predictor-fdf784bb4-h4pqq"
Jul 01 19:36:03.271866 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:03.271848    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-553af-predictor-serving-cert\""
Jul 01 19:36:03.272025 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:03.271850    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-553af-kube-rbac-proxy-sar-config\""
Jul 01 19:36:03.283401 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:03.283376    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-553af-predictor-fdf784bb4-h4pqq"]
Jul 01 19:36:03.311190 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:03.311165    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45d7a420-0414-48eb-af91-949b2f67ff63-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-553af-predictor-fdf784bb4-h4pqq\" (UID: \"45d7a420-0414-48eb-af91-949b2f67ff63\") " pod="kserve-ci-e2e-test/message-dumper-raw-553af-predictor-fdf784bb4-h4pqq"
Jul 01 19:36:03.311341 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:03.311197    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxfx6\" (UniqueName: \"kubernetes.io/projected/45d7a420-0414-48eb-af91-949b2f67ff63-kube-api-access-cxfx6\") pod \"message-dumper-raw-553af-predictor-fdf784bb4-h4pqq\" (UID: \"45d7a420-0414-48eb-af91-949b2f67ff63\") " pod="kserve-ci-e2e-test/message-dumper-raw-553af-predictor-fdf784bb4-h4pqq"
Jul 01 19:36:03.311341 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:03.311285    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45d7a420-0414-48eb-af91-949b2f67ff63-proxy-tls\") pod \"message-dumper-raw-553af-predictor-fdf784bb4-h4pqq\" (UID: \"45d7a420-0414-48eb-af91-949b2f67ff63\") " pod="kserve-ci-e2e-test/message-dumper-raw-553af-predictor-fdf784bb4-h4pqq"
Jul 01 19:36:03.375953 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:03.375919    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx"]
Jul 01 19:36:03.376188 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:03.376157    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx" podUID="4d770e95-639f-423c-8584-8ec945730bd8" containerName="kserve-container" containerID="cri-o://4aef491a37a2fd92cb55d8afaeb4098a839b309339c66776b947e705a0022007" gracePeriod=30
Jul 01 19:36:03.376245 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:03.376177    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx" podUID="4d770e95-639f-423c-8584-8ec945730bd8" containerName="kube-rbac-proxy" containerID="cri-o://b05b67849f48f32a74f0113fed0c4f3273cb05330d08bbc3f5bd822c47372586" gracePeriod=30
Jul 01 19:36:03.412154 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:03.412121    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45d7a420-0414-48eb-af91-949b2f67ff63-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-553af-predictor-fdf784bb4-h4pqq\" (UID: \"45d7a420-0414-48eb-af91-949b2f67ff63\") " pod="kserve-ci-e2e-test/message-dumper-raw-553af-predictor-fdf784bb4-h4pqq"
Jul 01 19:36:03.412283 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:03.412176    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxfx6\" (UniqueName: \"kubernetes.io/projected/45d7a420-0414-48eb-af91-949b2f67ff63-kube-api-access-cxfx6\") pod \"message-dumper-raw-553af-predictor-fdf784bb4-h4pqq\" (UID: \"45d7a420-0414-48eb-af91-949b2f67ff63\") " pod="kserve-ci-e2e-test/message-dumper-raw-553af-predictor-fdf784bb4-h4pqq"
Jul 01 19:36:03.412354 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:03.412340    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45d7a420-0414-48eb-af91-949b2f67ff63-proxy-tls\") pod \"message-dumper-raw-553af-predictor-fdf784bb4-h4pqq\" (UID: \"45d7a420-0414-48eb-af91-949b2f67ff63\") " pod="kserve-ci-e2e-test/message-dumper-raw-553af-predictor-fdf784bb4-h4pqq"
Jul 01 19:36:03.412506 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:36:03.412476    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-553af-predictor-serving-cert: secret "message-dumper-raw-553af-predictor-serving-cert" not found
Jul 01 19:36:03.412616 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:36:03.412563    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45d7a420-0414-48eb-af91-949b2f67ff63-proxy-tls podName:45d7a420-0414-48eb-af91-949b2f67ff63 nodeName:}" failed. No retries permitted until 2026-07-01 19:36:03.912543077 +0000 UTC m=+870.004542504 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/45d7a420-0414-48eb-af91-949b2f67ff63-proxy-tls") pod "message-dumper-raw-553af-predictor-fdf784bb4-h4pqq" (UID: "45d7a420-0414-48eb-af91-949b2f67ff63") : secret "message-dumper-raw-553af-predictor-serving-cert" not found
Jul 01 19:36:03.412885 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:03.412860    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45d7a420-0414-48eb-af91-949b2f67ff63-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-553af-predictor-fdf784bb4-h4pqq\" (UID: \"45d7a420-0414-48eb-af91-949b2f67ff63\") " pod="kserve-ci-e2e-test/message-dumper-raw-553af-predictor-fdf784bb4-h4pqq"
Jul 01 19:36:03.422603 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:03.422578    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxfx6\" (UniqueName: \"kubernetes.io/projected/45d7a420-0414-48eb-af91-949b2f67ff63-kube-api-access-cxfx6\") pod \"message-dumper-raw-553af-predictor-fdf784bb4-h4pqq\" (UID: \"45d7a420-0414-48eb-af91-949b2f67ff63\") " pod="kserve-ci-e2e-test/message-dumper-raw-553af-predictor-fdf784bb4-h4pqq"
Jul 01 19:36:03.437985 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:03.437957    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc"]
Jul 01 19:36:03.438220 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:03.438173    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc" podUID="d9ff26bb-9658-4d8b-b10a-eddf3932c62c" containerName="kserve-container" containerID="cri-o://84b614285eb84d913d63df7b5378ac2183ed8d9e2e05be3301cc3c44d0a59729" gracePeriod=30
Jul 01 19:36:03.438292 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:03.438212    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc" podUID="d9ff26bb-9658-4d8b-b10a-eddf3932c62c" containerName="kube-rbac-proxy" containerID="cri-o://89c56127dc9aecf30dbfbc3f47469d7bed912794bdb9c5577cb23298b2fc7c16" gracePeriod=30
Jul 01 19:36:03.916165 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:03.916131    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45d7a420-0414-48eb-af91-949b2f67ff63-proxy-tls\") pod \"message-dumper-raw-553af-predictor-fdf784bb4-h4pqq\" (UID: \"45d7a420-0414-48eb-af91-949b2f67ff63\") " pod="kserve-ci-e2e-test/message-dumper-raw-553af-predictor-fdf784bb4-h4pqq"
Jul 01 19:36:03.918444 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:03.918423    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45d7a420-0414-48eb-af91-949b2f67ff63-proxy-tls\") pod \"message-dumper-raw-553af-predictor-fdf784bb4-h4pqq\" (UID: \"45d7a420-0414-48eb-af91-949b2f67ff63\") " pod="kserve-ci-e2e-test/message-dumper-raw-553af-predictor-fdf784bb4-h4pqq"
Jul 01 19:36:04.178549 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:04.178465    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-553af-predictor-fdf784bb4-h4pqq"
Jul 01 19:36:04.308401 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:04.308379    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-553af-predictor-fdf784bb4-h4pqq"]
Jul 01 19:36:04.310548 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:36:04.310525    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45d7a420_0414_48eb_af91_949b2f67ff63.slice/crio-80d2f632bd0bf71a977f7db73c78688b7c8ac311f8e0e93b9136a9271a96acf2 WatchSource:0}: Error finding container 80d2f632bd0bf71a977f7db73c78688b7c8ac311f8e0e93b9136a9271a96acf2: Status 404 returned error can't find the container with id 80d2f632bd0bf71a977f7db73c78688b7c8ac311f8e0e93b9136a9271a96acf2
Jul 01 19:36:04.312388 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:04.312370    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 19:36:04.425028 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:04.424997    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-553af-predictor-fdf784bb4-h4pqq" event={"ID":"45d7a420-0414-48eb-af91-949b2f67ff63","Type":"ContainerStarted","Data":"80d2f632bd0bf71a977f7db73c78688b7c8ac311f8e0e93b9136a9271a96acf2"}
Jul 01 19:36:04.427106 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:04.427081    2574 generic.go:356] "Generic (PLEG): container finished" podID="4d770e95-639f-423c-8584-8ec945730bd8" containerID="b05b67849f48f32a74f0113fed0c4f3273cb05330d08bbc3f5bd822c47372586" exitCode=2
Jul 01 19:36:04.427229 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:04.427147    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx" event={"ID":"4d770e95-639f-423c-8584-8ec945730bd8","Type":"ContainerDied","Data":"b05b67849f48f32a74f0113fed0c4f3273cb05330d08bbc3f5bd822c47372586"}
Jul 01 19:36:04.429244 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:04.429180    2574 generic.go:356] "Generic (PLEG): container finished" podID="d9ff26bb-9658-4d8b-b10a-eddf3932c62c" containerID="89c56127dc9aecf30dbfbc3f47469d7bed912794bdb9c5577cb23298b2fc7c16" exitCode=2
Jul 01 19:36:04.429244 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:04.429207    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc" event={"ID":"d9ff26bb-9658-4d8b-b10a-eddf3932c62c","Type":"ContainerDied","Data":"89c56127dc9aecf30dbfbc3f47469d7bed912794bdb9c5577cb23298b2fc7c16"}
Jul 01 19:36:04.927478 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:04.927441    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc" podUID="d9ff26bb-9658-4d8b-b10a-eddf3932c62c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jul 01 19:36:04.927668 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:04.927451    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx" podUID="4d770e95-639f-423c-8584-8ec945730bd8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jul 01 19:36:04.932253 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:04.932233    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx" podUID="4d770e95-639f-423c-8584-8ec945730bd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 19:36:04.932339 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:04.932265    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc" podUID="d9ff26bb-9658-4d8b-b10a-eddf3932c62c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 01 19:36:06.441650 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:06.441613    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-553af-predictor-fdf784bb4-h4pqq" event={"ID":"45d7a420-0414-48eb-af91-949b2f67ff63","Type":"ContainerStarted","Data":"a15dd5ee7fe09737a7b6c1f268296cd431acf1e44e3b44ae745900aa93c49a77"}
Jul 01 19:36:06.442084 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:06.441665    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-553af-predictor-fdf784bb4-h4pqq" event={"ID":"45d7a420-0414-48eb-af91-949b2f67ff63","Type":"ContainerStarted","Data":"640c1d56fb44aaf34b879b1cd07699a0d8f6f6dd35bcc7aca1a15c98f6dbc845"}
Jul 01 19:36:06.442084 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:06.441691    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-553af-predictor-fdf784bb4-h4pqq"
Jul 01 19:36:06.442084 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:06.441749    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-553af-predictor-fdf784bb4-h4pqq"
Jul 01 19:36:06.443704 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:06.443686    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-553af-predictor-fdf784bb4-h4pqq"
Jul 01 19:36:06.459783 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:06.459735    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-553af-predictor-fdf784bb4-h4pqq" podStartSLOduration=2.181303746 podStartE2EDuration="3.459716273s" totalImagesPullingTime="1.278412527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:36:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:36:06.45881993 +0000 UTC m=+872.550819378" watchObservedRunningTime="2026-07-01 19:36:06.459716273 +0000 UTC m=+872.551715727"
Jul 01 19:36:07.096083 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:07.096060    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc"
Jul 01 19:36:07.144386 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:07.144300    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9ff26bb-9658-4d8b-b10a-eddf3932c62c-proxy-tls\") pod \"d9ff26bb-9658-4d8b-b10a-eddf3932c62c\" (UID: \"d9ff26bb-9658-4d8b-b10a-eddf3932c62c\") "
Jul 01 19:36:07.144386 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:07.144360    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49m68\" (UniqueName: \"kubernetes.io/projected/d9ff26bb-9658-4d8b-b10a-eddf3932c62c-kube-api-access-49m68\") pod \"d9ff26bb-9658-4d8b-b10a-eddf3932c62c\" (UID: \"d9ff26bb-9658-4d8b-b10a-eddf3932c62c\") "
Jul 01 19:36:07.144619 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:07.144388    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9ff26bb-9658-4d8b-b10a-eddf3932c62c-kserve-provision-location\") pod \"d9ff26bb-9658-4d8b-b10a-eddf3932c62c\" (UID: \"d9ff26bb-9658-4d8b-b10a-eddf3932c62c\") "
Jul 01 19:36:07.144619 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:07.144417    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9ff26bb-9658-4d8b-b10a-eddf3932c62c-cabundle-cert\") pod \"d9ff26bb-9658-4d8b-b10a-eddf3932c62c\" (UID: \"d9ff26bb-9658-4d8b-b10a-eddf3932c62c\") "
Jul 01 19:36:07.144619 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:07.144504    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9ff26bb-9658-4d8b-b10a-eddf3932c62c-kube-rbac-proxy-sar-config\") pod \"d9ff26bb-9658-4d8b-b10a-eddf3932c62c\" (UID: \"d9ff26bb-9658-4d8b-b10a-eddf3932c62c\") "
Jul 01 19:36:07.144772 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:07.144716    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ff26bb-9658-4d8b-b10a-eddf3932c62c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d9ff26bb-9658-4d8b-b10a-eddf3932c62c" (UID: "d9ff26bb-9658-4d8b-b10a-eddf3932c62c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:36:07.144978 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:07.144945    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9ff26bb-9658-4d8b-b10a-eddf3932c62c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d9ff26bb-9658-4d8b-b10a-eddf3932c62c" (UID: "d9ff26bb-9658-4d8b-b10a-eddf3932c62c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:36:07.145093 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:07.145008    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9ff26bb-9658-4d8b-b10a-eddf3932c62c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d9ff26bb-9658-4d8b-b10a-eddf3932c62c" (UID: "d9ff26bb-9658-4d8b-b10a-eddf3932c62c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:36:07.146570 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:07.146547    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9ff26bb-9658-4d8b-b10a-eddf3932c62c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d9ff26bb-9658-4d8b-b10a-eddf3932c62c" (UID: "d9ff26bb-9658-4d8b-b10a-eddf3932c62c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:36:07.146672 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:07.146613    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9ff26bb-9658-4d8b-b10a-eddf3932c62c-kube-api-access-49m68" (OuterVolumeSpecName: "kube-api-access-49m68") pod "d9ff26bb-9658-4d8b-b10a-eddf3932c62c" (UID: "d9ff26bb-9658-4d8b-b10a-eddf3932c62c"). InnerVolumeSpecName "kube-api-access-49m68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:36:07.246010 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:07.245983    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49m68\" (UniqueName: \"kubernetes.io/projected/d9ff26bb-9658-4d8b-b10a-eddf3932c62c-kube-api-access-49m68\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:36:07.246010 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:07.246005    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9ff26bb-9658-4d8b-b10a-eddf3932c62c-kserve-provision-location\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:36:07.246191 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:07.246019    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9ff26bb-9658-4d8b-b10a-eddf3932c62c-cabundle-cert\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:36:07.246191 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:07.246035    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9ff26bb-9658-4d8b-b10a-eddf3932c62c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:36:07.246191 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:07.246046    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9ff26bb-9658-4d8b-b10a-eddf3932c62c-proxy-tls\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:36:07.448080 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:07.447990    2574 generic.go:356] "Generic (PLEG): container finished" podID="d9ff26bb-9658-4d8b-b10a-eddf3932c62c" containerID="84b614285eb84d913d63df7b5378ac2183ed8d9e2e05be3301cc3c44d0a59729" exitCode=0
Jul 01 19:36:07.448080 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:07.448068    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc"
Jul 01 19:36:07.448538 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:07.448073    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc" event={"ID":"d9ff26bb-9658-4d8b-b10a-eddf3932c62c","Type":"ContainerDied","Data":"84b614285eb84d913d63df7b5378ac2183ed8d9e2e05be3301cc3c44d0a59729"}
Jul 01 19:36:07.448538 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:07.448150    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc" event={"ID":"d9ff26bb-9658-4d8b-b10a-eddf3932c62c","Type":"ContainerDied","Data":"cb0e9a712a8f4311e4f7b9ffaf0cc1e1c1699a5e9e967c32de5233e1f79a30fa"}
Jul 01 19:36:07.448538 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:07.448193    2574 scope.go:117] "RemoveContainer" containerID="89c56127dc9aecf30dbfbc3f47469d7bed912794bdb9c5577cb23298b2fc7c16"
Jul 01 19:36:07.463593 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:07.463569    2574 scope.go:117] "RemoveContainer" containerID="84b614285eb84d913d63df7b5378ac2183ed8d9e2e05be3301cc3c44d0a59729"
Jul 01 19:36:07.471700 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:07.471677    2574 scope.go:117] "RemoveContainer" containerID="feca1ce156cc2c8977e238f83b59c93d5d01be778e9278948f932d3db1dc4faa"
Jul 01 19:36:07.474623 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:07.474597    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc"]
Jul 01 19:36:07.478200 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:07.478174    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6e49a-predictor-844cbd6d5-zqdxc"]
Jul 01 19:36:07.480160 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:07.480147    2574 scope.go:117] "RemoveContainer" containerID="89c56127dc9aecf30dbfbc3f47469d7bed912794bdb9c5577cb23298b2fc7c16"
Jul 01 19:36:07.480412 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:36:07.480391    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89c56127dc9aecf30dbfbc3f47469d7bed912794bdb9c5577cb23298b2fc7c16\": container with ID starting with 89c56127dc9aecf30dbfbc3f47469d7bed912794bdb9c5577cb23298b2fc7c16 not found: ID does not exist" containerID="89c56127dc9aecf30dbfbc3f47469d7bed912794bdb9c5577cb23298b2fc7c16"
Jul 01 19:36:07.480487 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:07.480424    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89c56127dc9aecf30dbfbc3f47469d7bed912794bdb9c5577cb23298b2fc7c16"} err="failed to get container status \"89c56127dc9aecf30dbfbc3f47469d7bed912794bdb9c5577cb23298b2fc7c16\": rpc error: code = NotFound desc = could not find container \"89c56127dc9aecf30dbfbc3f47469d7bed912794bdb9c5577cb23298b2fc7c16\": container with ID starting with 89c56127dc9aecf30dbfbc3f47469d7bed912794bdb9c5577cb23298b2fc7c16 not found: ID does not exist"
Jul 01 19:36:07.480487 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:07.480450    2574 scope.go:117] "RemoveContainer" containerID="84b614285eb84d913d63df7b5378ac2183ed8d9e2e05be3301cc3c44d0a59729"
Jul 01 19:36:07.480704 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:36:07.480683    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84b614285eb84d913d63df7b5378ac2183ed8d9e2e05be3301cc3c44d0a59729\": container with ID starting with 84b614285eb84d913d63df7b5378ac2183ed8d9e2e05be3301cc3c44d0a59729 not found: ID does not exist" containerID="84b614285eb84d913d63df7b5378ac2183ed8d9e2e05be3301cc3c44d0a59729"
Jul 01 19:36:07.480744 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:07.480711    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84b614285eb84d913d63df7b5378ac2183ed8d9e2e05be3301cc3c44d0a59729"} err="failed to get container status \"84b614285eb84d913d63df7b5378ac2183ed8d9e2e05be3301cc3c44d0a59729\": rpc error: code = NotFound desc = could not find container \"84b614285eb84d913d63df7b5378ac2183ed8d9e2e05be3301cc3c44d0a59729\": container with ID starting with 84b614285eb84d913d63df7b5378ac2183ed8d9e2e05be3301cc3c44d0a59729 not found: ID does not exist"
Jul 01 19:36:07.480744 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:07.480727    2574 scope.go:117] "RemoveContainer" containerID="feca1ce156cc2c8977e238f83b59c93d5d01be778e9278948f932d3db1dc4faa"
Jul 01 19:36:07.480949 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:36:07.480932    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feca1ce156cc2c8977e238f83b59c93d5d01be778e9278948f932d3db1dc4faa\": container with ID starting with feca1ce156cc2c8977e238f83b59c93d5d01be778e9278948f932d3db1dc4faa not found: ID does not exist" containerID="feca1ce156cc2c8977e238f83b59c93d5d01be778e9278948f932d3db1dc4faa"
Jul 01 19:36:07.481013 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:07.480955    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feca1ce156cc2c8977e238f83b59c93d5d01be778e9278948f932d3db1dc4faa"} err="failed to get container status \"feca1ce156cc2c8977e238f83b59c93d5d01be778e9278948f932d3db1dc4faa\": rpc error: code = NotFound desc = could not find container \"feca1ce156cc2c8977e238f83b59c93d5d01be778e9278948f932d3db1dc4faa\": container with ID starting with feca1ce156cc2c8977e238f83b59c93d5d01be778e9278948f932d3db1dc4faa not found: ID does not exist"
Jul 01 19:36:08.127016 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:08.126992    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx"
Jul 01 19:36:08.153609 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:08.153583    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d770e95-639f-423c-8584-8ec945730bd8-cabundle-cert\") pod \"4d770e95-639f-423c-8584-8ec945730bd8\" (UID: \"4d770e95-639f-423c-8584-8ec945730bd8\") "
Jul 01 19:36:08.153760 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:08.153643    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d770e95-639f-423c-8584-8ec945730bd8-kube-rbac-proxy-sar-config\") pod \"4d770e95-639f-423c-8584-8ec945730bd8\" (UID: \"4d770e95-639f-423c-8584-8ec945730bd8\") "
Jul 01 19:36:08.153760 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:08.153674    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjpqh\" (UniqueName: \"kubernetes.io/projected/4d770e95-639f-423c-8584-8ec945730bd8-kube-api-access-pjpqh\") pod \"4d770e95-639f-423c-8584-8ec945730bd8\" (UID: \"4d770e95-639f-423c-8584-8ec945730bd8\") "
Jul 01 19:36:08.153760 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:08.153716    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d770e95-639f-423c-8584-8ec945730bd8-kserve-provision-location\") pod \"4d770e95-639f-423c-8584-8ec945730bd8\" (UID: \"4d770e95-639f-423c-8584-8ec945730bd8\") "
Jul 01 19:36:08.153865 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:08.153783    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d770e95-639f-423c-8584-8ec945730bd8-proxy-tls\") pod \"4d770e95-639f-423c-8584-8ec945730bd8\" (UID: \"4d770e95-639f-423c-8584-8ec945730bd8\") "
Jul 01 19:36:08.154050 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:08.154016    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d770e95-639f-423c-8584-8ec945730bd8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4d770e95-639f-423c-8584-8ec945730bd8" (UID: "4d770e95-639f-423c-8584-8ec945730bd8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:36:08.154190 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:08.154060    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d770e95-639f-423c-8584-8ec945730bd8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4d770e95-639f-423c-8584-8ec945730bd8" (UID: "4d770e95-639f-423c-8584-8ec945730bd8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:36:08.154335 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:08.154304    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d770e95-639f-423c-8584-8ec945730bd8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4d770e95-639f-423c-8584-8ec945730bd8" (UID: "4d770e95-639f-423c-8584-8ec945730bd8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:36:08.155934 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:08.155888    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d770e95-639f-423c-8584-8ec945730bd8-kube-api-access-pjpqh" (OuterVolumeSpecName: "kube-api-access-pjpqh") pod "4d770e95-639f-423c-8584-8ec945730bd8" (UID: "4d770e95-639f-423c-8584-8ec945730bd8"). InnerVolumeSpecName "kube-api-access-pjpqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:36:08.156019 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:08.155939    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d770e95-639f-423c-8584-8ec945730bd8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4d770e95-639f-423c-8584-8ec945730bd8" (UID: "4d770e95-639f-423c-8584-8ec945730bd8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:36:08.255118 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:08.255025    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d770e95-639f-423c-8584-8ec945730bd8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:36:08.255118 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:08.255055    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjpqh\" (UniqueName: \"kubernetes.io/projected/4d770e95-639f-423c-8584-8ec945730bd8-kube-api-access-pjpqh\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:36:08.255118 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:08.255065    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d770e95-639f-423c-8584-8ec945730bd8-kserve-provision-location\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:36:08.255118 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:08.255074    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d770e95-639f-423c-8584-8ec945730bd8-proxy-tls\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:36:08.255118 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:08.255084    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d770e95-639f-423c-8584-8ec945730bd8-cabundle-cert\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:36:08.357678 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:08.357641    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9ff26bb-9658-4d8b-b10a-eddf3932c62c" path="/var/lib/kubelet/pods/d9ff26bb-9658-4d8b-b10a-eddf3932c62c/volumes"
Jul 01 19:36:08.454917 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:08.454861    2574 generic.go:356] "Generic (PLEG): container finished" podID="4d770e95-639f-423c-8584-8ec945730bd8" containerID="4aef491a37a2fd92cb55d8afaeb4098a839b309339c66776b947e705a0022007" exitCode=0
Jul 01 19:36:08.455469 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:08.454935    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx"
Jul 01 19:36:08.455469 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:08.454931    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx" event={"ID":"4d770e95-639f-423c-8584-8ec945730bd8","Type":"ContainerDied","Data":"4aef491a37a2fd92cb55d8afaeb4098a839b309339c66776b947e705a0022007"}
Jul 01 19:36:08.455469 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:08.455012    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx" event={"ID":"4d770e95-639f-423c-8584-8ec945730bd8","Type":"ContainerDied","Data":"ccd2b38d87fb79ba5a3bc6cc37abc5a054fbe62435be885a418e2bc26cbee4c2"}
Jul 01 19:36:08.455469 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:08.455058    2574 scope.go:117] "RemoveContainer" containerID="b05b67849f48f32a74f0113fed0c4f3273cb05330d08bbc3f5bd822c47372586"
Jul 01 19:36:08.463879 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:08.463861    2574 scope.go:117] "RemoveContainer" containerID="4aef491a37a2fd92cb55d8afaeb4098a839b309339c66776b947e705a0022007"
Jul 01 19:36:08.472023 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:08.472004    2574 scope.go:117] "RemoveContainer" containerID="86b3a37eff22993260d050b41737673feee701fe79fc595aad72d56af7048642"
Jul 01 19:36:08.472361 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:08.472338    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx"]
Jul 01 19:36:08.475530 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:08.475509    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6e49a-predictor-86f574459-7bvcx"]
Jul 01 19:36:08.479964 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:08.479944    2574 scope.go:117] "RemoveContainer" containerID="b05b67849f48f32a74f0113fed0c4f3273cb05330d08bbc3f5bd822c47372586"
Jul 01 19:36:08.480275 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:36:08.480248    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b05b67849f48f32a74f0113fed0c4f3273cb05330d08bbc3f5bd822c47372586\": container with ID starting with b05b67849f48f32a74f0113fed0c4f3273cb05330d08bbc3f5bd822c47372586 not found: ID does not exist" containerID="b05b67849f48f32a74f0113fed0c4f3273cb05330d08bbc3f5bd822c47372586"
Jul 01 19:36:08.480324 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:08.480277    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b05b67849f48f32a74f0113fed0c4f3273cb05330d08bbc3f5bd822c47372586"} err="failed to get container status \"b05b67849f48f32a74f0113fed0c4f3273cb05330d08bbc3f5bd822c47372586\": rpc error: code = NotFound desc = could not find container \"b05b67849f48f32a74f0113fed0c4f3273cb05330d08bbc3f5bd822c47372586\": container with ID starting with b05b67849f48f32a74f0113fed0c4f3273cb05330d08bbc3f5bd822c47372586 not found: ID does not exist"
Jul 01 19:36:08.480324 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:08.480297    2574 scope.go:117] "RemoveContainer" containerID="4aef491a37a2fd92cb55d8afaeb4098a839b309339c66776b947e705a0022007"
Jul 01 19:36:08.480549 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:36:08.480524    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aef491a37a2fd92cb55d8afaeb4098a839b309339c66776b947e705a0022007\": container with ID starting with 4aef491a37a2fd92cb55d8afaeb4098a839b309339c66776b947e705a0022007 not found: ID does not exist" containerID="4aef491a37a2fd92cb55d8afaeb4098a839b309339c66776b947e705a0022007"
Jul 01 19:36:08.480589 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:08.480549    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aef491a37a2fd92cb55d8afaeb4098a839b309339c66776b947e705a0022007"} err="failed to get container status \"4aef491a37a2fd92cb55d8afaeb4098a839b309339c66776b947e705a0022007\": rpc error: code = NotFound desc = could not find container \"4aef491a37a2fd92cb55d8afaeb4098a839b309339c66776b947e705a0022007\": container with ID starting with 4aef491a37a2fd92cb55d8afaeb4098a839b309339c66776b947e705a0022007 not found: ID does not exist"
Jul 01 19:36:08.480589 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:08.480571    2574 scope.go:117] "RemoveContainer" containerID="86b3a37eff22993260d050b41737673feee701fe79fc595aad72d56af7048642"
Jul 01 19:36:08.480796 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:36:08.480780    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86b3a37eff22993260d050b41737673feee701fe79fc595aad72d56af7048642\": container with ID starting with 86b3a37eff22993260d050b41737673feee701fe79fc595aad72d56af7048642 not found: ID does not exist" containerID="86b3a37eff22993260d050b41737673feee701fe79fc595aad72d56af7048642"
Jul 01 19:36:08.480837 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:08.480800    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86b3a37eff22993260d050b41737673feee701fe79fc595aad72d56af7048642"} err="failed to get container status \"86b3a37eff22993260d050b41737673feee701fe79fc595aad72d56af7048642\": rpc error: code = NotFound desc = could not find container \"86b3a37eff22993260d050b41737673feee701fe79fc595aad72d56af7048642\": container with ID starting with 86b3a37eff22993260d050b41737673feee701fe79fc595aad72d56af7048642 not found: ID does not exist"
Jul 01 19:36:10.356806 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:10.356768    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d770e95-639f-423c-8584-8ec945730bd8" path="/var/lib/kubelet/pods/4d770e95-639f-423c-8584-8ec945730bd8/volumes"
Jul 01 19:36:13.461414 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:13.461386    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-553af-predictor-fdf784bb4-h4pqq"
Jul 01 19:36:23.287977 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:23.287922    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5"]
Jul 01 19:36:23.288364 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:23.288316    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9ff26bb-9658-4d8b-b10a-eddf3932c62c" containerName="kserve-container"
Jul 01 19:36:23.288364 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:23.288329    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9ff26bb-9658-4d8b-b10a-eddf3932c62c" containerName="kserve-container"
Jul 01 19:36:23.288364 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:23.288341    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d770e95-639f-423c-8584-8ec945730bd8" containerName="kserve-container"
Jul 01 19:36:23.288364 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:23.288347    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d770e95-639f-423c-8584-8ec945730bd8" containerName="kserve-container"
Jul 01 19:36:23.288364 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:23.288356    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d770e95-639f-423c-8584-8ec945730bd8" containerName="storage-initializer"
Jul 01 19:36:23.288364 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:23.288361    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d770e95-639f-423c-8584-8ec945730bd8" containerName="storage-initializer"
Jul 01 19:36:23.288364 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:23.288367    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9ff26bb-9658-4d8b-b10a-eddf3932c62c" containerName="kube-rbac-proxy"
Jul 01 19:36:23.288583 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:23.288373    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9ff26bb-9658-4d8b-b10a-eddf3932c62c" containerName="kube-rbac-proxy"
Jul 01 19:36:23.288583 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:23.288393    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d770e95-639f-423c-8584-8ec945730bd8" containerName="kube-rbac-proxy"
Jul 01 19:36:23.288583 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:23.288397    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d770e95-639f-423c-8584-8ec945730bd8" containerName="kube-rbac-proxy"
Jul 01 19:36:23.288583 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:23.288403    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9ff26bb-9658-4d8b-b10a-eddf3932c62c" containerName="storage-initializer"
Jul 01 19:36:23.288583 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:23.288408    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9ff26bb-9658-4d8b-b10a-eddf3932c62c" containerName="storage-initializer"
Jul 01 19:36:23.288583 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:23.288462    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d770e95-639f-423c-8584-8ec945730bd8" containerName="kube-rbac-proxy"
Jul 01 19:36:23.288583 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:23.288472    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9ff26bb-9658-4d8b-b10a-eddf3932c62c" containerName="kserve-container"
Jul 01 19:36:23.288583 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:23.288479    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d770e95-639f-423c-8584-8ec945730bd8" containerName="kserve-container"
Jul 01 19:36:23.288583 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:23.288486    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9ff26bb-9658-4d8b-b10a-eddf3932c62c" containerName="kube-rbac-proxy"
Jul 01 19:36:23.293544 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:23.293526    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5"
Jul 01 19:36:23.296020 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:23.295996    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-553af-predictor-serving-cert\""
Jul 01 19:36:23.296020 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:23.296011    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 19:36:23.296192 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:23.296044    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-553af-kube-rbac-proxy-sar-config\""
Jul 01 19:36:23.296240 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:23.296224    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 19:36:23.302061 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:23.302031    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5"]
Jul 01 19:36:23.387025 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:23.386994    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b4c4555-8e4a-4a46-8ecb-779c792c0955-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5\" (UID: \"4b4c4555-8e4a-4a46-8ecb-779c792c0955\") " pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5"
Jul 01 19:36:23.387141 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:23.387042    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b4c4555-8e4a-4a46-8ecb-779c792c0955-cabundle-cert\") pod \"isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5\" (UID: \"4b4c4555-8e4a-4a46-8ecb-779c792c0955\") " pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5"
Jul 01 19:36:23.387141 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:23.387070    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b4c4555-8e4a-4a46-8ecb-779c792c0955-kserve-provision-location\") pod \"isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5\" (UID: \"4b4c4555-8e4a-4a46-8ecb-779c792c0955\") " pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5"
Jul 01 19:36:23.387141 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:23.387092    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b4c4555-8e4a-4a46-8ecb-779c792c0955-proxy-tls\") pod \"isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5\" (UID: \"4b4c4555-8e4a-4a46-8ecb-779c792c0955\") " pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5"
Jul 01 19:36:23.387141 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:23.387108    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxbhp\" (UniqueName: \"kubernetes.io/projected/4b4c4555-8e4a-4a46-8ecb-779c792c0955-kube-api-access-zxbhp\") pod \"isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5\" (UID: \"4b4c4555-8e4a-4a46-8ecb-779c792c0955\") " pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5"
Jul 01 19:36:23.487843 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:23.487810    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b4c4555-8e4a-4a46-8ecb-779c792c0955-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5\" (UID: \"4b4c4555-8e4a-4a46-8ecb-779c792c0955\") " pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5"
Jul 01 19:36:23.488117 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:23.488076    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b4c4555-8e4a-4a46-8ecb-779c792c0955-cabundle-cert\") pod \"isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5\" (UID: \"4b4c4555-8e4a-4a46-8ecb-779c792c0955\") " pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5"
Jul 01 19:36:23.488257 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:23.488162    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b4c4555-8e4a-4a46-8ecb-779c792c0955-kserve-provision-location\") pod \"isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5\" (UID: \"4b4c4555-8e4a-4a46-8ecb-779c792c0955\") " pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5"
Jul 01 19:36:23.488257 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:23.488208    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b4c4555-8e4a-4a46-8ecb-779c792c0955-proxy-tls\") pod \"isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5\" (UID: \"4b4c4555-8e4a-4a46-8ecb-779c792c0955\") " pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5"
Jul 01 19:36:23.488257 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:23.488238    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxbhp\" (UniqueName: \"kubernetes.io/projected/4b4c4555-8e4a-4a46-8ecb-779c792c0955-kube-api-access-zxbhp\") pod \"isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5\" (UID: \"4b4c4555-8e4a-4a46-8ecb-779c792c0955\") " pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5"
Jul 01 19:36:23.488436 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:36:23.488374    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-serving-cert: secret "isvc-logger-raw-553af-predictor-serving-cert" not found
Jul 01 19:36:23.488505 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:36:23.488452    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b4c4555-8e4a-4a46-8ecb-779c792c0955-proxy-tls podName:4b4c4555-8e4a-4a46-8ecb-779c792c0955 nodeName:}" failed. No retries permitted until 2026-07-01 19:36:23.988432883 +0000 UTC m=+890.080432320 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4b4c4555-8e4a-4a46-8ecb-779c792c0955-proxy-tls") pod "isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" (UID: "4b4c4555-8e4a-4a46-8ecb-779c792c0955") : secret "isvc-logger-raw-553af-predictor-serving-cert" not found
Jul 01 19:36:23.488572 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:23.488515    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b4c4555-8e4a-4a46-8ecb-779c792c0955-kserve-provision-location\") pod \"isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5\" (UID: \"4b4c4555-8e4a-4a46-8ecb-779c792c0955\") " pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5"
Jul 01 19:36:23.488650 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:23.488625    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b4c4555-8e4a-4a46-8ecb-779c792c0955-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5\" (UID: \"4b4c4555-8e4a-4a46-8ecb-779c792c0955\") " pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5"
Jul 01 19:36:23.488720 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:23.488697    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b4c4555-8e4a-4a46-8ecb-779c792c0955-cabundle-cert\") pod \"isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5\" (UID: \"4b4c4555-8e4a-4a46-8ecb-779c792c0955\") " pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5"
Jul 01 19:36:23.498579 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:23.498553    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxbhp\" (UniqueName: \"kubernetes.io/projected/4b4c4555-8e4a-4a46-8ecb-779c792c0955-kube-api-access-zxbhp\") pod \"isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5\" (UID: \"4b4c4555-8e4a-4a46-8ecb-779c792c0955\") " pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5"
Jul 01 19:36:23.992064 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:23.992016    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b4c4555-8e4a-4a46-8ecb-779c792c0955-proxy-tls\") pod \"isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5\" (UID: \"4b4c4555-8e4a-4a46-8ecb-779c792c0955\") " pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5"
Jul 01 19:36:23.994628 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:23.994597    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b4c4555-8e4a-4a46-8ecb-779c792c0955-proxy-tls\") pod \"isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5\" (UID: \"4b4c4555-8e4a-4a46-8ecb-779c792c0955\") " pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5"
Jul 01 19:36:24.226364 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:24.226316    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5"
Jul 01 19:36:24.360587 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:24.360556    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5"]
Jul 01 19:36:24.363958 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:36:24.363926    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b4c4555_8e4a_4a46_8ecb_779c792c0955.slice/crio-71b27f9c5fdf2f0ce7850806a7948d2c715ebce27e2762c879364953f2d392e2 WatchSource:0}: Error finding container 71b27f9c5fdf2f0ce7850806a7948d2c715ebce27e2762c879364953f2d392e2: Status 404 returned error can't find the container with id 71b27f9c5fdf2f0ce7850806a7948d2c715ebce27e2762c879364953f2d392e2
Jul 01 19:36:24.534031 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:24.533932    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" event={"ID":"4b4c4555-8e4a-4a46-8ecb-779c792c0955","Type":"ContainerStarted","Data":"7e63d63d7132f289405417fa1373864c406cd67680abda9a3f3bcee4040a95d5"}
Jul 01 19:36:24.534031 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:24.534015    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" event={"ID":"4b4c4555-8e4a-4a46-8ecb-779c792c0955","Type":"ContainerStarted","Data":"71b27f9c5fdf2f0ce7850806a7948d2c715ebce27e2762c879364953f2d392e2"}
Jul 01 19:36:28.555676 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:28.555644    2574 generic.go:356] "Generic (PLEG): container finished" podID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerID="7e63d63d7132f289405417fa1373864c406cd67680abda9a3f3bcee4040a95d5" exitCode=0
Jul 01 19:36:28.556099 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:28.555724    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" event={"ID":"4b4c4555-8e4a-4a46-8ecb-779c792c0955","Type":"ContainerDied","Data":"7e63d63d7132f289405417fa1373864c406cd67680abda9a3f3bcee4040a95d5"}
Jul 01 19:36:29.563316 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:29.563278    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" event={"ID":"4b4c4555-8e4a-4a46-8ecb-779c792c0955","Type":"ContainerStarted","Data":"41b7c541de3b4fb5066e8800b25438c7edab3d72934baab204d38dbfafce4357"}
Jul 01 19:36:29.563743 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:29.563349    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" event={"ID":"4b4c4555-8e4a-4a46-8ecb-779c792c0955","Type":"ContainerStarted","Data":"88ca6624e1ba02a3235533c33c099fcea2863459d8cfc34c4c7c4ae718ef5cd9"}
Jul 01 19:36:29.563743 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:29.563376    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" event={"ID":"4b4c4555-8e4a-4a46-8ecb-779c792c0955","Type":"ContainerStarted","Data":"1177eadf365a1c52f51ce033a6d9bccc41ac46dff638ad3de40de920a7da7274"}
Jul 01 19:36:29.563743 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:29.563665    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5"
Jul 01 19:36:29.563924 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:29.563819    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5"
Jul 01 19:36:29.565142 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:29.565114    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 01 19:36:29.585953 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:29.585888    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" podStartSLOduration=3.168048758 podStartE2EDuration="6.585872985s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.417824227s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:36:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:36:29.583751981 +0000 UTC m=+895.675751441" watchObservedRunningTime="2026-07-01 19:36:29.585872985 +0000 UTC m=+895.677872432"
Jul 01 19:36:30.567827 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:30.567793    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5"
Jul 01 19:36:30.568297 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:30.567778    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 01 19:36:30.569128 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:30.569098    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:36:31.572226 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:31.572180    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 01 19:36:31.572735 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:31.572709    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:36:34.376926 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:34.376879    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bplsm_bfb845e6-341d-492e-91c9-9c698a0b8d29/ovn-acl-logging/0.log"
Jul 01 19:36:34.378180 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:34.378158    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bplsm_bfb845e6-341d-492e-91c9-9c698a0b8d29/ovn-acl-logging/0.log"
Jul 01 19:36:36.576748 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:36.576714    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5"
Jul 01 19:36:36.577368 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:36.577338    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 01 19:36:36.577609 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:36.577583    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:36:46.577740 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:46.577697    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 01 19:36:46.582498 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:46.578244    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:36:56.577715 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:56.577674    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 01 19:36:56.578266 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:36:56.578151    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:37:06.577989 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:06.577944    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 01 19:37:06.578479 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:06.578457    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:37:16.577885 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:16.577839    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 01 19:37:16.578366 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:16.578343    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:37:26.577516 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:26.577476    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 01 19:37:26.577952 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:26.577929    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:37:36.577454 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:36.577407    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 01 19:37:36.577987 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:36.577804    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:37:46.577966 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:46.577937    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5"
Jul 01 19:37:46.578474 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:46.578454    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5"
Jul 01 19:37:58.309253 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.309203    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-553af-predictor-fdf784bb4-h4pqq_45d7a420-0414-48eb-af91-949b2f67ff63/kserve-container/0.log"
Jul 01 19:37:58.588089 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.588009    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5"]
Jul 01 19:37:58.588276 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.588253    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="kserve-container" containerID="cri-o://1177eadf365a1c52f51ce033a6d9bccc41ac46dff638ad3de40de920a7da7274" gracePeriod=30
Jul 01 19:37:58.588386 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.588362    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="kube-rbac-proxy" containerID="cri-o://88ca6624e1ba02a3235533c33c099fcea2863459d8cfc34c4c7c4ae718ef5cd9" gracePeriod=30
Jul 01 19:37:58.588463 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.588351    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="agent" containerID="cri-o://41b7c541de3b4fb5066e8800b25438c7edab3d72934baab204d38dbfafce4357" gracePeriod=30
Jul 01 19:37:58.604574 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.604546    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b"]
Jul 01 19:37:58.608382 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.608363    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b"
Jul 01 19:37:58.611312 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.611066    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-88764-kube-rbac-proxy-sar-config\""
Jul 01 19:37:58.611453 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.611403    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-88764-predictor-serving-cert\""
Jul 01 19:37:58.625639 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.625609    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b"]
Jul 01 19:37:58.652123 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.652093    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-553af-predictor-fdf784bb4-h4pqq"]
Jul 01 19:37:58.652340 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.652316    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-553af-predictor-fdf784bb4-h4pqq" podUID="45d7a420-0414-48eb-af91-949b2f67ff63" containerName="kserve-container" containerID="cri-o://640c1d56fb44aaf34b879b1cd07699a0d8f6f6dd35bcc7aca1a15c98f6dbc845" gracePeriod=30
Jul 01 19:37:58.652402 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.652347    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-553af-predictor-fdf784bb4-h4pqq" podUID="45d7a420-0414-48eb-af91-949b2f67ff63" containerName="kube-rbac-proxy" containerID="cri-o://a15dd5ee7fe09737a7b6c1f268296cd431acf1e44e3b44ae745900aa93c49a77" gracePeriod=30
Jul 01 19:37:58.694832 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.694797    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8a5ec24-3a26-4c1f-859f-0f58c21010d5-proxy-tls\") pod \"isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b\" (UID: \"d8a5ec24-3a26-4c1f-859f-0f58c21010d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b"
Jul 01 19:37:58.695011 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.694875    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8a5ec24-3a26-4c1f-859f-0f58c21010d5-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b\" (UID: \"d8a5ec24-3a26-4c1f-859f-0f58c21010d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b"
Jul 01 19:37:58.695011 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.694935    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d8a5ec24-3a26-4c1f-859f-0f58c21010d5-cabundle-cert\") pod \"isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b\" (UID: \"d8a5ec24-3a26-4c1f-859f-0f58c21010d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b"
Jul 01 19:37:58.695011 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.694969    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djs6g\" (UniqueName: \"kubernetes.io/projected/d8a5ec24-3a26-4c1f-859f-0f58c21010d5-kube-api-access-djs6g\") pod \"isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b\" (UID: \"d8a5ec24-3a26-4c1f-859f-0f58c21010d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b"
Jul 01 19:37:58.695144 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.695009    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8a5ec24-3a26-4c1f-859f-0f58c21010d5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b\" (UID: \"d8a5ec24-3a26-4c1f-859f-0f58c21010d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b"
Jul 01 19:37:58.796262 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.796219    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8a5ec24-3a26-4c1f-859f-0f58c21010d5-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b\" (UID: \"d8a5ec24-3a26-4c1f-859f-0f58c21010d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b"
Jul 01 19:37:58.796447 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.796268    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d8a5ec24-3a26-4c1f-859f-0f58c21010d5-cabundle-cert\") pod \"isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b\" (UID: \"d8a5ec24-3a26-4c1f-859f-0f58c21010d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b"
Jul 01 19:37:58.796447 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.796297    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djs6g\" (UniqueName: \"kubernetes.io/projected/d8a5ec24-3a26-4c1f-859f-0f58c21010d5-kube-api-access-djs6g\") pod \"isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b\" (UID: \"d8a5ec24-3a26-4c1f-859f-0f58c21010d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b"
Jul 01 19:37:58.796447 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.796328    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8a5ec24-3a26-4c1f-859f-0f58c21010d5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b\" (UID: \"d8a5ec24-3a26-4c1f-859f-0f58c21010d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b"
Jul 01 19:37:58.796447 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.796361    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8a5ec24-3a26-4c1f-859f-0f58c21010d5-proxy-tls\") pod \"isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b\" (UID: \"d8a5ec24-3a26-4c1f-859f-0f58c21010d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b"
Jul 01 19:37:58.796685 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:37:58.796487    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-serving-cert: secret "isvc-sklearn-scale-raw-88764-predictor-serving-cert" not found
Jul 01 19:37:58.796685 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:37:58.796549    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8a5ec24-3a26-4c1f-859f-0f58c21010d5-proxy-tls podName:d8a5ec24-3a26-4c1f-859f-0f58c21010d5 nodeName:}" failed. No retries permitted until 2026-07-01 19:37:59.296532526 +0000 UTC m=+985.388531974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d8a5ec24-3a26-4c1f-859f-0f58c21010d5-proxy-tls") pod "isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b" (UID: "d8a5ec24-3a26-4c1f-859f-0f58c21010d5") : secret "isvc-sklearn-scale-raw-88764-predictor-serving-cert" not found
Jul 01 19:37:58.796813 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.796775    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8a5ec24-3a26-4c1f-859f-0f58c21010d5-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b\" (UID: \"d8a5ec24-3a26-4c1f-859f-0f58c21010d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b"
Jul 01 19:37:58.797064 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.797043    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d8a5ec24-3a26-4c1f-859f-0f58c21010d5-cabundle-cert\") pod \"isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b\" (UID: \"d8a5ec24-3a26-4c1f-859f-0f58c21010d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b"
Jul 01 19:37:58.797399 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.797374    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8a5ec24-3a26-4c1f-859f-0f58c21010d5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b\" (UID: \"d8a5ec24-3a26-4c1f-859f-0f58c21010d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b"
Jul 01 19:37:58.806462 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.806438    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djs6g\" (UniqueName: \"kubernetes.io/projected/d8a5ec24-3a26-4c1f-859f-0f58c21010d5-kube-api-access-djs6g\") pod \"isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b\" (UID: \"d8a5ec24-3a26-4c1f-859f-0f58c21010d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b"
Jul 01 19:37:58.907788 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.907756    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-553af-predictor-fdf784bb4-h4pqq"
Jul 01 19:37:58.981369 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.981335    2574 generic.go:356] "Generic (PLEG): container finished" podID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerID="88ca6624e1ba02a3235533c33c099fcea2863459d8cfc34c4c7c4ae718ef5cd9" exitCode=2
Jul 01 19:37:58.981543 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.981414    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" event={"ID":"4b4c4555-8e4a-4a46-8ecb-779c792c0955","Type":"ContainerDied","Data":"88ca6624e1ba02a3235533c33c099fcea2863459d8cfc34c4c7c4ae718ef5cd9"}
Jul 01 19:37:58.982924 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.982886    2574 generic.go:356] "Generic (PLEG): container finished" podID="45d7a420-0414-48eb-af91-949b2f67ff63" containerID="a15dd5ee7fe09737a7b6c1f268296cd431acf1e44e3b44ae745900aa93c49a77" exitCode=2
Jul 01 19:37:58.982924 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.982923    2574 generic.go:356] "Generic (PLEG): container finished" podID="45d7a420-0414-48eb-af91-949b2f67ff63" containerID="640c1d56fb44aaf34b879b1cd07699a0d8f6f6dd35bcc7aca1a15c98f6dbc845" exitCode=2
Jul 01 19:37:58.983059 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.982935    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-553af-predictor-fdf784bb4-h4pqq" event={"ID":"45d7a420-0414-48eb-af91-949b2f67ff63","Type":"ContainerDied","Data":"a15dd5ee7fe09737a7b6c1f268296cd431acf1e44e3b44ae745900aa93c49a77"}
Jul 01 19:37:58.983059 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.982952    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-553af-predictor-fdf784bb4-h4pqq"
Jul 01 19:37:58.983059 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.982987    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-553af-predictor-fdf784bb4-h4pqq" event={"ID":"45d7a420-0414-48eb-af91-949b2f67ff63","Type":"ContainerDied","Data":"640c1d56fb44aaf34b879b1cd07699a0d8f6f6dd35bcc7aca1a15c98f6dbc845"}
Jul 01 19:37:58.983059 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.983011    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-553af-predictor-fdf784bb4-h4pqq" event={"ID":"45d7a420-0414-48eb-af91-949b2f67ff63","Type":"ContainerDied","Data":"80d2f632bd0bf71a977f7db73c78688b7c8ac311f8e0e93b9136a9271a96acf2"}
Jul 01 19:37:58.983059 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.983034    2574 scope.go:117] "RemoveContainer" containerID="a15dd5ee7fe09737a7b6c1f268296cd431acf1e44e3b44ae745900aa93c49a77"
Jul 01 19:37:58.991202 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.991185    2574 scope.go:117] "RemoveContainer" containerID="640c1d56fb44aaf34b879b1cd07699a0d8f6f6dd35bcc7aca1a15c98f6dbc845"
Jul 01 19:37:58.997345 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.997323    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45d7a420-0414-48eb-af91-949b2f67ff63-proxy-tls\") pod \"45d7a420-0414-48eb-af91-949b2f67ff63\" (UID: \"45d7a420-0414-48eb-af91-949b2f67ff63\") "
Jul 01 19:37:58.997436 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.997368    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxfx6\" (UniqueName: \"kubernetes.io/projected/45d7a420-0414-48eb-af91-949b2f67ff63-kube-api-access-cxfx6\") pod \"45d7a420-0414-48eb-af91-949b2f67ff63\" (UID: \"45d7a420-0414-48eb-af91-949b2f67ff63\") "
Jul 01 19:37:58.997436 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.997427    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45d7a420-0414-48eb-af91-949b2f67ff63-kube-rbac-proxy-sar-config\") pod \"45d7a420-0414-48eb-af91-949b2f67ff63\" (UID: \"45d7a420-0414-48eb-af91-949b2f67ff63\") "
Jul 01 19:37:58.997821 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.997799    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45d7a420-0414-48eb-af91-949b2f67ff63-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "45d7a420-0414-48eb-af91-949b2f67ff63" (UID: "45d7a420-0414-48eb-af91-949b2f67ff63"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:37:58.999057 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.999033    2574 scope.go:117] "RemoveContainer" containerID="a15dd5ee7fe09737a7b6c1f268296cd431acf1e44e3b44ae745900aa93c49a77"
Jul 01 19:37:58.999327 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:37:58.999308    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a15dd5ee7fe09737a7b6c1f268296cd431acf1e44e3b44ae745900aa93c49a77\": container with ID starting with a15dd5ee7fe09737a7b6c1f268296cd431acf1e44e3b44ae745900aa93c49a77 not found: ID does not exist" containerID="a15dd5ee7fe09737a7b6c1f268296cd431acf1e44e3b44ae745900aa93c49a77"
Jul 01 19:37:58.999405 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.999337    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a15dd5ee7fe09737a7b6c1f268296cd431acf1e44e3b44ae745900aa93c49a77"} err="failed to get container status \"a15dd5ee7fe09737a7b6c1f268296cd431acf1e44e3b44ae745900aa93c49a77\": rpc error: code = NotFound desc = could not find container \"a15dd5ee7fe09737a7b6c1f268296cd431acf1e44e3b44ae745900aa93c49a77\": container with ID starting with a15dd5ee7fe09737a7b6c1f268296cd431acf1e44e3b44ae745900aa93c49a77 not found: ID does not exist"
Jul 01 19:37:58.999405 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.999362    2574 scope.go:117] "RemoveContainer" containerID="640c1d56fb44aaf34b879b1cd07699a0d8f6f6dd35bcc7aca1a15c98f6dbc845"
Jul 01 19:37:58.999405 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.999389    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45d7a420-0414-48eb-af91-949b2f67ff63-kube-api-access-cxfx6" (OuterVolumeSpecName: "kube-api-access-cxfx6") pod "45d7a420-0414-48eb-af91-949b2f67ff63" (UID: "45d7a420-0414-48eb-af91-949b2f67ff63"). InnerVolumeSpecName "kube-api-access-cxfx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:37:59.000016 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:58.999444    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45d7a420-0414-48eb-af91-949b2f67ff63-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "45d7a420-0414-48eb-af91-949b2f67ff63" (UID: "45d7a420-0414-48eb-af91-949b2f67ff63"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:37:59.000086 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:37:59.000048    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"640c1d56fb44aaf34b879b1cd07699a0d8f6f6dd35bcc7aca1a15c98f6dbc845\": container with ID starting with 640c1d56fb44aaf34b879b1cd07699a0d8f6f6dd35bcc7aca1a15c98f6dbc845 not found: ID does not exist" containerID="640c1d56fb44aaf34b879b1cd07699a0d8f6f6dd35bcc7aca1a15c98f6dbc845"
Jul 01 19:37:59.000086 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:59.000075    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"640c1d56fb44aaf34b879b1cd07699a0d8f6f6dd35bcc7aca1a15c98f6dbc845"} err="failed to get container status \"640c1d56fb44aaf34b879b1cd07699a0d8f6f6dd35bcc7aca1a15c98f6dbc845\": rpc error: code = NotFound desc = could not find container \"640c1d56fb44aaf34b879b1cd07699a0d8f6f6dd35bcc7aca1a15c98f6dbc845\": container with ID starting with 640c1d56fb44aaf34b879b1cd07699a0d8f6f6dd35bcc7aca1a15c98f6dbc845 not found: ID does not exist"
Jul 01 19:37:59.000163 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:59.000092    2574 scope.go:117] "RemoveContainer" containerID="a15dd5ee7fe09737a7b6c1f268296cd431acf1e44e3b44ae745900aa93c49a77"
Jul 01 19:37:59.000358 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:59.000334    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a15dd5ee7fe09737a7b6c1f268296cd431acf1e44e3b44ae745900aa93c49a77"} err="failed to get container status \"a15dd5ee7fe09737a7b6c1f268296cd431acf1e44e3b44ae745900aa93c49a77\": rpc error: code = NotFound desc = could not find container \"a15dd5ee7fe09737a7b6c1f268296cd431acf1e44e3b44ae745900aa93c49a77\": container with ID starting with a15dd5ee7fe09737a7b6c1f268296cd431acf1e44e3b44ae745900aa93c49a77 not found: ID does not exist"
Jul 01 19:37:59.000451 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:59.000358    2574 scope.go:117] "RemoveContainer" containerID="640c1d56fb44aaf34b879b1cd07699a0d8f6f6dd35bcc7aca1a15c98f6dbc845"
Jul 01 19:37:59.000611 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:59.000592    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"640c1d56fb44aaf34b879b1cd07699a0d8f6f6dd35bcc7aca1a15c98f6dbc845"} err="failed to get container status \"640c1d56fb44aaf34b879b1cd07699a0d8f6f6dd35bcc7aca1a15c98f6dbc845\": rpc error: code = NotFound desc = could not find container \"640c1d56fb44aaf34b879b1cd07699a0d8f6f6dd35bcc7aca1a15c98f6dbc845\": container with ID starting with 640c1d56fb44aaf34b879b1cd07699a0d8f6f6dd35bcc7aca1a15c98f6dbc845 not found: ID does not exist"
Jul 01 19:37:59.098865 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:59.098829    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45d7a420-0414-48eb-af91-949b2f67ff63-proxy-tls\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:37:59.098865 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:59.098859    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxfx6\" (UniqueName: \"kubernetes.io/projected/45d7a420-0414-48eb-af91-949b2f67ff63-kube-api-access-cxfx6\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:37:59.098865 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:59.098870    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45d7a420-0414-48eb-af91-949b2f67ff63-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:37:59.300671 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:59.300641    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8a5ec24-3a26-4c1f-859f-0f58c21010d5-proxy-tls\") pod \"isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b\" (UID: \"d8a5ec24-3a26-4c1f-859f-0f58c21010d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b"
Jul 01 19:37:59.303420 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:59.303394    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8a5ec24-3a26-4c1f-859f-0f58c21010d5-proxy-tls\") pod \"isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b\" (UID: \"d8a5ec24-3a26-4c1f-859f-0f58c21010d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b"
Jul 01 19:37:59.304049 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:59.303997    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-553af-predictor-fdf784bb4-h4pqq"]
Jul 01 19:37:59.305766 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:59.305743    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-553af-predictor-fdf784bb4-h4pqq"]
Jul 01 19:37:59.520295 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:59.520260    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b"
Jul 01 19:37:59.656859 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:59.656825    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b"]
Jul 01 19:37:59.658162 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:37:59.658135    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8a5ec24_3a26_4c1f_859f_0f58c21010d5.slice/crio-4c7630bc01e03e9ad132a125c254fcb620dd574e9b471192ab66199598a97f1e WatchSource:0}: Error finding container 4c7630bc01e03e9ad132a125c254fcb620dd574e9b471192ab66199598a97f1e: Status 404 returned error can't find the container with id 4c7630bc01e03e9ad132a125c254fcb620dd574e9b471192ab66199598a97f1e
Jul 01 19:37:59.990234 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:59.990198    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b" event={"ID":"d8a5ec24-3a26-4c1f-859f-0f58c21010d5","Type":"ContainerStarted","Data":"81321d5a47a01280c9502c3d48d0a8440fee6ba43d0a27e0d1d1c32bde950547"}
Jul 01 19:37:59.990440 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:37:59.990269    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b" event={"ID":"d8a5ec24-3a26-4c1f-859f-0f58c21010d5","Type":"ContainerStarted","Data":"4c7630bc01e03e9ad132a125c254fcb620dd574e9b471192ab66199598a97f1e"}
Jul 01 19:38:00.357652 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:00.357609    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45d7a420-0414-48eb-af91-949b2f67ff63" path="/var/lib/kubelet/pods/45d7a420-0414-48eb-af91-949b2f67ff63/volumes"
Jul 01 19:38:01.573116 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:01.573077    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Jul 01 19:38:04.010920 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:04.010877    2574 generic.go:356] "Generic (PLEG): container finished" podID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerID="1177eadf365a1c52f51ce033a6d9bccc41ac46dff638ad3de40de920a7da7274" exitCode=0
Jul 01 19:38:04.011343 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:04.010949    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" event={"ID":"4b4c4555-8e4a-4a46-8ecb-779c792c0955","Type":"ContainerDied","Data":"1177eadf365a1c52f51ce033a6d9bccc41ac46dff638ad3de40de920a7da7274"}
Jul 01 19:38:04.012336 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:04.012315    2574 generic.go:356] "Generic (PLEG): container finished" podID="d8a5ec24-3a26-4c1f-859f-0f58c21010d5" containerID="81321d5a47a01280c9502c3d48d0a8440fee6ba43d0a27e0d1d1c32bde950547" exitCode=0
Jul 01 19:38:04.012427 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:04.012354    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b" event={"ID":"d8a5ec24-3a26-4c1f-859f-0f58c21010d5","Type":"ContainerDied","Data":"81321d5a47a01280c9502c3d48d0a8440fee6ba43d0a27e0d1d1c32bde950547"}
Jul 01 19:38:05.019073 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:05.019037    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b" event={"ID":"d8a5ec24-3a26-4c1f-859f-0f58c21010d5","Type":"ContainerStarted","Data":"72d10e8f83900be4e331fc53e3b2d02783720784a3a5c3c12d21667516fedfcc"}
Jul 01 19:38:05.019473 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:05.019106    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b" event={"ID":"d8a5ec24-3a26-4c1f-859f-0f58c21010d5","Type":"ContainerStarted","Data":"ba8d5c0433f29fd1c1e14002d9c31b90fb944b00e87f2b44faecc1f479e94680"}
Jul 01 19:38:05.019473 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:05.019296    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b"
Jul 01 19:38:05.040563 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:05.040509    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b" podStartSLOduration=3.341322537 podStartE2EDuration="7.040494356s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.699171819s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:37:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:38:05.037862997 +0000 UTC m=+991.129862447" watchObservedRunningTime="2026-07-01 19:38:05.040494356 +0000 UTC m=+991.132493803"
Jul 01 19:38:06.024319 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:06.024279    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b"
Jul 01 19:38:06.025554 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:06.025528    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b" podUID="d8a5ec24-3a26-4c1f-859f-0f58c21010d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 01 19:38:06.572722 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:06.572684    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Jul 01 19:38:06.578518 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:06.578484    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:38:06.578644 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:06.578592    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 01 19:38:07.029195 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:07.029157    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b" podUID="d8a5ec24-3a26-4c1f-859f-0f58c21010d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 01 19:38:11.572639 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:11.572597    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Jul 01 19:38:11.573148 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:11.572715    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5"
Jul 01 19:38:12.034915 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:12.034872    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b"
Jul 01 19:38:12.035468 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:12.035439    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b" podUID="d8a5ec24-3a26-4c1f-859f-0f58c21010d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 01 19:38:16.572999 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:16.572954    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Jul 01 19:38:16.577597 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:16.577560    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:38:16.578691 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:16.578667    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 01 19:38:21.572584 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:21.572546    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Jul 01 19:38:22.036111 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:22.036073    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b" podUID="d8a5ec24-3a26-4c1f-859f-0f58c21010d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 01 19:38:26.572617 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:26.572567    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Jul 01 19:38:26.578376 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:26.578342    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:38:26.578520 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:26.578485    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5"
Jul 01 19:38:26.578520 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:26.578501    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 01 19:38:26.578715 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:26.578696    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5"
Jul 01 19:38:28.745160 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:28.745131    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5"
Jul 01 19:38:28.856405 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:28.856374    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b4c4555-8e4a-4a46-8ecb-779c792c0955-kserve-provision-location\") pod \"4b4c4555-8e4a-4a46-8ecb-779c792c0955\" (UID: \"4b4c4555-8e4a-4a46-8ecb-779c792c0955\") "
Jul 01 19:38:28.856564 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:28.856411    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxbhp\" (UniqueName: \"kubernetes.io/projected/4b4c4555-8e4a-4a46-8ecb-779c792c0955-kube-api-access-zxbhp\") pod \"4b4c4555-8e4a-4a46-8ecb-779c792c0955\" (UID: \"4b4c4555-8e4a-4a46-8ecb-779c792c0955\") "
Jul 01 19:38:28.856564 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:28.856459    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b4c4555-8e4a-4a46-8ecb-779c792c0955-cabundle-cert\") pod \"4b4c4555-8e4a-4a46-8ecb-779c792c0955\" (UID: \"4b4c4555-8e4a-4a46-8ecb-779c792c0955\") "
Jul 01 19:38:28.856564 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:28.856514    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b4c4555-8e4a-4a46-8ecb-779c792c0955-kube-rbac-proxy-sar-config\") pod \"4b4c4555-8e4a-4a46-8ecb-779c792c0955\" (UID: \"4b4c4555-8e4a-4a46-8ecb-779c792c0955\") "
Jul 01 19:38:28.856564 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:28.856557    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b4c4555-8e4a-4a46-8ecb-779c792c0955-proxy-tls\") pod \"4b4c4555-8e4a-4a46-8ecb-779c792c0955\" (UID: \"4b4c4555-8e4a-4a46-8ecb-779c792c0955\") "
Jul 01 19:38:28.856823 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:28.856784    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b4c4555-8e4a-4a46-8ecb-779c792c0955-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4b4c4555-8e4a-4a46-8ecb-779c792c0955" (UID: "4b4c4555-8e4a-4a46-8ecb-779c792c0955"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:38:28.856976 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:28.856925    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b4c4555-8e4a-4a46-8ecb-779c792c0955-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4b4c4555-8e4a-4a46-8ecb-779c792c0955" (UID: "4b4c4555-8e4a-4a46-8ecb-779c792c0955"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:38:28.856976 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:28.856944    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b4c4555-8e4a-4a46-8ecb-779c792c0955-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4b4c4555-8e4a-4a46-8ecb-779c792c0955" (UID: "4b4c4555-8e4a-4a46-8ecb-779c792c0955"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:38:28.858581 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:28.858561    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b4c4555-8e4a-4a46-8ecb-779c792c0955-kube-api-access-zxbhp" (OuterVolumeSpecName: "kube-api-access-zxbhp") pod "4b4c4555-8e4a-4a46-8ecb-779c792c0955" (UID: "4b4c4555-8e4a-4a46-8ecb-779c792c0955"). InnerVolumeSpecName "kube-api-access-zxbhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:38:28.858659 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:28.858607    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b4c4555-8e4a-4a46-8ecb-779c792c0955-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4b4c4555-8e4a-4a46-8ecb-779c792c0955" (UID: "4b4c4555-8e4a-4a46-8ecb-779c792c0955"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:38:28.958027 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:28.957988    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b4c4555-8e4a-4a46-8ecb-779c792c0955-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:38:28.958027 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:28.958021    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b4c4555-8e4a-4a46-8ecb-779c792c0955-proxy-tls\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:38:28.958027 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:28.958032    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b4c4555-8e4a-4a46-8ecb-779c792c0955-kserve-provision-location\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:38:28.958276 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:28.958042    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxbhp\" (UniqueName: \"kubernetes.io/projected/4b4c4555-8e4a-4a46-8ecb-779c792c0955-kube-api-access-zxbhp\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:38:28.958276 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:28.958051    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b4c4555-8e4a-4a46-8ecb-779c792c0955-cabundle-cert\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:38:29.131430 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:29.131350    2574 generic.go:356] "Generic (PLEG): container finished" podID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerID="41b7c541de3b4fb5066e8800b25438c7edab3d72934baab204d38dbfafce4357" exitCode=0
Jul 01 19:38:29.131430 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:29.131408    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5"
Jul 01 19:38:29.131430 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:29.131417    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" event={"ID":"4b4c4555-8e4a-4a46-8ecb-779c792c0955","Type":"ContainerDied","Data":"41b7c541de3b4fb5066e8800b25438c7edab3d72934baab204d38dbfafce4357"}
Jul 01 19:38:29.131655 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:29.131470    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5" event={"ID":"4b4c4555-8e4a-4a46-8ecb-779c792c0955","Type":"ContainerDied","Data":"71b27f9c5fdf2f0ce7850806a7948d2c715ebce27e2762c879364953f2d392e2"}
Jul 01 19:38:29.131655 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:29.131499    2574 scope.go:117] "RemoveContainer" containerID="41b7c541de3b4fb5066e8800b25438c7edab3d72934baab204d38dbfafce4357"
Jul 01 19:38:29.140120 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:29.140099    2574 scope.go:117] "RemoveContainer" containerID="88ca6624e1ba02a3235533c33c099fcea2863459d8cfc34c4c7c4ae718ef5cd9"
Jul 01 19:38:29.147341 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:29.147324    2574 scope.go:117] "RemoveContainer" containerID="1177eadf365a1c52f51ce033a6d9bccc41ac46dff638ad3de40de920a7da7274"
Jul 01 19:38:29.153573 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:29.153548    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5"]
Jul 01 19:38:29.155074 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:29.155052    2574 scope.go:117] "RemoveContainer" containerID="7e63d63d7132f289405417fa1373864c406cd67680abda9a3f3bcee4040a95d5"
Jul 01 19:38:29.157852 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:29.157828    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-553af-predictor-558fb6ddb6-4rnf5"]
Jul 01 19:38:29.163423 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:29.163403    2574 scope.go:117] "RemoveContainer" containerID="41b7c541de3b4fb5066e8800b25438c7edab3d72934baab204d38dbfafce4357"
Jul 01 19:38:29.163696 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:38:29.163670    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41b7c541de3b4fb5066e8800b25438c7edab3d72934baab204d38dbfafce4357\": container with ID starting with 41b7c541de3b4fb5066e8800b25438c7edab3d72934baab204d38dbfafce4357 not found: ID does not exist" containerID="41b7c541de3b4fb5066e8800b25438c7edab3d72934baab204d38dbfafce4357"
Jul 01 19:38:29.163750 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:29.163705    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41b7c541de3b4fb5066e8800b25438c7edab3d72934baab204d38dbfafce4357"} err="failed to get container status \"41b7c541de3b4fb5066e8800b25438c7edab3d72934baab204d38dbfafce4357\": rpc error: code = NotFound desc = could not find container \"41b7c541de3b4fb5066e8800b25438c7edab3d72934baab204d38dbfafce4357\": container with ID starting with 41b7c541de3b4fb5066e8800b25438c7edab3d72934baab204d38dbfafce4357 not found: ID does not exist"
Jul 01 19:38:29.163750 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:29.163722    2574 scope.go:117] "RemoveContainer" containerID="88ca6624e1ba02a3235533c33c099fcea2863459d8cfc34c4c7c4ae718ef5cd9"
Jul 01 19:38:29.163977 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:38:29.163959    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88ca6624e1ba02a3235533c33c099fcea2863459d8cfc34c4c7c4ae718ef5cd9\": container with ID starting with 88ca6624e1ba02a3235533c33c099fcea2863459d8cfc34c4c7c4ae718ef5cd9 not found: ID does not exist" containerID="88ca6624e1ba02a3235533c33c099fcea2863459d8cfc34c4c7c4ae718ef5cd9"
Jul 01 19:38:29.164024 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:29.163984    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88ca6624e1ba02a3235533c33c099fcea2863459d8cfc34c4c7c4ae718ef5cd9"} err="failed to get container status \"88ca6624e1ba02a3235533c33c099fcea2863459d8cfc34c4c7c4ae718ef5cd9\": rpc error: code = NotFound desc = could not find container \"88ca6624e1ba02a3235533c33c099fcea2863459d8cfc34c4c7c4ae718ef5cd9\": container with ID starting with 88ca6624e1ba02a3235533c33c099fcea2863459d8cfc34c4c7c4ae718ef5cd9 not found: ID does not exist"
Jul 01 19:38:29.164024 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:29.164000    2574 scope.go:117] "RemoveContainer" containerID="1177eadf365a1c52f51ce033a6d9bccc41ac46dff638ad3de40de920a7da7274"
Jul 01 19:38:29.164215 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:38:29.164200    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1177eadf365a1c52f51ce033a6d9bccc41ac46dff638ad3de40de920a7da7274\": container with ID starting with 1177eadf365a1c52f51ce033a6d9bccc41ac46dff638ad3de40de920a7da7274 not found: ID does not exist" containerID="1177eadf365a1c52f51ce033a6d9bccc41ac46dff638ad3de40de920a7da7274"
Jul 01 19:38:29.164309 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:29.164219    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1177eadf365a1c52f51ce033a6d9bccc41ac46dff638ad3de40de920a7da7274"} err="failed to get container status \"1177eadf365a1c52f51ce033a6d9bccc41ac46dff638ad3de40de920a7da7274\": rpc error: code = NotFound desc = could not find container \"1177eadf365a1c52f51ce033a6d9bccc41ac46dff638ad3de40de920a7da7274\": container with ID starting with 1177eadf365a1c52f51ce033a6d9bccc41ac46dff638ad3de40de920a7da7274 not found: ID does not exist"
Jul 01 19:38:29.164309 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:29.164231    2574 scope.go:117] "RemoveContainer" containerID="7e63d63d7132f289405417fa1373864c406cd67680abda9a3f3bcee4040a95d5"
Jul 01 19:38:29.164472 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:38:29.164457    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e63d63d7132f289405417fa1373864c406cd67680abda9a3f3bcee4040a95d5\": container with ID starting with 7e63d63d7132f289405417fa1373864c406cd67680abda9a3f3bcee4040a95d5 not found: ID does not exist" containerID="7e63d63d7132f289405417fa1373864c406cd67680abda9a3f3bcee4040a95d5"
Jul 01 19:38:29.164514 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:29.164475    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e63d63d7132f289405417fa1373864c406cd67680abda9a3f3bcee4040a95d5"} err="failed to get container status \"7e63d63d7132f289405417fa1373864c406cd67680abda9a3f3bcee4040a95d5\": rpc error: code = NotFound desc = could not find container \"7e63d63d7132f289405417fa1373864c406cd67680abda9a3f3bcee4040a95d5\": container with ID starting with 7e63d63d7132f289405417fa1373864c406cd67680abda9a3f3bcee4040a95d5 not found: ID does not exist"
Jul 01 19:38:30.356400 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:30.356367    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" path="/var/lib/kubelet/pods/4b4c4555-8e4a-4a46-8ecb-779c792c0955/volumes"
Jul 01 19:38:32.035971 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:32.035926    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b" podUID="d8a5ec24-3a26-4c1f-859f-0f58c21010d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 01 19:38:42.035994 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:42.035957    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b" podUID="d8a5ec24-3a26-4c1f-859f-0f58c21010d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 01 19:38:52.036281 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:38:52.036240    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b" podUID="d8a5ec24-3a26-4c1f-859f-0f58c21010d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 01 19:39:02.035587 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:39:02.035539    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b" podUID="d8a5ec24-3a26-4c1f-859f-0f58c21010d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 01 19:39:12.036144 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:39:12.036102    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b" podUID="d8a5ec24-3a26-4c1f-859f-0f58c21010d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 01 19:39:13.351300 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:39:13.351255    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b" podUID="d8a5ec24-3a26-4c1f-859f-0f58c21010d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 01 19:39:23.351193 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:39:23.351155    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b" podUID="d8a5ec24-3a26-4c1f-859f-0f58c21010d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 01 19:39:33.351508 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:39:33.351416    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b" podUID="d8a5ec24-3a26-4c1f-859f-0f58c21010d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 01 19:39:43.351982 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:39:43.351935    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b" podUID="d8a5ec24-3a26-4c1f-859f-0f58c21010d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 01 19:39:53.351235 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:39:53.351183    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b" podUID="d8a5ec24-3a26-4c1f-859f-0f58c21010d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 01 19:40:03.351432 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:03.351391    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b" podUID="d8a5ec24-3a26-4c1f-859f-0f58c21010d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 01 19:40:13.351673 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:13.351625    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b" podUID="d8a5ec24-3a26-4c1f-859f-0f58c21010d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 01 19:40:23.351982 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:23.351937    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b" podUID="d8a5ec24-3a26-4c1f-859f-0f58c21010d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 01 19:40:25.351824 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:25.351773    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b" podUID="d8a5ec24-3a26-4c1f-859f-0f58c21010d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 01 19:40:35.352489 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:35.352443    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b" podUID="d8a5ec24-3a26-4c1f-859f-0f58c21010d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 01 19:40:45.352972 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:45.352939    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b"
Jul 01 19:40:48.739398 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:48.739363    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b"]
Jul 01 19:40:48.739832 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:48.739650    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b" podUID="d8a5ec24-3a26-4c1f-859f-0f58c21010d5" containerName="kserve-container" containerID="cri-o://ba8d5c0433f29fd1c1e14002d9c31b90fb944b00e87f2b44faecc1f479e94680" gracePeriod=30
Jul 01 19:40:48.739832 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:48.739707    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b" podUID="d8a5ec24-3a26-4c1f-859f-0f58c21010d5" containerName="kube-rbac-proxy" containerID="cri-o://72d10e8f83900be4e331fc53e3b2d02783720784a3a5c3c12d21667516fedfcc" gracePeriod=30
Jul 01 19:40:49.778580 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:49.778545    2574 generic.go:356] "Generic (PLEG): container finished" podID="d8a5ec24-3a26-4c1f-859f-0f58c21010d5" containerID="72d10e8f83900be4e331fc53e3b2d02783720784a3a5c3c12d21667516fedfcc" exitCode=2
Jul 01 19:40:49.778979 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:49.778618    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b" event={"ID":"d8a5ec24-3a26-4c1f-859f-0f58c21010d5","Type":"ContainerDied","Data":"72d10e8f83900be4e331fc53e3b2d02783720784a3a5c3c12d21667516fedfcc"}
Jul 01 19:40:52.029393 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:52.029355    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b" podUID="d8a5ec24-3a26-4c1f-859f-0f58c21010d5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Jul 01 19:40:53.873846 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:53.873809    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk"]
Jul 01 19:40:53.874276 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:53.874254    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="kserve-container"
Jul 01 19:40:53.874276 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:53.874269    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="kserve-container"
Jul 01 19:40:53.874371 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:53.874280    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="agent"
Jul 01 19:40:53.874371 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:53.874285    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="agent"
Jul 01 19:40:53.874371 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:53.874297    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45d7a420-0414-48eb-af91-949b2f67ff63" containerName="kube-rbac-proxy"
Jul 01 19:40:53.874371 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:53.874303    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="45d7a420-0414-48eb-af91-949b2f67ff63" containerName="kube-rbac-proxy"
Jul 01 19:40:53.874371 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:53.874311    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="storage-initializer"
Jul 01 19:40:53.874371 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:53.874316    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="storage-initializer"
Jul 01 19:40:53.874371 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:53.874323    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="kube-rbac-proxy"
Jul 01 19:40:53.874371 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:53.874328    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="kube-rbac-proxy"
Jul 01 19:40:53.874371 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:53.874351    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45d7a420-0414-48eb-af91-949b2f67ff63" containerName="kserve-container"
Jul 01 19:40:53.874371 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:53.874361    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="45d7a420-0414-48eb-af91-949b2f67ff63" containerName="kserve-container"
Jul 01 19:40:53.874668 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:53.874417    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="45d7a420-0414-48eb-af91-949b2f67ff63" containerName="kserve-container"
Jul 01 19:40:53.874668 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:53.874429    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="kube-rbac-proxy"
Jul 01 19:40:53.874668 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:53.874440    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="agent"
Jul 01 19:40:53.874668 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:53.874446    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="45d7a420-0414-48eb-af91-949b2f67ff63" containerName="kube-rbac-proxy"
Jul 01 19:40:53.874668 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:53.874452    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b4c4555-8e4a-4a46-8ecb-779c792c0955" containerName="kserve-container"
Jul 01 19:40:53.877519 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:53.877503    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk"
Jul 01 19:40:53.879420 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:53.879394    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-6ef841-predictor-serving-cert\""
Jul 01 19:40:53.879551 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:53.879533    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-6ef841-kube-rbac-proxy-sar-config\""
Jul 01 19:40:53.886454 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:53.886432    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk"]
Jul 01 19:40:53.969877 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:53.969849    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctbpw\" (UniqueName: \"kubernetes.io/projected/fee55a5f-1926-488e-afe7-8678e24e8e8a-kube-api-access-ctbpw\") pod \"isvc-primary-6ef841-predictor-7f47d7b458-bz4zk\" (UID: \"fee55a5f-1926-488e-afe7-8678e24e8e8a\") " pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk"
Jul 01 19:40:53.970056 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:53.969888    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fee55a5f-1926-488e-afe7-8678e24e8e8a-proxy-tls\") pod \"isvc-primary-6ef841-predictor-7f47d7b458-bz4zk\" (UID: \"fee55a5f-1926-488e-afe7-8678e24e8e8a\") " pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk"
Jul 01 19:40:53.970056 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:53.969935    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fee55a5f-1926-488e-afe7-8678e24e8e8a-kube-rbac-proxy-sar-config\") pod \"isvc-primary-6ef841-predictor-7f47d7b458-bz4zk\" (UID: \"fee55a5f-1926-488e-afe7-8678e24e8e8a\") " pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk"
Jul 01 19:40:53.970056 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:53.969955    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fee55a5f-1926-488e-afe7-8678e24e8e8a-cabundle-cert\") pod \"isvc-primary-6ef841-predictor-7f47d7b458-bz4zk\" (UID: \"fee55a5f-1926-488e-afe7-8678e24e8e8a\") " pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk"
Jul 01 19:40:53.970056 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:53.969977    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fee55a5f-1926-488e-afe7-8678e24e8e8a-kserve-provision-location\") pod \"isvc-primary-6ef841-predictor-7f47d7b458-bz4zk\" (UID: \"fee55a5f-1926-488e-afe7-8678e24e8e8a\") " pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk"
Jul 01 19:40:54.070959 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:54.070896    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctbpw\" (UniqueName: \"kubernetes.io/projected/fee55a5f-1926-488e-afe7-8678e24e8e8a-kube-api-access-ctbpw\") pod \"isvc-primary-6ef841-predictor-7f47d7b458-bz4zk\" (UID: \"fee55a5f-1926-488e-afe7-8678e24e8e8a\") " pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk"
Jul 01 19:40:54.071146 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:54.070978    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fee55a5f-1926-488e-afe7-8678e24e8e8a-proxy-tls\") pod \"isvc-primary-6ef841-predictor-7f47d7b458-bz4zk\" (UID: \"fee55a5f-1926-488e-afe7-8678e24e8e8a\") " pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk"
Jul 01 19:40:54.071146 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:54.071008    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fee55a5f-1926-488e-afe7-8678e24e8e8a-kube-rbac-proxy-sar-config\") pod \"isvc-primary-6ef841-predictor-7f47d7b458-bz4zk\" (UID: \"fee55a5f-1926-488e-afe7-8678e24e8e8a\") " pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk"
Jul 01 19:40:54.071146 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:54.071029    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fee55a5f-1926-488e-afe7-8678e24e8e8a-cabundle-cert\") pod \"isvc-primary-6ef841-predictor-7f47d7b458-bz4zk\" (UID: \"fee55a5f-1926-488e-afe7-8678e24e8e8a\") " pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk"
Jul 01 19:40:54.071146 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:54.071055    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fee55a5f-1926-488e-afe7-8678e24e8e8a-kserve-provision-location\") pod \"isvc-primary-6ef841-predictor-7f47d7b458-bz4zk\" (UID: \"fee55a5f-1926-488e-afe7-8678e24e8e8a\") " pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk"
Jul 01 19:40:54.071373 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:40:54.071172    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-6ef841-predictor-serving-cert: secret "isvc-primary-6ef841-predictor-serving-cert" not found
Jul 01 19:40:54.071373 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:40:54.071258    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fee55a5f-1926-488e-afe7-8678e24e8e8a-proxy-tls podName:fee55a5f-1926-488e-afe7-8678e24e8e8a nodeName:}" failed. No retries permitted until 2026-07-01 19:40:54.571239787 +0000 UTC m=+1160.663239225 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fee55a5f-1926-488e-afe7-8678e24e8e8a-proxy-tls") pod "isvc-primary-6ef841-predictor-7f47d7b458-bz4zk" (UID: "fee55a5f-1926-488e-afe7-8678e24e8e8a") : secret "isvc-primary-6ef841-predictor-serving-cert" not found
Jul 01 19:40:54.071569 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:54.071548    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fee55a5f-1926-488e-afe7-8678e24e8e8a-kserve-provision-location\") pod \"isvc-primary-6ef841-predictor-7f47d7b458-bz4zk\" (UID: \"fee55a5f-1926-488e-afe7-8678e24e8e8a\") " pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk"
Jul 01 19:40:54.071747 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:54.071725    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fee55a5f-1926-488e-afe7-8678e24e8e8a-cabundle-cert\") pod \"isvc-primary-6ef841-predictor-7f47d7b458-bz4zk\" (UID: \"fee55a5f-1926-488e-afe7-8678e24e8e8a\") " pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk"
Jul 01 19:40:54.071813 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:54.071763    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fee55a5f-1926-488e-afe7-8678e24e8e8a-kube-rbac-proxy-sar-config\") pod \"isvc-primary-6ef841-predictor-7f47d7b458-bz4zk\" (UID: \"fee55a5f-1926-488e-afe7-8678e24e8e8a\") " pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk"
Jul 01 19:40:54.079631 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:54.079607    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctbpw\" (UniqueName: \"kubernetes.io/projected/fee55a5f-1926-488e-afe7-8678e24e8e8a-kube-api-access-ctbpw\") pod \"isvc-primary-6ef841-predictor-7f47d7b458-bz4zk\" (UID: \"fee55a5f-1926-488e-afe7-8678e24e8e8a\") " pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk"
Jul 01 19:40:54.576089 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:54.576055    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fee55a5f-1926-488e-afe7-8678e24e8e8a-proxy-tls\") pod \"isvc-primary-6ef841-predictor-7f47d7b458-bz4zk\" (UID: \"fee55a5f-1926-488e-afe7-8678e24e8e8a\") " pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk"
Jul 01 19:40:54.576260 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:40:54.576182    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-6ef841-predictor-serving-cert: secret "isvc-primary-6ef841-predictor-serving-cert" not found
Jul 01 19:40:54.576260 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:40:54.576248    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fee55a5f-1926-488e-afe7-8678e24e8e8a-proxy-tls podName:fee55a5f-1926-488e-afe7-8678e24e8e8a nodeName:}" failed. No retries permitted until 2026-07-01 19:40:55.576232743 +0000 UTC m=+1161.668232169 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fee55a5f-1926-488e-afe7-8678e24e8e8a-proxy-tls") pod "isvc-primary-6ef841-predictor-7f47d7b458-bz4zk" (UID: "fee55a5f-1926-488e-afe7-8678e24e8e8a") : secret "isvc-primary-6ef841-predictor-serving-cert" not found
Jul 01 19:40:55.352566 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:55.352527    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b" podUID="d8a5ec24-3a26-4c1f-859f-0f58c21010d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 01 19:40:55.585281 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:55.585200    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fee55a5f-1926-488e-afe7-8678e24e8e8a-proxy-tls\") pod \"isvc-primary-6ef841-predictor-7f47d7b458-bz4zk\" (UID: \"fee55a5f-1926-488e-afe7-8678e24e8e8a\") " pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk"
Jul 01 19:40:55.587678 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:55.587655    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fee55a5f-1926-488e-afe7-8678e24e8e8a-proxy-tls\") pod \"isvc-primary-6ef841-predictor-7f47d7b458-bz4zk\" (UID: \"fee55a5f-1926-488e-afe7-8678e24e8e8a\") " pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk"
Jul 01 19:40:55.688135 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:55.688051    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk"
Jul 01 19:40:55.815891 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:55.815867    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk"]
Jul 01 19:40:55.817712 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:40:55.817685    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfee55a5f_1926_488e_afe7_8678e24e8e8a.slice/crio-7975ce3b4eec79629b5b87ceab6b1ce79c06c0f7728b26f837bca22870a6d265 WatchSource:0}: Error finding container 7975ce3b4eec79629b5b87ceab6b1ce79c06c0f7728b26f837bca22870a6d265: Status 404 returned error can't find the container with id 7975ce3b4eec79629b5b87ceab6b1ce79c06c0f7728b26f837bca22870a6d265
Jul 01 19:40:56.813967 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:56.813926    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk" event={"ID":"fee55a5f-1926-488e-afe7-8678e24e8e8a","Type":"ContainerStarted","Data":"c06b3d634e4ac540a40cd3397eb5649f70bc17ca63c7fa5290311137de1bdb49"}
Jul 01 19:40:56.814359 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:56.814006    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk" event={"ID":"fee55a5f-1926-488e-afe7-8678e24e8e8a","Type":"ContainerStarted","Data":"7975ce3b4eec79629b5b87ceab6b1ce79c06c0f7728b26f837bca22870a6d265"}
Jul 01 19:40:57.030030 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:57.029982    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b" podUID="d8a5ec24-3a26-4c1f-859f-0f58c21010d5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Jul 01 19:40:58.587943 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:58.587917    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b"
Jul 01 19:40:58.711663 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:58.711582    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8a5ec24-3a26-4c1f-859f-0f58c21010d5-proxy-tls\") pod \"d8a5ec24-3a26-4c1f-859f-0f58c21010d5\" (UID: \"d8a5ec24-3a26-4c1f-859f-0f58c21010d5\") "
Jul 01 19:40:58.711663 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:58.711632    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djs6g\" (UniqueName: \"kubernetes.io/projected/d8a5ec24-3a26-4c1f-859f-0f58c21010d5-kube-api-access-djs6g\") pod \"d8a5ec24-3a26-4c1f-859f-0f58c21010d5\" (UID: \"d8a5ec24-3a26-4c1f-859f-0f58c21010d5\") "
Jul 01 19:40:58.711896 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:58.711677    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8a5ec24-3a26-4c1f-859f-0f58c21010d5-kube-rbac-proxy-sar-config\") pod \"d8a5ec24-3a26-4c1f-859f-0f58c21010d5\" (UID: \"d8a5ec24-3a26-4c1f-859f-0f58c21010d5\") "
Jul 01 19:40:58.711896 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:58.711696    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d8a5ec24-3a26-4c1f-859f-0f58c21010d5-cabundle-cert\") pod \"d8a5ec24-3a26-4c1f-859f-0f58c21010d5\" (UID: \"d8a5ec24-3a26-4c1f-859f-0f58c21010d5\") "
Jul 01 19:40:58.711896 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:58.711733    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8a5ec24-3a26-4c1f-859f-0f58c21010d5-kserve-provision-location\") pod \"d8a5ec24-3a26-4c1f-859f-0f58c21010d5\" (UID: \"d8a5ec24-3a26-4c1f-859f-0f58c21010d5\") "
Jul 01 19:40:58.712166 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:58.712140    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8a5ec24-3a26-4c1f-859f-0f58c21010d5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d8a5ec24-3a26-4c1f-859f-0f58c21010d5" (UID: "d8a5ec24-3a26-4c1f-859f-0f58c21010d5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:40:58.712233 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:58.712160    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8a5ec24-3a26-4c1f-859f-0f58c21010d5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d8a5ec24-3a26-4c1f-859f-0f58c21010d5" (UID: "d8a5ec24-3a26-4c1f-859f-0f58c21010d5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:40:58.712233 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:58.712138    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8a5ec24-3a26-4c1f-859f-0f58c21010d5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d8a5ec24-3a26-4c1f-859f-0f58c21010d5" (UID: "d8a5ec24-3a26-4c1f-859f-0f58c21010d5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:40:58.713820 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:58.713796    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8a5ec24-3a26-4c1f-859f-0f58c21010d5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d8a5ec24-3a26-4c1f-859f-0f58c21010d5" (UID: "d8a5ec24-3a26-4c1f-859f-0f58c21010d5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:40:58.713820 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:58.713804    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8a5ec24-3a26-4c1f-859f-0f58c21010d5-kube-api-access-djs6g" (OuterVolumeSpecName: "kube-api-access-djs6g") pod "d8a5ec24-3a26-4c1f-859f-0f58c21010d5" (UID: "d8a5ec24-3a26-4c1f-859f-0f58c21010d5"). InnerVolumeSpecName "kube-api-access-djs6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:40:58.813171 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:58.813132    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8a5ec24-3a26-4c1f-859f-0f58c21010d5-proxy-tls\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:40:58.813171 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:58.813163    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djs6g\" (UniqueName: \"kubernetes.io/projected/d8a5ec24-3a26-4c1f-859f-0f58c21010d5-kube-api-access-djs6g\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:40:58.813171 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:58.813173    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8a5ec24-3a26-4c1f-859f-0f58c21010d5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:40:58.813416 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:58.813194    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d8a5ec24-3a26-4c1f-859f-0f58c21010d5-cabundle-cert\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:40:58.813416 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:58.813204    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8a5ec24-3a26-4c1f-859f-0f58c21010d5-kserve-provision-location\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:40:58.824868 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:58.824839    2574 generic.go:356] "Generic (PLEG): container finished" podID="d8a5ec24-3a26-4c1f-859f-0f58c21010d5" containerID="ba8d5c0433f29fd1c1e14002d9c31b90fb944b00e87f2b44faecc1f479e94680" exitCode=0
Jul 01 19:40:58.825013 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:58.824910    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b"
Jul 01 19:40:58.825013 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:58.824913    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b" event={"ID":"d8a5ec24-3a26-4c1f-859f-0f58c21010d5","Type":"ContainerDied","Data":"ba8d5c0433f29fd1c1e14002d9c31b90fb944b00e87f2b44faecc1f479e94680"}
Jul 01 19:40:58.825013 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:58.824979    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b" event={"ID":"d8a5ec24-3a26-4c1f-859f-0f58c21010d5","Type":"ContainerDied","Data":"4c7630bc01e03e9ad132a125c254fcb620dd574e9b471192ab66199598a97f1e"}
Jul 01 19:40:58.825013 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:58.825008    2574 scope.go:117] "RemoveContainer" containerID="72d10e8f83900be4e331fc53e3b2d02783720784a3a5c3c12d21667516fedfcc"
Jul 01 19:40:58.833612 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:58.833595    2574 scope.go:117] "RemoveContainer" containerID="ba8d5c0433f29fd1c1e14002d9c31b90fb944b00e87f2b44faecc1f479e94680"
Jul 01 19:40:58.840933 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:58.840884    2574 scope.go:117] "RemoveContainer" containerID="81321d5a47a01280c9502c3d48d0a8440fee6ba43d0a27e0d1d1c32bde950547"
Jul 01 19:40:58.846496 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:58.846474    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b"]
Jul 01 19:40:58.848704 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:58.848685    2574 scope.go:117] "RemoveContainer" containerID="72d10e8f83900be4e331fc53e3b2d02783720784a3a5c3c12d21667516fedfcc"
Jul 01 19:40:58.849217 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:40:58.849072    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72d10e8f83900be4e331fc53e3b2d02783720784a3a5c3c12d21667516fedfcc\": container with ID starting with 72d10e8f83900be4e331fc53e3b2d02783720784a3a5c3c12d21667516fedfcc not found: ID does not exist" containerID="72d10e8f83900be4e331fc53e3b2d02783720784a3a5c3c12d21667516fedfcc"
Jul 01 19:40:58.849217 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:58.849107    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72d10e8f83900be4e331fc53e3b2d02783720784a3a5c3c12d21667516fedfcc"} err="failed to get container status \"72d10e8f83900be4e331fc53e3b2d02783720784a3a5c3c12d21667516fedfcc\": rpc error: code = NotFound desc = could not find container \"72d10e8f83900be4e331fc53e3b2d02783720784a3a5c3c12d21667516fedfcc\": container with ID starting with 72d10e8f83900be4e331fc53e3b2d02783720784a3a5c3c12d21667516fedfcc not found: ID does not exist"
Jul 01 19:40:58.849217 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:58.849129    2574 scope.go:117] "RemoveContainer" containerID="ba8d5c0433f29fd1c1e14002d9c31b90fb944b00e87f2b44faecc1f479e94680"
Jul 01 19:40:58.849651 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:40:58.849631    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba8d5c0433f29fd1c1e14002d9c31b90fb944b00e87f2b44faecc1f479e94680\": container with ID starting with ba8d5c0433f29fd1c1e14002d9c31b90fb944b00e87f2b44faecc1f479e94680 not found: ID does not exist" containerID="ba8d5c0433f29fd1c1e14002d9c31b90fb944b00e87f2b44faecc1f479e94680"
Jul 01 19:40:58.849692 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:58.849656    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba8d5c0433f29fd1c1e14002d9c31b90fb944b00e87f2b44faecc1f479e94680"} err="failed to get container status \"ba8d5c0433f29fd1c1e14002d9c31b90fb944b00e87f2b44faecc1f479e94680\": rpc error: code = NotFound desc = could not find container \"ba8d5c0433f29fd1c1e14002d9c31b90fb944b00e87f2b44faecc1f479e94680\": container with ID starting with ba8d5c0433f29fd1c1e14002d9c31b90fb944b00e87f2b44faecc1f479e94680 not found: ID does not exist"
Jul 01 19:40:58.849692 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:58.849671    2574 scope.go:117] "RemoveContainer" containerID="81321d5a47a01280c9502c3d48d0a8440fee6ba43d0a27e0d1d1c32bde950547"
Jul 01 19:40:58.849942 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:40:58.849895    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81321d5a47a01280c9502c3d48d0a8440fee6ba43d0a27e0d1d1c32bde950547\": container with ID starting with 81321d5a47a01280c9502c3d48d0a8440fee6ba43d0a27e0d1d1c32bde950547 not found: ID does not exist" containerID="81321d5a47a01280c9502c3d48d0a8440fee6ba43d0a27e0d1d1c32bde950547"
Jul 01 19:40:58.850059 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:58.849945    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81321d5a47a01280c9502c3d48d0a8440fee6ba43d0a27e0d1d1c32bde950547"} err="failed to get container status \"81321d5a47a01280c9502c3d48d0a8440fee6ba43d0a27e0d1d1c32bde950547\": rpc error: code = NotFound desc = could not find container \"81321d5a47a01280c9502c3d48d0a8440fee6ba43d0a27e0d1d1c32bde950547\": container with ID starting with 81321d5a47a01280c9502c3d48d0a8440fee6ba43d0a27e0d1d1c32bde950547 not found: ID does not exist"
Jul 01 19:40:58.850895 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:58.850873    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-88764-predictor-56c748fb89-w494b"]
Jul 01 19:40:59.831281 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:59.831191    2574 generic.go:356] "Generic (PLEG): container finished" podID="fee55a5f-1926-488e-afe7-8678e24e8e8a" containerID="c06b3d634e4ac540a40cd3397eb5649f70bc17ca63c7fa5290311137de1bdb49" exitCode=0
Jul 01 19:40:59.831281 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:40:59.831269    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk" event={"ID":"fee55a5f-1926-488e-afe7-8678e24e8e8a","Type":"ContainerDied","Data":"c06b3d634e4ac540a40cd3397eb5649f70bc17ca63c7fa5290311137de1bdb49"}
Jul 01 19:41:00.357173 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:41:00.357139    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8a5ec24-3a26-4c1f-859f-0f58c21010d5" path="/var/lib/kubelet/pods/d8a5ec24-3a26-4c1f-859f-0f58c21010d5/volumes"
Jul 01 19:41:00.838962 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:41:00.838876    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk" event={"ID":"fee55a5f-1926-488e-afe7-8678e24e8e8a","Type":"ContainerStarted","Data":"8740bcb8354236673cce58b008315219f0bb68772c4b7b23341168282b8c8f74"}
Jul 01 19:41:00.839358 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:41:00.838964    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk" event={"ID":"fee55a5f-1926-488e-afe7-8678e24e8e8a","Type":"ContainerStarted","Data":"0e82c89b4022dac61025fc1cffe4e9004a1e5b5e530db07ab74aa62d83749daf"}
Jul 01 19:41:00.839358 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:41:00.839235    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk"
Jul 01 19:41:00.839358 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:41:00.839288    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk"
Jul 01 19:41:00.840509 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:41:00.840485    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk" podUID="fee55a5f-1926-488e-afe7-8678e24e8e8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 01 19:41:00.858223 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:41:00.858167    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk" podStartSLOduration=4.268283318 podStartE2EDuration="7.858148808s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.58986549s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:40:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:41:00.857344887 +0000 UTC m=+1166.949344333" watchObservedRunningTime="2026-07-01 19:41:00.858148808 +0000 UTC m=+1166.950148260"
Jul 01 19:41:01.843989 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:41:01.843949    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk" podUID="fee55a5f-1926-488e-afe7-8678e24e8e8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 01 19:41:06.848800 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:41:06.848772    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk"
Jul 01 19:41:06.849359 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:41:06.849331    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk" podUID="fee55a5f-1926-488e-afe7-8678e24e8e8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 01 19:41:16.849876 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:41:16.849836    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk" podUID="fee55a5f-1926-488e-afe7-8678e24e8e8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 01 19:41:26.850034 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:41:26.849992    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk" podUID="fee55a5f-1926-488e-afe7-8678e24e8e8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 01 19:41:34.406667 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:41:34.406637    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bplsm_bfb845e6-341d-492e-91c9-9c698a0b8d29/ovn-acl-logging/0.log"
Jul 01 19:41:34.408764 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:41:34.408743    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bplsm_bfb845e6-341d-492e-91c9-9c698a0b8d29/ovn-acl-logging/0.log"
Jul 01 19:41:36.850156 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:41:36.850116    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk" podUID="fee55a5f-1926-488e-afe7-8678e24e8e8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 01 19:41:46.849274 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:41:46.849234    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk" podUID="fee55a5f-1926-488e-afe7-8678e24e8e8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 01 19:41:56.849580 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:41:56.849538    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk" podUID="fee55a5f-1926-488e-afe7-8678e24e8e8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 01 19:42:06.849896 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:06.849858    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk" podUID="fee55a5f-1926-488e-afe7-8678e24e8e8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 01 19:42:07.351446 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:07.351411    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk" podUID="fee55a5f-1926-488e-afe7-8678e24e8e8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 01 19:42:17.352636 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:17.352606    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk"
Jul 01 19:42:24.111511 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:24.111473    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-6ef841-predictor-59958cfbbf-2472d"]
Jul 01 19:42:24.112107 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:24.112073    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8a5ec24-3a26-4c1f-859f-0f58c21010d5" containerName="kserve-container"
Jul 01 19:42:24.112107 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:24.112095    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8a5ec24-3a26-4c1f-859f-0f58c21010d5" containerName="kserve-container"
Jul 01 19:42:24.112235 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:24.112140    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8a5ec24-3a26-4c1f-859f-0f58c21010d5" containerName="storage-initializer"
Jul 01 19:42:24.112235 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:24.112150    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8a5ec24-3a26-4c1f-859f-0f58c21010d5" containerName="storage-initializer"
Jul 01 19:42:24.112235 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:24.112162    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8a5ec24-3a26-4c1f-859f-0f58c21010d5" containerName="kube-rbac-proxy"
Jul 01 19:42:24.112235 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:24.112171    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8a5ec24-3a26-4c1f-859f-0f58c21010d5" containerName="kube-rbac-proxy"
Jul 01 19:42:24.112435 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:24.112251    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8a5ec24-3a26-4c1f-859f-0f58c21010d5" containerName="kserve-container"
Jul 01 19:42:24.112435 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:24.112267    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8a5ec24-3a26-4c1f-859f-0f58c21010d5" containerName="kube-rbac-proxy"
Jul 01 19:42:24.117028 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:24.117007    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-6ef841-predictor-59958cfbbf-2472d"
Jul 01 19:42:24.119441 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:24.119406    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-6ef841-predictor-serving-cert\""
Jul 01 19:42:24.119550 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:24.119442    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-6ef841-kube-rbac-proxy-sar-config\""
Jul 01 19:42:24.119550 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:24.119482    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-6ef841-dockercfg-gxq9w\""
Jul 01 19:42:24.119550 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:24.119411    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-6ef841\""
Jul 01 19:42:24.125769 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:24.125746    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-6ef841-predictor-59958cfbbf-2472d"]
Jul 01 19:42:24.241354 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:24.241318    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84c2aa03-5f7a-4a9d-a570-d3fd181cb242-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-6ef841-predictor-59958cfbbf-2472d\" (UID: \"84c2aa03-5f7a-4a9d-a570-d3fd181cb242\") " pod="kserve-ci-e2e-test/isvc-secondary-6ef841-predictor-59958cfbbf-2472d"
Jul 01 19:42:24.241527 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:24.241367    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84c2aa03-5f7a-4a9d-a570-d3fd181cb242-cabundle-cert\") pod \"isvc-secondary-6ef841-predictor-59958cfbbf-2472d\" (UID: \"84c2aa03-5f7a-4a9d-a570-d3fd181cb242\") " pod="kserve-ci-e2e-test/isvc-secondary-6ef841-predictor-59958cfbbf-2472d"
Jul 01 19:42:24.241527 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:24.241448    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84c2aa03-5f7a-4a9d-a570-d3fd181cb242-proxy-tls\") pod \"isvc-secondary-6ef841-predictor-59958cfbbf-2472d\" (UID: \"84c2aa03-5f7a-4a9d-a570-d3fd181cb242\") " pod="kserve-ci-e2e-test/isvc-secondary-6ef841-predictor-59958cfbbf-2472d"
Jul 01 19:42:24.241527 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:24.241486    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84c2aa03-5f7a-4a9d-a570-d3fd181cb242-kserve-provision-location\") pod \"isvc-secondary-6ef841-predictor-59958cfbbf-2472d\" (UID: \"84c2aa03-5f7a-4a9d-a570-d3fd181cb242\") " pod="kserve-ci-e2e-test/isvc-secondary-6ef841-predictor-59958cfbbf-2472d"
Jul 01 19:42:24.241527 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:24.241522    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d5np\" (UniqueName: \"kubernetes.io/projected/84c2aa03-5f7a-4a9d-a570-d3fd181cb242-kube-api-access-9d5np\") pod \"isvc-secondary-6ef841-predictor-59958cfbbf-2472d\" (UID: \"84c2aa03-5f7a-4a9d-a570-d3fd181cb242\") " pod="kserve-ci-e2e-test/isvc-secondary-6ef841-predictor-59958cfbbf-2472d"
Jul 01 19:42:24.342441 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:24.342401    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84c2aa03-5f7a-4a9d-a570-d3fd181cb242-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-6ef841-predictor-59958cfbbf-2472d\" (UID: \"84c2aa03-5f7a-4a9d-a570-d3fd181cb242\") " pod="kserve-ci-e2e-test/isvc-secondary-6ef841-predictor-59958cfbbf-2472d"
Jul 01 19:42:24.342635 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:24.342460    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84c2aa03-5f7a-4a9d-a570-d3fd181cb242-cabundle-cert\") pod \"isvc-secondary-6ef841-predictor-59958cfbbf-2472d\" (UID: \"84c2aa03-5f7a-4a9d-a570-d3fd181cb242\") " pod="kserve-ci-e2e-test/isvc-secondary-6ef841-predictor-59958cfbbf-2472d"
Jul 01 19:42:24.342635 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:24.342494    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84c2aa03-5f7a-4a9d-a570-d3fd181cb242-proxy-tls\") pod \"isvc-secondary-6ef841-predictor-59958cfbbf-2472d\" (UID: \"84c2aa03-5f7a-4a9d-a570-d3fd181cb242\") " pod="kserve-ci-e2e-test/isvc-secondary-6ef841-predictor-59958cfbbf-2472d"
Jul 01 19:42:24.342635 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:24.342535    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84c2aa03-5f7a-4a9d-a570-d3fd181cb242-kserve-provision-location\") pod \"isvc-secondary-6ef841-predictor-59958cfbbf-2472d\" (UID: \"84c2aa03-5f7a-4a9d-a570-d3fd181cb242\") " pod="kserve-ci-e2e-test/isvc-secondary-6ef841-predictor-59958cfbbf-2472d"
Jul 01 19:42:24.342635 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:24.342578    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d5np\" (UniqueName: \"kubernetes.io/projected/84c2aa03-5f7a-4a9d-a570-d3fd181cb242-kube-api-access-9d5np\") pod \"isvc-secondary-6ef841-predictor-59958cfbbf-2472d\" (UID: \"84c2aa03-5f7a-4a9d-a570-d3fd181cb242\") " pod="kserve-ci-e2e-test/isvc-secondary-6ef841-predictor-59958cfbbf-2472d"
Jul 01 19:42:24.342978 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:24.342954    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84c2aa03-5f7a-4a9d-a570-d3fd181cb242-kserve-provision-location\") pod \"isvc-secondary-6ef841-predictor-59958cfbbf-2472d\" (UID: \"84c2aa03-5f7a-4a9d-a570-d3fd181cb242\") " pod="kserve-ci-e2e-test/isvc-secondary-6ef841-predictor-59958cfbbf-2472d"
Jul 01 19:42:24.343156 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:24.343136    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84c2aa03-5f7a-4a9d-a570-d3fd181cb242-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-6ef841-predictor-59958cfbbf-2472d\" (UID: \"84c2aa03-5f7a-4a9d-a570-d3fd181cb242\") " pod="kserve-ci-e2e-test/isvc-secondary-6ef841-predictor-59958cfbbf-2472d"
Jul 01 19:42:24.343223 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:24.343204    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84c2aa03-5f7a-4a9d-a570-d3fd181cb242-cabundle-cert\") pod \"isvc-secondary-6ef841-predictor-59958cfbbf-2472d\" (UID: \"84c2aa03-5f7a-4a9d-a570-d3fd181cb242\") " pod="kserve-ci-e2e-test/isvc-secondary-6ef841-predictor-59958cfbbf-2472d"
Jul 01 19:42:24.344978 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:24.344956    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84c2aa03-5f7a-4a9d-a570-d3fd181cb242-proxy-tls\") pod \"isvc-secondary-6ef841-predictor-59958cfbbf-2472d\" (UID: \"84c2aa03-5f7a-4a9d-a570-d3fd181cb242\") " pod="kserve-ci-e2e-test/isvc-secondary-6ef841-predictor-59958cfbbf-2472d"
Jul 01 19:42:24.349791 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:24.349770    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d5np\" (UniqueName: \"kubernetes.io/projected/84c2aa03-5f7a-4a9d-a570-d3fd181cb242-kube-api-access-9d5np\") pod \"isvc-secondary-6ef841-predictor-59958cfbbf-2472d\" (UID: \"84c2aa03-5f7a-4a9d-a570-d3fd181cb242\") " pod="kserve-ci-e2e-test/isvc-secondary-6ef841-predictor-59958cfbbf-2472d"
Jul 01 19:42:24.428100 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:24.428043    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-6ef841-predictor-59958cfbbf-2472d"
Jul 01 19:42:24.561690 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:24.561664    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-6ef841-predictor-59958cfbbf-2472d"]
Jul 01 19:42:24.564087 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:42:24.564049    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84c2aa03_5f7a_4a9d_a570_d3fd181cb242.slice/crio-d352da94e4f8ef6f599928fdab54984251588964ada06dbe7ca7a997a5d220b0 WatchSource:0}: Error finding container d352da94e4f8ef6f599928fdab54984251588964ada06dbe7ca7a997a5d220b0: Status 404 returned error can't find the container with id d352da94e4f8ef6f599928fdab54984251588964ada06dbe7ca7a997a5d220b0
Jul 01 19:42:25.242446 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:25.242411    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-6ef841-predictor-59958cfbbf-2472d" event={"ID":"84c2aa03-5f7a-4a9d-a570-d3fd181cb242","Type":"ContainerStarted","Data":"f4310b073d42ade7e8866265c04a825f11c76570b0e9ca1e50be0b4465826733"}
Jul 01 19:42:25.242952 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:25.242480    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-6ef841-predictor-59958cfbbf-2472d" event={"ID":"84c2aa03-5f7a-4a9d-a570-d3fd181cb242","Type":"ContainerStarted","Data":"d352da94e4f8ef6f599928fdab54984251588964ada06dbe7ca7a997a5d220b0"}
Jul 01 19:42:29.262640 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:29.262563    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-6ef841-predictor-59958cfbbf-2472d_84c2aa03-5f7a-4a9d-a570-d3fd181cb242/storage-initializer/0.log"
Jul 01 19:42:29.262640 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:29.262602    2574 generic.go:356] "Generic (PLEG): container finished" podID="84c2aa03-5f7a-4a9d-a570-d3fd181cb242" containerID="f4310b073d42ade7e8866265c04a825f11c76570b0e9ca1e50be0b4465826733" exitCode=1
Jul 01 19:42:29.263124 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:29.262668    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-6ef841-predictor-59958cfbbf-2472d" event={"ID":"84c2aa03-5f7a-4a9d-a570-d3fd181cb242","Type":"ContainerDied","Data":"f4310b073d42ade7e8866265c04a825f11c76570b0e9ca1e50be0b4465826733"}
Jul 01 19:42:30.269335 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:30.269306    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-6ef841-predictor-59958cfbbf-2472d_84c2aa03-5f7a-4a9d-a570-d3fd181cb242/storage-initializer/0.log"
Jul 01 19:42:30.269718 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:30.269408    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-6ef841-predictor-59958cfbbf-2472d" event={"ID":"84c2aa03-5f7a-4a9d-a570-d3fd181cb242","Type":"ContainerStarted","Data":"fd416a52d5d0599e9714ac5233c8b89cca780cb26b261b2330e3e550dca4cfca"}
Jul 01 19:42:35.294694 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:35.294663    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-6ef841-predictor-59958cfbbf-2472d_84c2aa03-5f7a-4a9d-a570-d3fd181cb242/storage-initializer/1.log"
Jul 01 19:42:35.295152 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:35.295056    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-6ef841-predictor-59958cfbbf-2472d_84c2aa03-5f7a-4a9d-a570-d3fd181cb242/storage-initializer/0.log"
Jul 01 19:42:35.295152 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:35.295088    2574 generic.go:356] "Generic (PLEG): container finished" podID="84c2aa03-5f7a-4a9d-a570-d3fd181cb242" containerID="fd416a52d5d0599e9714ac5233c8b89cca780cb26b261b2330e3e550dca4cfca" exitCode=1
Jul 01 19:42:35.295240 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:35.295159    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-6ef841-predictor-59958cfbbf-2472d" event={"ID":"84c2aa03-5f7a-4a9d-a570-d3fd181cb242","Type":"ContainerDied","Data":"fd416a52d5d0599e9714ac5233c8b89cca780cb26b261b2330e3e550dca4cfca"}
Jul 01 19:42:35.295240 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:35.295220    2574 scope.go:117] "RemoveContainer" containerID="f4310b073d42ade7e8866265c04a825f11c76570b0e9ca1e50be0b4465826733"
Jul 01 19:42:35.295588 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:35.295568    2574 scope.go:117] "RemoveContainer" containerID="f4310b073d42ade7e8866265c04a825f11c76570b0e9ca1e50be0b4465826733"
Jul 01 19:42:35.305656 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:42:35.305624    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-6ef841-predictor-59958cfbbf-2472d_kserve-ci-e2e-test(84c2aa03-5f7a-4a9d-a570-d3fd181cb242)\"" pod="kserve-ci-e2e-test/isvc-secondary-6ef841-predictor-59958cfbbf-2472d" podUID="84c2aa03-5f7a-4a9d-a570-d3fd181cb242"
Jul 01 19:42:35.305777 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:42:35.305751    2574 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-6ef841-predictor-59958cfbbf-2472d_kserve-ci-e2e-test_84c2aa03-5f7a-4a9d-a570-d3fd181cb242_0 in pod sandbox d352da94e4f8ef6f599928fdab54984251588964ada06dbe7ca7a997a5d220b0 from index: no such id: 'f4310b073d42ade7e8866265c04a825f11c76570b0e9ca1e50be0b4465826733'" containerID="f4310b073d42ade7e8866265c04a825f11c76570b0e9ca1e50be0b4465826733"
Jul 01 19:42:35.305827 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:35.305787    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4310b073d42ade7e8866265c04a825f11c76570b0e9ca1e50be0b4465826733"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-6ef841-predictor-59958cfbbf-2472d_kserve-ci-e2e-test_84c2aa03-5f7a-4a9d-a570-d3fd181cb242_0 in pod sandbox d352da94e4f8ef6f599928fdab54984251588964ada06dbe7ca7a997a5d220b0 from index: no such id: 'f4310b073d42ade7e8866265c04a825f11c76570b0e9ca1e50be0b4465826733'"
Jul 01 19:42:36.301208 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:36.301175    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-6ef841-predictor-59958cfbbf-2472d_84c2aa03-5f7a-4a9d-a570-d3fd181cb242/storage-initializer/1.log"
Jul 01 19:42:40.106110 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:40.106079    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk"]
Jul 01 19:42:40.106545 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:40.106325    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk" podUID="fee55a5f-1926-488e-afe7-8678e24e8e8a" containerName="kserve-container" containerID="cri-o://0e82c89b4022dac61025fc1cffe4e9004a1e5b5e530db07ab74aa62d83749daf" gracePeriod=30
Jul 01 19:42:40.106545 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:40.106379    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk" podUID="fee55a5f-1926-488e-afe7-8678e24e8e8a" containerName="kube-rbac-proxy" containerID="cri-o://8740bcb8354236673cce58b008315219f0bb68772c4b7b23341168282b8c8f74" gracePeriod=30
Jul 01 19:42:40.175957 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:40.175922    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-6ef841-predictor-59958cfbbf-2472d"]
Jul 01 19:42:40.304468 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:40.304444    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-6ef841-predictor-59958cfbbf-2472d_84c2aa03-5f7a-4a9d-a570-d3fd181cb242/storage-initializer/1.log"
Jul 01 19:42:40.306574 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:40.306559    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-6ef841-predictor-59958cfbbf-2472d"
Jul 01 19:42:40.331481 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:40.331452    2574 generic.go:356] "Generic (PLEG): container finished" podID="fee55a5f-1926-488e-afe7-8678e24e8e8a" containerID="8740bcb8354236673cce58b008315219f0bb68772c4b7b23341168282b8c8f74" exitCode=2
Jul 01 19:42:40.331601 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:40.331524    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk" event={"ID":"fee55a5f-1926-488e-afe7-8678e24e8e8a","Type":"ContainerDied","Data":"8740bcb8354236673cce58b008315219f0bb68772c4b7b23341168282b8c8f74"}
Jul 01 19:42:40.332746 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:40.332726    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-6ef841-predictor-59958cfbbf-2472d_84c2aa03-5f7a-4a9d-a570-d3fd181cb242/storage-initializer/1.log"
Jul 01 19:42:40.332841 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:40.332802    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-6ef841-predictor-59958cfbbf-2472d" event={"ID":"84c2aa03-5f7a-4a9d-a570-d3fd181cb242","Type":"ContainerDied","Data":"d352da94e4f8ef6f599928fdab54984251588964ada06dbe7ca7a997a5d220b0"}
Jul 01 19:42:40.332886 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:40.332851    2574 scope.go:117] "RemoveContainer" containerID="fd416a52d5d0599e9714ac5233c8b89cca780cb26b261b2330e3e550dca4cfca"
Jul 01 19:42:40.332963 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:40.332808    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-6ef841-predictor-59958cfbbf-2472d"
Jul 01 19:42:40.377881 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:40.377810    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84c2aa03-5f7a-4a9d-a570-d3fd181cb242-kube-rbac-proxy-sar-config\") pod \"84c2aa03-5f7a-4a9d-a570-d3fd181cb242\" (UID: \"84c2aa03-5f7a-4a9d-a570-d3fd181cb242\") "
Jul 01 19:42:40.377881 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:40.377855    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84c2aa03-5f7a-4a9d-a570-d3fd181cb242-proxy-tls\") pod \"84c2aa03-5f7a-4a9d-a570-d3fd181cb242\" (UID: \"84c2aa03-5f7a-4a9d-a570-d3fd181cb242\") "
Jul 01 19:42:40.378107 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:40.377948    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84c2aa03-5f7a-4a9d-a570-d3fd181cb242-kserve-provision-location\") pod \"84c2aa03-5f7a-4a9d-a570-d3fd181cb242\" (UID: \"84c2aa03-5f7a-4a9d-a570-d3fd181cb242\") "
Jul 01 19:42:40.378107 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:40.378013    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84c2aa03-5f7a-4a9d-a570-d3fd181cb242-cabundle-cert\") pod \"84c2aa03-5f7a-4a9d-a570-d3fd181cb242\" (UID: \"84c2aa03-5f7a-4a9d-a570-d3fd181cb242\") "
Jul 01 19:42:40.378107 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:40.378059    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9d5np\" (UniqueName: \"kubernetes.io/projected/84c2aa03-5f7a-4a9d-a570-d3fd181cb242-kube-api-access-9d5np\") pod \"84c2aa03-5f7a-4a9d-a570-d3fd181cb242\" (UID: \"84c2aa03-5f7a-4a9d-a570-d3fd181cb242\") "
Jul 01 19:42:40.378265 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:40.378197    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84c2aa03-5f7a-4a9d-a570-d3fd181cb242-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "84c2aa03-5f7a-4a9d-a570-d3fd181cb242" (UID: "84c2aa03-5f7a-4a9d-a570-d3fd181cb242"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:42:40.378265 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:40.378224    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c2aa03-5f7a-4a9d-a570-d3fd181cb242-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "84c2aa03-5f7a-4a9d-a570-d3fd181cb242" (UID: "84c2aa03-5f7a-4a9d-a570-d3fd181cb242"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:42:40.378401 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:40.378315    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84c2aa03-5f7a-4a9d-a570-d3fd181cb242-kserve-provision-location\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:42:40.378401 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:40.378333    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84c2aa03-5f7a-4a9d-a570-d3fd181cb242-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:42:40.378401 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:40.378355    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84c2aa03-5f7a-4a9d-a570-d3fd181cb242-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "84c2aa03-5f7a-4a9d-a570-d3fd181cb242" (UID: "84c2aa03-5f7a-4a9d-a570-d3fd181cb242"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:42:40.379932 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:40.379891    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84c2aa03-5f7a-4a9d-a570-d3fd181cb242-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "84c2aa03-5f7a-4a9d-a570-d3fd181cb242" (UID: "84c2aa03-5f7a-4a9d-a570-d3fd181cb242"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:42:40.380022 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:40.380005    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84c2aa03-5f7a-4a9d-a570-d3fd181cb242-kube-api-access-9d5np" (OuterVolumeSpecName: "kube-api-access-9d5np") pod "84c2aa03-5f7a-4a9d-a570-d3fd181cb242" (UID: "84c2aa03-5f7a-4a9d-a570-d3fd181cb242"). InnerVolumeSpecName "kube-api-access-9d5np". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:42:40.479744 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:40.479708    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84c2aa03-5f7a-4a9d-a570-d3fd181cb242-cabundle-cert\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:42:40.479744 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:40.479734    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d5np\" (UniqueName: \"kubernetes.io/projected/84c2aa03-5f7a-4a9d-a570-d3fd181cb242-kube-api-access-9d5np\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:42:40.479744 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:40.479747    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84c2aa03-5f7a-4a9d-a570-d3fd181cb242-proxy-tls\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:42:40.691006 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:40.690969    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-6ef841-predictor-59958cfbbf-2472d"]
Jul 01 19:42:40.703644 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:40.703613    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-6ef841-predictor-59958cfbbf-2472d"]
Jul 01 19:42:41.844697 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:41.844653    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk" podUID="fee55a5f-1926-488e-afe7-8678e24e8e8a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Jul 01 19:42:42.356358 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:42.356322    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84c2aa03-5f7a-4a9d-a570-d3fd181cb242" path="/var/lib/kubelet/pods/84c2aa03-5f7a-4a9d-a570-d3fd181cb242/volumes"
Jul 01 19:42:44.655017 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:44.654994    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk"
Jul 01 19:42:44.715301 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:44.715221    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctbpw\" (UniqueName: \"kubernetes.io/projected/fee55a5f-1926-488e-afe7-8678e24e8e8a-kube-api-access-ctbpw\") pod \"fee55a5f-1926-488e-afe7-8678e24e8e8a\" (UID: \"fee55a5f-1926-488e-afe7-8678e24e8e8a\") "
Jul 01 19:42:44.715301 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:44.715272    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fee55a5f-1926-488e-afe7-8678e24e8e8a-kube-rbac-proxy-sar-config\") pod \"fee55a5f-1926-488e-afe7-8678e24e8e8a\" (UID: \"fee55a5f-1926-488e-afe7-8678e24e8e8a\") "
Jul 01 19:42:44.715301 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:44.715291    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fee55a5f-1926-488e-afe7-8678e24e8e8a-cabundle-cert\") pod \"fee55a5f-1926-488e-afe7-8678e24e8e8a\" (UID: \"fee55a5f-1926-488e-afe7-8678e24e8e8a\") "
Jul 01 19:42:44.715530 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:44.715325    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fee55a5f-1926-488e-afe7-8678e24e8e8a-kserve-provision-location\") pod \"fee55a5f-1926-488e-afe7-8678e24e8e8a\" (UID: \"fee55a5f-1926-488e-afe7-8678e24e8e8a\") "
Jul 01 19:42:44.715530 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:44.715352    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fee55a5f-1926-488e-afe7-8678e24e8e8a-proxy-tls\") pod \"fee55a5f-1926-488e-afe7-8678e24e8e8a\" (UID: \"fee55a5f-1926-488e-afe7-8678e24e8e8a\") "
Jul 01 19:42:44.715723 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:44.715700    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fee55a5f-1926-488e-afe7-8678e24e8e8a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fee55a5f-1926-488e-afe7-8678e24e8e8a" (UID: "fee55a5f-1926-488e-afe7-8678e24e8e8a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:42:44.715783 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:44.715713    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fee55a5f-1926-488e-afe7-8678e24e8e8a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fee55a5f-1926-488e-afe7-8678e24e8e8a" (UID: "fee55a5f-1926-488e-afe7-8678e24e8e8a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:42:44.715783 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:44.715717    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fee55a5f-1926-488e-afe7-8678e24e8e8a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fee55a5f-1926-488e-afe7-8678e24e8e8a" (UID: "fee55a5f-1926-488e-afe7-8678e24e8e8a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:42:44.717398 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:44.717379    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fee55a5f-1926-488e-afe7-8678e24e8e8a-kube-api-access-ctbpw" (OuterVolumeSpecName: "kube-api-access-ctbpw") pod "fee55a5f-1926-488e-afe7-8678e24e8e8a" (UID: "fee55a5f-1926-488e-afe7-8678e24e8e8a"). InnerVolumeSpecName "kube-api-access-ctbpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:42:44.717470 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:44.717444    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fee55a5f-1926-488e-afe7-8678e24e8e8a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fee55a5f-1926-488e-afe7-8678e24e8e8a" (UID: "fee55a5f-1926-488e-afe7-8678e24e8e8a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:42:44.816857 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:44.816822    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctbpw\" (UniqueName: \"kubernetes.io/projected/fee55a5f-1926-488e-afe7-8678e24e8e8a-kube-api-access-ctbpw\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:42:44.816857 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:44.816850    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fee55a5f-1926-488e-afe7-8678e24e8e8a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:42:44.816857 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:44.816860    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fee55a5f-1926-488e-afe7-8678e24e8e8a-cabundle-cert\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:42:44.817111 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:44.816869    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fee55a5f-1926-488e-afe7-8678e24e8e8a-kserve-provision-location\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:42:44.817111 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:44.816880    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fee55a5f-1926-488e-afe7-8678e24e8e8a-proxy-tls\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:42:45.359977 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.359937    2574 generic.go:356] "Generic (PLEG): container finished" podID="fee55a5f-1926-488e-afe7-8678e24e8e8a" containerID="0e82c89b4022dac61025fc1cffe4e9004a1e5b5e530db07ab74aa62d83749daf" exitCode=0
Jul 01 19:42:45.360187 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.360034    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk" event={"ID":"fee55a5f-1926-488e-afe7-8678e24e8e8a","Type":"ContainerDied","Data":"0e82c89b4022dac61025fc1cffe4e9004a1e5b5e530db07ab74aa62d83749daf"}
Jul 01 19:42:45.360187 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.360103    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk" event={"ID":"fee55a5f-1926-488e-afe7-8678e24e8e8a","Type":"ContainerDied","Data":"7975ce3b4eec79629b5b87ceab6b1ce79c06c0f7728b26f837bca22870a6d265"}
Jul 01 19:42:45.360187 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.360136    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk"
Jul 01 19:42:45.360538 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.360146    2574 scope.go:117] "RemoveContainer" containerID="8740bcb8354236673cce58b008315219f0bb68772c4b7b23341168282b8c8f74"
Jul 01 19:42:45.369715 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.369695    2574 scope.go:117] "RemoveContainer" containerID="0e82c89b4022dac61025fc1cffe4e9004a1e5b5e530db07ab74aa62d83749daf"
Jul 01 19:42:45.370658 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.370567    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd"]
Jul 01 19:42:45.371001 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.370984    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fee55a5f-1926-488e-afe7-8678e24e8e8a" containerName="kube-rbac-proxy"
Jul 01 19:42:45.371093 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.371004    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fee55a5f-1926-488e-afe7-8678e24e8e8a" containerName="kube-rbac-proxy"
Jul 01 19:42:45.371093 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.371022    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84c2aa03-5f7a-4a9d-a570-d3fd181cb242" containerName="storage-initializer"
Jul 01 19:42:45.371093 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.371031    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="84c2aa03-5f7a-4a9d-a570-d3fd181cb242" containerName="storage-initializer"
Jul 01 19:42:45.371093 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.371043    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fee55a5f-1926-488e-afe7-8678e24e8e8a" containerName="storage-initializer"
Jul 01 19:42:45.371093 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.371052    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fee55a5f-1926-488e-afe7-8678e24e8e8a" containerName="storage-initializer"
Jul 01 19:42:45.371093 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.371068    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fee55a5f-1926-488e-afe7-8678e24e8e8a" containerName="kserve-container"
Jul 01 19:42:45.371093 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.371076    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fee55a5f-1926-488e-afe7-8678e24e8e8a" containerName="kserve-container"
Jul 01 19:42:45.371433 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.371223    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="84c2aa03-5f7a-4a9d-a570-d3fd181cb242" containerName="storage-initializer"
Jul 01 19:42:45.371433 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.371240    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="84c2aa03-5f7a-4a9d-a570-d3fd181cb242" containerName="storage-initializer"
Jul 01 19:42:45.371433 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.371251    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fee55a5f-1926-488e-afe7-8678e24e8e8a" containerName="kserve-container"
Jul 01 19:42:45.371433 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.371264    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fee55a5f-1926-488e-afe7-8678e24e8e8a" containerName="kube-rbac-proxy"
Jul 01 19:42:45.371433 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.371364    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84c2aa03-5f7a-4a9d-a570-d3fd181cb242" containerName="storage-initializer"
Jul 01 19:42:45.371433 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.371374    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="84c2aa03-5f7a-4a9d-a570-d3fd181cb242" containerName="storage-initializer"
Jul 01 19:42:45.375790 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.375770    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd"
Jul 01 19:42:45.377230 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.377213    2574 scope.go:117] "RemoveContainer" containerID="c06b3d634e4ac540a40cd3397eb5649f70bc17ca63c7fa5290311137de1bdb49"
Jul 01 19:42:45.378342 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.378280    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-7fef16-kube-rbac-proxy-sar-config\""
Jul 01 19:42:45.378446 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.378361    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-7fef16-dockercfg-4l5vg\""
Jul 01 19:42:45.378446 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.378364    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 19:42:45.379358 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.378832    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 19:42:45.379358 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.378845    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 19:42:45.379358 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.378887    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-7fef16-predictor-serving-cert\""
Jul 01 19:42:45.379554 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.379472    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-7fef16\""
Jul 01 19:42:45.386707 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.386686    2574 scope.go:117] "RemoveContainer" containerID="8740bcb8354236673cce58b008315219f0bb68772c4b7b23341168282b8c8f74"
Jul 01 19:42:45.387157 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:42:45.387130    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8740bcb8354236673cce58b008315219f0bb68772c4b7b23341168282b8c8f74\": container with ID starting with 8740bcb8354236673cce58b008315219f0bb68772c4b7b23341168282b8c8f74 not found: ID does not exist" containerID="8740bcb8354236673cce58b008315219f0bb68772c4b7b23341168282b8c8f74"
Jul 01 19:42:45.387736 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.387167    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8740bcb8354236673cce58b008315219f0bb68772c4b7b23341168282b8c8f74"} err="failed to get container status \"8740bcb8354236673cce58b008315219f0bb68772c4b7b23341168282b8c8f74\": rpc error: code = NotFound desc = could not find container \"8740bcb8354236673cce58b008315219f0bb68772c4b7b23341168282b8c8f74\": container with ID starting with 8740bcb8354236673cce58b008315219f0bb68772c4b7b23341168282b8c8f74 not found: ID does not exist"
Jul 01 19:42:45.387736 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.387190    2574 scope.go:117] "RemoveContainer" containerID="0e82c89b4022dac61025fc1cffe4e9004a1e5b5e530db07ab74aa62d83749daf"
Jul 01 19:42:45.388074 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:42:45.388012    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e82c89b4022dac61025fc1cffe4e9004a1e5b5e530db07ab74aa62d83749daf\": container with ID starting with 0e82c89b4022dac61025fc1cffe4e9004a1e5b5e530db07ab74aa62d83749daf not found: ID does not exist" containerID="0e82c89b4022dac61025fc1cffe4e9004a1e5b5e530db07ab74aa62d83749daf"
Jul 01 19:42:45.388074 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.388040    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e82c89b4022dac61025fc1cffe4e9004a1e5b5e530db07ab74aa62d83749daf"} err="failed to get container status \"0e82c89b4022dac61025fc1cffe4e9004a1e5b5e530db07ab74aa62d83749daf\": rpc error: code = NotFound desc = could not find container \"0e82c89b4022dac61025fc1cffe4e9004a1e5b5e530db07ab74aa62d83749daf\": container with ID starting with 0e82c89b4022dac61025fc1cffe4e9004a1e5b5e530db07ab74aa62d83749daf not found: ID does not exist"
Jul 01 19:42:45.388074 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.388061    2574 scope.go:117] "RemoveContainer" containerID="c06b3d634e4ac540a40cd3397eb5649f70bc17ca63c7fa5290311137de1bdb49"
Jul 01 19:42:45.388863 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:42:45.388841    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c06b3d634e4ac540a40cd3397eb5649f70bc17ca63c7fa5290311137de1bdb49\": container with ID starting with c06b3d634e4ac540a40cd3397eb5649f70bc17ca63c7fa5290311137de1bdb49 not found: ID does not exist" containerID="c06b3d634e4ac540a40cd3397eb5649f70bc17ca63c7fa5290311137de1bdb49"
Jul 01 19:42:45.388986 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.388870    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c06b3d634e4ac540a40cd3397eb5649f70bc17ca63c7fa5290311137de1bdb49"} err="failed to get container status \"c06b3d634e4ac540a40cd3397eb5649f70bc17ca63c7fa5290311137de1bdb49\": rpc error: code = NotFound desc = could not find container \"c06b3d634e4ac540a40cd3397eb5649f70bc17ca63c7fa5290311137de1bdb49\": container with ID starting with c06b3d634e4ac540a40cd3397eb5649f70bc17ca63c7fa5290311137de1bdb49 not found: ID does not exist"
Jul 01 19:42:45.390088 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.390060    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk"]
Jul 01 19:42:45.391878 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.391858    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd"]
Jul 01 19:42:45.398360 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.398341    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-6ef841-predictor-7f47d7b458-bz4zk"]
Jul 01 19:42:45.420733 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.420715    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/125505ca-8bb7-46d6-884c-d07530770a89-kserve-provision-location\") pod \"isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd\" (UID: \"125505ca-8bb7-46d6-884c-d07530770a89\") " pod="kserve-ci-e2e-test/isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd"
Jul 01 19:42:45.420834 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.420768    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/125505ca-8bb7-46d6-884c-d07530770a89-cabundle-cert\") pod \"isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd\" (UID: \"125505ca-8bb7-46d6-884c-d07530770a89\") " pod="kserve-ci-e2e-test/isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd"
Jul 01 19:42:45.420882 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.420859    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/125505ca-8bb7-46d6-884c-d07530770a89-proxy-tls\") pod \"isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd\" (UID: \"125505ca-8bb7-46d6-884c-d07530770a89\") " pod="kserve-ci-e2e-test/isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd"
Jul 01 19:42:45.420947 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.420888    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/125505ca-8bb7-46d6-884c-d07530770a89-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd\" (UID: \"125505ca-8bb7-46d6-884c-d07530770a89\") " pod="kserve-ci-e2e-test/isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd"
Jul 01 19:42:45.420947 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.420924    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj8xx\" (UniqueName: \"kubernetes.io/projected/125505ca-8bb7-46d6-884c-d07530770a89-kube-api-access-wj8xx\") pod \"isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd\" (UID: \"125505ca-8bb7-46d6-884c-d07530770a89\") " pod="kserve-ci-e2e-test/isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd"
Jul 01 19:42:45.521979 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.521947    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/125505ca-8bb7-46d6-884c-d07530770a89-proxy-tls\") pod \"isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd\" (UID: \"125505ca-8bb7-46d6-884c-d07530770a89\") " pod="kserve-ci-e2e-test/isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd"
Jul 01 19:42:45.521979 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.521982    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/125505ca-8bb7-46d6-884c-d07530770a89-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd\" (UID: \"125505ca-8bb7-46d6-884c-d07530770a89\") " pod="kserve-ci-e2e-test/isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd"
Jul 01 19:42:45.522249 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.522003    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj8xx\" (UniqueName: \"kubernetes.io/projected/125505ca-8bb7-46d6-884c-d07530770a89-kube-api-access-wj8xx\") pod \"isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd\" (UID: \"125505ca-8bb7-46d6-884c-d07530770a89\") " pod="kserve-ci-e2e-test/isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd"
Jul 01 19:42:45.522249 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.522071    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/125505ca-8bb7-46d6-884c-d07530770a89-kserve-provision-location\") pod \"isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd\" (UID: \"125505ca-8bb7-46d6-884c-d07530770a89\") " pod="kserve-ci-e2e-test/isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd"
Jul 01 19:42:45.522249 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.522166    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/125505ca-8bb7-46d6-884c-d07530770a89-cabundle-cert\") pod \"isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd\" (UID: \"125505ca-8bb7-46d6-884c-d07530770a89\") " pod="kserve-ci-e2e-test/isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd"
Jul 01 19:42:45.522249 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:42:45.522184    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-7fef16-predictor-serving-cert: secret "isvc-init-fail-7fef16-predictor-serving-cert" not found
Jul 01 19:42:45.522462 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:42:45.522256    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/125505ca-8bb7-46d6-884c-d07530770a89-proxy-tls podName:125505ca-8bb7-46d6-884c-d07530770a89 nodeName:}" failed. No retries permitted until 2026-07-01 19:42:46.022234556 +0000 UTC m=+1272.114233987 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/125505ca-8bb7-46d6-884c-d07530770a89-proxy-tls") pod "isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd" (UID: "125505ca-8bb7-46d6-884c-d07530770a89") : secret "isvc-init-fail-7fef16-predictor-serving-cert" not found
Jul 01 19:42:45.522534 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.522515    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/125505ca-8bb7-46d6-884c-d07530770a89-kserve-provision-location\") pod \"isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd\" (UID: \"125505ca-8bb7-46d6-884c-d07530770a89\") " pod="kserve-ci-e2e-test/isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd"
Jul 01 19:42:45.522751 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.522724    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/125505ca-8bb7-46d6-884c-d07530770a89-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd\" (UID: \"125505ca-8bb7-46d6-884c-d07530770a89\") " pod="kserve-ci-e2e-test/isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd"
Jul 01 19:42:45.522854 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.522830    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/125505ca-8bb7-46d6-884c-d07530770a89-cabundle-cert\") pod \"isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd\" (UID: \"125505ca-8bb7-46d6-884c-d07530770a89\") " pod="kserve-ci-e2e-test/isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd"
Jul 01 19:42:45.540028 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:45.540007    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj8xx\" (UniqueName: \"kubernetes.io/projected/125505ca-8bb7-46d6-884c-d07530770a89-kube-api-access-wj8xx\") pod \"isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd\" (UID: \"125505ca-8bb7-46d6-884c-d07530770a89\") " pod="kserve-ci-e2e-test/isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd"
Jul 01 19:42:46.026051 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:46.026004    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/125505ca-8bb7-46d6-884c-d07530770a89-proxy-tls\") pod \"isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd\" (UID: \"125505ca-8bb7-46d6-884c-d07530770a89\") " pod="kserve-ci-e2e-test/isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd"
Jul 01 19:42:46.028316 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:46.028296    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/125505ca-8bb7-46d6-884c-d07530770a89-proxy-tls\") pod \"isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd\" (UID: \"125505ca-8bb7-46d6-884c-d07530770a89\") " pod="kserve-ci-e2e-test/isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd"
Jul 01 19:42:46.289846 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:46.289751    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd"
Jul 01 19:42:46.359260 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:46.359156    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fee55a5f-1926-488e-afe7-8678e24e8e8a" path="/var/lib/kubelet/pods/fee55a5f-1926-488e-afe7-8678e24e8e8a/volumes"
Jul 01 19:42:46.425201 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:46.425176    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd"]
Jul 01 19:42:46.427172 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:42:46.427147    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod125505ca_8bb7_46d6_884c_d07530770a89.slice/crio-9ec9eb5ea5a8703d1145f19582401e0e1da8a58bef632d1a69b893976b74421d WatchSource:0}: Error finding container 9ec9eb5ea5a8703d1145f19582401e0e1da8a58bef632d1a69b893976b74421d: Status 404 returned error can't find the container with id 9ec9eb5ea5a8703d1145f19582401e0e1da8a58bef632d1a69b893976b74421d
Jul 01 19:42:47.374004 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:47.373967    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd" event={"ID":"125505ca-8bb7-46d6-884c-d07530770a89","Type":"ContainerStarted","Data":"69b8445bd7ad83909b7c644ad8caaaa4f7597703c0914fe428e16eb59b6fbf85"}
Jul 01 19:42:47.374397 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:47.374025    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd" event={"ID":"125505ca-8bb7-46d6-884c-d07530770a89","Type":"ContainerStarted","Data":"9ec9eb5ea5a8703d1145f19582401e0e1da8a58bef632d1a69b893976b74421d"}
Jul 01 19:42:51.392467 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:51.392444    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd_125505ca-8bb7-46d6-884c-d07530770a89/storage-initializer/0.log"
Jul 01 19:42:51.392796 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:51.392481    2574 generic.go:356] "Generic (PLEG): container finished" podID="125505ca-8bb7-46d6-884c-d07530770a89" containerID="69b8445bd7ad83909b7c644ad8caaaa4f7597703c0914fe428e16eb59b6fbf85" exitCode=1
Jul 01 19:42:51.392796 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:51.392559    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd" event={"ID":"125505ca-8bb7-46d6-884c-d07530770a89","Type":"ContainerDied","Data":"69b8445bd7ad83909b7c644ad8caaaa4f7597703c0914fe428e16eb59b6fbf85"}
Jul 01 19:42:52.398778 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:52.398751    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd_125505ca-8bb7-46d6-884c-d07530770a89/storage-initializer/0.log"
Jul 01 19:42:52.399429 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:52.398833    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd" event={"ID":"125505ca-8bb7-46d6-884c-d07530770a89","Type":"ContainerStarted","Data":"66ae1b931d1f4ea36ae720cd0d43a9bb0e13c3e0226ea3be75e6940bafba1f2e"}
Jul 01 19:42:55.369946 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:55.369916    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd"]
Jul 01 19:42:55.370423 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:55.370127    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd" podUID="125505ca-8bb7-46d6-884c-d07530770a89" containerName="storage-initializer" containerID="cri-o://66ae1b931d1f4ea36ae720cd0d43a9bb0e13c3e0226ea3be75e6940bafba1f2e" gracePeriod=30
Jul 01 19:42:55.413740 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:55.413710    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd_125505ca-8bb7-46d6-884c-d07530770a89/storage-initializer/1.log"
Jul 01 19:42:55.414155 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:55.414138    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd_125505ca-8bb7-46d6-884c-d07530770a89/storage-initializer/0.log"
Jul 01 19:42:55.414255 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:55.414171    2574 generic.go:356] "Generic (PLEG): container finished" podID="125505ca-8bb7-46d6-884c-d07530770a89" containerID="66ae1b931d1f4ea36ae720cd0d43a9bb0e13c3e0226ea3be75e6940bafba1f2e" exitCode=1
Jul 01 19:42:55.414255 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:55.414218    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd" event={"ID":"125505ca-8bb7-46d6-884c-d07530770a89","Type":"ContainerDied","Data":"66ae1b931d1f4ea36ae720cd0d43a9bb0e13c3e0226ea3be75e6940bafba1f2e"}
Jul 01 19:42:55.414368 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:55.414270    2574 scope.go:117] "RemoveContainer" containerID="69b8445bd7ad83909b7c644ad8caaaa4f7597703c0914fe428e16eb59b6fbf85"
Jul 01 19:42:55.503773 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:55.503747    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd_125505ca-8bb7-46d6-884c-d07530770a89/storage-initializer/1.log"
Jul 01 19:42:55.507392 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:55.507374    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd"
Jul 01 19:42:55.608035 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:55.608005    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/125505ca-8bb7-46d6-884c-d07530770a89-proxy-tls\") pod \"125505ca-8bb7-46d6-884c-d07530770a89\" (UID: \"125505ca-8bb7-46d6-884c-d07530770a89\") "
Jul 01 19:42:55.608035 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:55.608040    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wj8xx\" (UniqueName: \"kubernetes.io/projected/125505ca-8bb7-46d6-884c-d07530770a89-kube-api-access-wj8xx\") pod \"125505ca-8bb7-46d6-884c-d07530770a89\" (UID: \"125505ca-8bb7-46d6-884c-d07530770a89\") "
Jul 01 19:42:55.608273 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:55.608090    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/125505ca-8bb7-46d6-884c-d07530770a89-kserve-provision-location\") pod \"125505ca-8bb7-46d6-884c-d07530770a89\" (UID: \"125505ca-8bb7-46d6-884c-d07530770a89\") "
Jul 01 19:42:55.608273 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:55.608107    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/125505ca-8bb7-46d6-884c-d07530770a89-kube-rbac-proxy-sar-config\") pod \"125505ca-8bb7-46d6-884c-d07530770a89\" (UID: \"125505ca-8bb7-46d6-884c-d07530770a89\") "
Jul 01 19:42:55.608273 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:55.608141    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/125505ca-8bb7-46d6-884c-d07530770a89-cabundle-cert\") pod \"125505ca-8bb7-46d6-884c-d07530770a89\" (UID: \"125505ca-8bb7-46d6-884c-d07530770a89\") "
Jul 01 19:42:55.608451 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:55.608425    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/125505ca-8bb7-46d6-884c-d07530770a89-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "125505ca-8bb7-46d6-884c-d07530770a89" (UID: "125505ca-8bb7-46d6-884c-d07530770a89"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:42:55.608519 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:55.608497    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/125505ca-8bb7-46d6-884c-d07530770a89-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "125505ca-8bb7-46d6-884c-d07530770a89" (UID: "125505ca-8bb7-46d6-884c-d07530770a89"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:42:55.608576 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:55.608523    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/125505ca-8bb7-46d6-884c-d07530770a89-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "125505ca-8bb7-46d6-884c-d07530770a89" (UID: "125505ca-8bb7-46d6-884c-d07530770a89"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:42:55.610240 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:55.610219    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/125505ca-8bb7-46d6-884c-d07530770a89-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "125505ca-8bb7-46d6-884c-d07530770a89" (UID: "125505ca-8bb7-46d6-884c-d07530770a89"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:42:55.610240 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:55.610230    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/125505ca-8bb7-46d6-884c-d07530770a89-kube-api-access-wj8xx" (OuterVolumeSpecName: "kube-api-access-wj8xx") pod "125505ca-8bb7-46d6-884c-d07530770a89" (UID: "125505ca-8bb7-46d6-884c-d07530770a89"). InnerVolumeSpecName "kube-api-access-wj8xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:42:55.709604 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:55.709508    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/125505ca-8bb7-46d6-884c-d07530770a89-cabundle-cert\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:42:55.709604 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:55.709551    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/125505ca-8bb7-46d6-884c-d07530770a89-proxy-tls\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:42:55.709604 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:55.709560    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj8xx\" (UniqueName: \"kubernetes.io/projected/125505ca-8bb7-46d6-884c-d07530770a89-kube-api-access-wj8xx\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:42:55.709604 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:55.709587    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/125505ca-8bb7-46d6-884c-d07530770a89-kserve-provision-location\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:42:55.709604 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:55.709598    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/125505ca-8bb7-46d6-884c-d07530770a89-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:42:56.419692 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:56.419664    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd_125505ca-8bb7-46d6-884c-d07530770a89/storage-initializer/1.log"
Jul 01 19:42:56.420165 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:56.419732    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd" event={"ID":"125505ca-8bb7-46d6-884c-d07530770a89","Type":"ContainerDied","Data":"9ec9eb5ea5a8703d1145f19582401e0e1da8a58bef632d1a69b893976b74421d"}
Jul 01 19:42:56.420165 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:56.419765    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd"
Jul 01 19:42:56.420165 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:56.419806    2574 scope.go:117] "RemoveContainer" containerID="66ae1b931d1f4ea36ae720cd0d43a9bb0e13c3e0226ea3be75e6940bafba1f2e"
Jul 01 19:42:56.453228 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:56.453200    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd"]
Jul 01 19:42:56.457122 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:56.457099    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-7fef16-predictor-75b4dbcc4f-mrqrd"]
Jul 01 19:42:58.356431 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:42:58.356395    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="125505ca-8bb7-46d6-884c-d07530770a89" path="/var/lib/kubelet/pods/125505ca-8bb7-46d6-884c-d07530770a89/volumes"
Jul 01 19:43:00.582064 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:00.582025    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx"]
Jul 01 19:43:00.582458 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:00.582382    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="125505ca-8bb7-46d6-884c-d07530770a89" containerName="storage-initializer"
Jul 01 19:43:00.582458 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:00.582396    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="125505ca-8bb7-46d6-884c-d07530770a89" containerName="storage-initializer"
Jul 01 19:43:00.582458 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:00.582407    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="125505ca-8bb7-46d6-884c-d07530770a89" containerName="storage-initializer"
Jul 01 19:43:00.582458 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:00.582413    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="125505ca-8bb7-46d6-884c-d07530770a89" containerName="storage-initializer"
Jul 01 19:43:00.582638 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:00.582480    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="125505ca-8bb7-46d6-884c-d07530770a89" containerName="storage-initializer"
Jul 01 19:43:00.582638 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:00.582611    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="125505ca-8bb7-46d6-884c-d07530770a89" containerName="storage-initializer"
Jul 01 19:43:00.587154 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:00.587135    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx"
Jul 01 19:43:00.589789 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:00.589766    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 19:43:00.589985 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:00.589795    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 19:43:00.589985 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:00.589794    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-93e68-predictor-serving-cert\""
Jul 01 19:43:00.589985 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:00.589835    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 19:43:00.589985 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:00.589765    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-93e68-kube-rbac-proxy-sar-config\""
Jul 01 19:43:00.589985 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:00.589932    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-95l22\""
Jul 01 19:43:00.590515 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:00.590502    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 19:43:00.598195 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:00.598168    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx"]
Jul 01 19:43:00.648021 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:00.647991    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bdf51e1f-4915-4a9c-acff-4d11d63530d0-proxy-tls\") pod \"raw-sklearn-93e68-predictor-56757bf6ff-5sbjx\" (UID: \"bdf51e1f-4915-4a9c-acff-4d11d63530d0\") " pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx"
Jul 01 19:43:00.648175 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:00.648030    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9bkc\" (UniqueName: \"kubernetes.io/projected/bdf51e1f-4915-4a9c-acff-4d11d63530d0-kube-api-access-w9bkc\") pod \"raw-sklearn-93e68-predictor-56757bf6ff-5sbjx\" (UID: \"bdf51e1f-4915-4a9c-acff-4d11d63530d0\") " pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx"
Jul 01 19:43:00.648175 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:00.648059    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bdf51e1f-4915-4a9c-acff-4d11d63530d0-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-93e68-predictor-56757bf6ff-5sbjx\" (UID: \"bdf51e1f-4915-4a9c-acff-4d11d63530d0\") " pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx"
Jul 01 19:43:00.648254 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:00.648173    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdf51e1f-4915-4a9c-acff-4d11d63530d0-kserve-provision-location\") pod \"raw-sklearn-93e68-predictor-56757bf6ff-5sbjx\" (UID: \"bdf51e1f-4915-4a9c-acff-4d11d63530d0\") " pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx"
Jul 01 19:43:00.648254 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:00.648206    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bdf51e1f-4915-4a9c-acff-4d11d63530d0-cabundle-cert\") pod \"raw-sklearn-93e68-predictor-56757bf6ff-5sbjx\" (UID: \"bdf51e1f-4915-4a9c-acff-4d11d63530d0\") " pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx"
Jul 01 19:43:00.749001 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:00.748957    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdf51e1f-4915-4a9c-acff-4d11d63530d0-kserve-provision-location\") pod \"raw-sklearn-93e68-predictor-56757bf6ff-5sbjx\" (UID: \"bdf51e1f-4915-4a9c-acff-4d11d63530d0\") " pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx"
Jul 01 19:43:00.749164 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:00.749014    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bdf51e1f-4915-4a9c-acff-4d11d63530d0-cabundle-cert\") pod \"raw-sklearn-93e68-predictor-56757bf6ff-5sbjx\" (UID: \"bdf51e1f-4915-4a9c-acff-4d11d63530d0\") " pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx"
Jul 01 19:43:00.749164 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:00.749074    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bdf51e1f-4915-4a9c-acff-4d11d63530d0-proxy-tls\") pod \"raw-sklearn-93e68-predictor-56757bf6ff-5sbjx\" (UID: \"bdf51e1f-4915-4a9c-acff-4d11d63530d0\") " pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx"
Jul 01 19:43:00.749164 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:00.749109    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9bkc\" (UniqueName: \"kubernetes.io/projected/bdf51e1f-4915-4a9c-acff-4d11d63530d0-kube-api-access-w9bkc\") pod \"raw-sklearn-93e68-predictor-56757bf6ff-5sbjx\" (UID: \"bdf51e1f-4915-4a9c-acff-4d11d63530d0\") " pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx"
Jul 01 19:43:00.749164 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:00.749154    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bdf51e1f-4915-4a9c-acff-4d11d63530d0-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-93e68-predictor-56757bf6ff-5sbjx\" (UID: \"bdf51e1f-4915-4a9c-acff-4d11d63530d0\") " pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx"
Jul 01 19:43:00.749336 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:43:00.749223    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-93e68-predictor-serving-cert: secret "raw-sklearn-93e68-predictor-serving-cert" not found
Jul 01 19:43:00.749336 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:43:00.749300    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdf51e1f-4915-4a9c-acff-4d11d63530d0-proxy-tls podName:bdf51e1f-4915-4a9c-acff-4d11d63530d0 nodeName:}" failed. No retries permitted until 2026-07-01 19:43:01.249278729 +0000 UTC m=+1287.341278155 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bdf51e1f-4915-4a9c-acff-4d11d63530d0-proxy-tls") pod "raw-sklearn-93e68-predictor-56757bf6ff-5sbjx" (UID: "bdf51e1f-4915-4a9c-acff-4d11d63530d0") : secret "raw-sklearn-93e68-predictor-serving-cert" not found
Jul 01 19:43:00.749472 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:00.749453    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdf51e1f-4915-4a9c-acff-4d11d63530d0-kserve-provision-location\") pod \"raw-sklearn-93e68-predictor-56757bf6ff-5sbjx\" (UID: \"bdf51e1f-4915-4a9c-acff-4d11d63530d0\") " pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx"
Jul 01 19:43:00.749742 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:00.749724    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bdf51e1f-4915-4a9c-acff-4d11d63530d0-cabundle-cert\") pod \"raw-sklearn-93e68-predictor-56757bf6ff-5sbjx\" (UID: \"bdf51e1f-4915-4a9c-acff-4d11d63530d0\") " pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx"
Jul 01 19:43:00.749925 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:00.749881    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bdf51e1f-4915-4a9c-acff-4d11d63530d0-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-93e68-predictor-56757bf6ff-5sbjx\" (UID: \"bdf51e1f-4915-4a9c-acff-4d11d63530d0\") " pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx"
Jul 01 19:43:00.759668 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:00.759644    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9bkc\" (UniqueName: \"kubernetes.io/projected/bdf51e1f-4915-4a9c-acff-4d11d63530d0-kube-api-access-w9bkc\") pod \"raw-sklearn-93e68-predictor-56757bf6ff-5sbjx\" (UID: \"bdf51e1f-4915-4a9c-acff-4d11d63530d0\") " pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx"
Jul 01 19:43:01.253626 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:01.253590    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bdf51e1f-4915-4a9c-acff-4d11d63530d0-proxy-tls\") pod \"raw-sklearn-93e68-predictor-56757bf6ff-5sbjx\" (UID: \"bdf51e1f-4915-4a9c-acff-4d11d63530d0\") " pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx"
Jul 01 19:43:01.256099 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:01.256080    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bdf51e1f-4915-4a9c-acff-4d11d63530d0-proxy-tls\") pod \"raw-sklearn-93e68-predictor-56757bf6ff-5sbjx\" (UID: \"bdf51e1f-4915-4a9c-acff-4d11d63530d0\") " pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx"
Jul 01 19:43:01.498061 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:01.498026    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx"
Jul 01 19:43:01.628375 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:01.628332    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx"]
Jul 01 19:43:01.630964 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:43:01.630931    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdf51e1f_4915_4a9c_acff_4d11d63530d0.slice/crio-63233040fa2dfe9a57a905a0887050cbf92e5b3a51a605a24cdaa65bcc7cecb2 WatchSource:0}: Error finding container 63233040fa2dfe9a57a905a0887050cbf92e5b3a51a605a24cdaa65bcc7cecb2: Status 404 returned error can't find the container with id 63233040fa2dfe9a57a905a0887050cbf92e5b3a51a605a24cdaa65bcc7cecb2
Jul 01 19:43:02.447975 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:02.447936    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx" event={"ID":"bdf51e1f-4915-4a9c-acff-4d11d63530d0","Type":"ContainerStarted","Data":"2fc8e2500d003eef2737ec6a7d050224975d5480e8337d90b2082afaf5bac85f"}
Jul 01 19:43:02.448188 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:02.447995    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx" event={"ID":"bdf51e1f-4915-4a9c-acff-4d11d63530d0","Type":"ContainerStarted","Data":"63233040fa2dfe9a57a905a0887050cbf92e5b3a51a605a24cdaa65bcc7cecb2"}
Jul 01 19:43:05.464842 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:05.464752    2574 generic.go:356] "Generic (PLEG): container finished" podID="bdf51e1f-4915-4a9c-acff-4d11d63530d0" containerID="2fc8e2500d003eef2737ec6a7d050224975d5480e8337d90b2082afaf5bac85f" exitCode=0
Jul 01 19:43:05.464842 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:05.464812    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx" event={"ID":"bdf51e1f-4915-4a9c-acff-4d11d63530d0","Type":"ContainerDied","Data":"2fc8e2500d003eef2737ec6a7d050224975d5480e8337d90b2082afaf5bac85f"}
Jul 01 19:43:06.471709 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:06.471670    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx" event={"ID":"bdf51e1f-4915-4a9c-acff-4d11d63530d0","Type":"ContainerStarted","Data":"a59eb6453c33e4bcf63e4df1ae18760da71547435931f229f637e404522c83a6"}
Jul 01 19:43:06.472102 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:06.471739    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx" event={"ID":"bdf51e1f-4915-4a9c-acff-4d11d63530d0","Type":"ContainerStarted","Data":"8ebc228198e8e5cfc438c8ac2a8e1a8823c48c5aca49f3786be92f15dc5d8600"}
Jul 01 19:43:06.472102 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:06.471997    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx"
Jul 01 19:43:06.496044 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:06.495997    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx" podStartSLOduration=3.021517791 podStartE2EDuration="6.495981815s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.474464024s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:43:06.494098956 +0000 UTC m=+1292.586098406" watchObservedRunningTime="2026-07-01 19:43:06.495981815 +0000 UTC m=+1292.587981263"
Jul 01 19:43:07.476084 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:07.476049    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx"
Jul 01 19:43:07.477321 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:07.477293    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx" podUID="bdf51e1f-4915-4a9c-acff-4d11d63530d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 01 19:43:08.480382 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:08.480343    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx" podUID="bdf51e1f-4915-4a9c-acff-4d11d63530d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 01 19:43:13.485380 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:13.485347    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx"
Jul 01 19:43:13.486025 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:13.485851    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx" podUID="bdf51e1f-4915-4a9c-acff-4d11d63530d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 01 19:43:23.485735 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:23.485698    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx" podUID="bdf51e1f-4915-4a9c-acff-4d11d63530d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 01 19:43:33.486781 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:33.486738    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx" podUID="bdf51e1f-4915-4a9c-acff-4d11d63530d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 01 19:43:43.486428 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:43.486385    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx" podUID="bdf51e1f-4915-4a9c-acff-4d11d63530d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 01 19:43:53.486520 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:43:53.486472    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx" podUID="bdf51e1f-4915-4a9c-acff-4d11d63530d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 01 19:44:03.486460 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:03.486408    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx" podUID="bdf51e1f-4915-4a9c-acff-4d11d63530d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 01 19:44:13.485811 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:13.485772    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx" podUID="bdf51e1f-4915-4a9c-acff-4d11d63530d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 01 19:44:23.487076 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:23.487050    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx"
Jul 01 19:44:30.746609 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:30.746573    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx"]
Jul 01 19:44:30.747160 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:30.746862    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx" podUID="bdf51e1f-4915-4a9c-acff-4d11d63530d0" containerName="kserve-container" containerID="cri-o://8ebc228198e8e5cfc438c8ac2a8e1a8823c48c5aca49f3786be92f15dc5d8600" gracePeriod=30
Jul 01 19:44:30.747160 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:30.746951    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx" podUID="bdf51e1f-4915-4a9c-acff-4d11d63530d0" containerName="kube-rbac-proxy" containerID="cri-o://a59eb6453c33e4bcf63e4df1ae18760da71547435931f229f637e404522c83a6" gracePeriod=30
Jul 01 19:44:31.857816 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:31.857781    2574 generic.go:356] "Generic (PLEG): container finished" podID="bdf51e1f-4915-4a9c-acff-4d11d63530d0" containerID="a59eb6453c33e4bcf63e4df1ae18760da71547435931f229f637e404522c83a6" exitCode=2
Jul 01 19:44:31.858323 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:31.857853    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx" event={"ID":"bdf51e1f-4915-4a9c-acff-4d11d63530d0","Type":"ContainerDied","Data":"a59eb6453c33e4bcf63e4df1ae18760da71547435931f229f637e404522c83a6"}
Jul 01 19:44:33.481598 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:33.481547    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx" podUID="bdf51e1f-4915-4a9c-acff-4d11d63530d0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Jul 01 19:44:33.486209 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:33.486175    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx" podUID="bdf51e1f-4915-4a9c-acff-4d11d63530d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 01 19:44:35.303425 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.303399    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx"
Jul 01 19:44:35.353866 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.353843    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bdf51e1f-4915-4a9c-acff-4d11d63530d0-cabundle-cert\") pod \"bdf51e1f-4915-4a9c-acff-4d11d63530d0\" (UID: \"bdf51e1f-4915-4a9c-acff-4d11d63530d0\") "
Jul 01 19:44:35.354024 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.353883    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bdf51e1f-4915-4a9c-acff-4d11d63530d0-proxy-tls\") pod \"bdf51e1f-4915-4a9c-acff-4d11d63530d0\" (UID: \"bdf51e1f-4915-4a9c-acff-4d11d63530d0\") "
Jul 01 19:44:35.354024 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.353928    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bdf51e1f-4915-4a9c-acff-4d11d63530d0-kube-rbac-proxy-sar-config\") pod \"bdf51e1f-4915-4a9c-acff-4d11d63530d0\" (UID: \"bdf51e1f-4915-4a9c-acff-4d11d63530d0\") "
Jul 01 19:44:35.354024 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.353974    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdf51e1f-4915-4a9c-acff-4d11d63530d0-kserve-provision-location\") pod \"bdf51e1f-4915-4a9c-acff-4d11d63530d0\" (UID: \"bdf51e1f-4915-4a9c-acff-4d11d63530d0\") "
Jul 01 19:44:35.354167 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.354030    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9bkc\" (UniqueName: \"kubernetes.io/projected/bdf51e1f-4915-4a9c-acff-4d11d63530d0-kube-api-access-w9bkc\") pod \"bdf51e1f-4915-4a9c-acff-4d11d63530d0\" (UID: \"bdf51e1f-4915-4a9c-acff-4d11d63530d0\") "
Jul 01 19:44:35.354369 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.354341    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdf51e1f-4915-4a9c-acff-4d11d63530d0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bdf51e1f-4915-4a9c-acff-4d11d63530d0" (UID: "bdf51e1f-4915-4a9c-acff-4d11d63530d0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:44:35.354369 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.354348    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdf51e1f-4915-4a9c-acff-4d11d63530d0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bdf51e1f-4915-4a9c-acff-4d11d63530d0" (UID: "bdf51e1f-4915-4a9c-acff-4d11d63530d0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:44:35.354512 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.354352    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdf51e1f-4915-4a9c-acff-4d11d63530d0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bdf51e1f-4915-4a9c-acff-4d11d63530d0" (UID: "bdf51e1f-4915-4a9c-acff-4d11d63530d0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:44:35.356028 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.356008    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdf51e1f-4915-4a9c-acff-4d11d63530d0-kube-api-access-w9bkc" (OuterVolumeSpecName: "kube-api-access-w9bkc") pod "bdf51e1f-4915-4a9c-acff-4d11d63530d0" (UID: "bdf51e1f-4915-4a9c-acff-4d11d63530d0"). InnerVolumeSpecName "kube-api-access-w9bkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:44:35.356140 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.356070    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdf51e1f-4915-4a9c-acff-4d11d63530d0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bdf51e1f-4915-4a9c-acff-4d11d63530d0" (UID: "bdf51e1f-4915-4a9c-acff-4d11d63530d0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:44:35.455308 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.455279    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9bkc\" (UniqueName: \"kubernetes.io/projected/bdf51e1f-4915-4a9c-acff-4d11d63530d0-kube-api-access-w9bkc\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:44:35.455308 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.455305    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bdf51e1f-4915-4a9c-acff-4d11d63530d0-cabundle-cert\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:44:35.455308 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.455315    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bdf51e1f-4915-4a9c-acff-4d11d63530d0-proxy-tls\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:44:35.455521 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.455323    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bdf51e1f-4915-4a9c-acff-4d11d63530d0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:44:35.455521 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.455332    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdf51e1f-4915-4a9c-acff-4d11d63530d0-kserve-provision-location\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:44:35.878158 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.878114    2574 generic.go:356] "Generic (PLEG): container finished" podID="bdf51e1f-4915-4a9c-acff-4d11d63530d0" containerID="8ebc228198e8e5cfc438c8ac2a8e1a8823c48c5aca49f3786be92f15dc5d8600" exitCode=0
Jul 01 19:44:35.878333 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.878176    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx"
Jul 01 19:44:35.878333 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.878196    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx" event={"ID":"bdf51e1f-4915-4a9c-acff-4d11d63530d0","Type":"ContainerDied","Data":"8ebc228198e8e5cfc438c8ac2a8e1a8823c48c5aca49f3786be92f15dc5d8600"}
Jul 01 19:44:35.878333 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.878261    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx" event={"ID":"bdf51e1f-4915-4a9c-acff-4d11d63530d0","Type":"ContainerDied","Data":"63233040fa2dfe9a57a905a0887050cbf92e5b3a51a605a24cdaa65bcc7cecb2"}
Jul 01 19:44:35.878333 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.878290    2574 scope.go:117] "RemoveContainer" containerID="a59eb6453c33e4bcf63e4df1ae18760da71547435931f229f637e404522c83a6"
Jul 01 19:44:35.885610 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.885587    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf"]
Jul 01 19:44:35.885977 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.885958    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdf51e1f-4915-4a9c-acff-4d11d63530d0" containerName="storage-initializer"
Jul 01 19:44:35.886096 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.885986    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdf51e1f-4915-4a9c-acff-4d11d63530d0" containerName="storage-initializer"
Jul 01 19:44:35.886096 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.886021    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdf51e1f-4915-4a9c-acff-4d11d63530d0" containerName="kserve-container"
Jul 01 19:44:35.886096 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.886031    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdf51e1f-4915-4a9c-acff-4d11d63530d0" containerName="kserve-container"
Jul 01 19:44:35.886096 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.886041    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdf51e1f-4915-4a9c-acff-4d11d63530d0" containerName="kube-rbac-proxy"
Jul 01 19:44:35.886096 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.886049    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdf51e1f-4915-4a9c-acff-4d11d63530d0" containerName="kube-rbac-proxy"
Jul 01 19:44:35.886405 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.886164    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdf51e1f-4915-4a9c-acff-4d11d63530d0" containerName="kserve-container"
Jul 01 19:44:35.886405 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.886179    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdf51e1f-4915-4a9c-acff-4d11d63530d0" containerName="kube-rbac-proxy"
Jul 01 19:44:35.887435 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.887389    2574 scope.go:117] "RemoveContainer" containerID="8ebc228198e8e5cfc438c8ac2a8e1a8823c48c5aca49f3786be92f15dc5d8600"
Jul 01 19:44:35.891800 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.891424    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf"
Jul 01 19:44:35.894089 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.894053    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-4e2bb-predictor-serving-cert\""
Jul 01 19:44:35.894089 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.894084    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 19:44:35.894268 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.894095    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-4e2bb-kube-rbac-proxy-sar-config\""
Jul 01 19:44:35.894268 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.894194    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 19:44:35.894268 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.894250    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 19:44:35.894268 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.894260    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-95l22\""
Jul 01 19:44:35.894484 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.894323    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 19:44:35.897397 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.897339    2574 scope.go:117] "RemoveContainer" containerID="2fc8e2500d003eef2737ec6a7d050224975d5480e8337d90b2082afaf5bac85f"
Jul 01 19:44:35.900135 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.900112    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf"]
Jul 01 19:44:35.904818 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.904795    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx"]
Jul 01 19:44:35.907164 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.907144    2574 scope.go:117] "RemoveContainer" containerID="a59eb6453c33e4bcf63e4df1ae18760da71547435931f229f637e404522c83a6"
Jul 01 19:44:35.907449 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:44:35.907431    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a59eb6453c33e4bcf63e4df1ae18760da71547435931f229f637e404522c83a6\": container with ID starting with a59eb6453c33e4bcf63e4df1ae18760da71547435931f229f637e404522c83a6 not found: ID does not exist" containerID="a59eb6453c33e4bcf63e4df1ae18760da71547435931f229f637e404522c83a6"
Jul 01 19:44:35.907525 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.907457    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a59eb6453c33e4bcf63e4df1ae18760da71547435931f229f637e404522c83a6"} err="failed to get container status \"a59eb6453c33e4bcf63e4df1ae18760da71547435931f229f637e404522c83a6\": rpc error: code = NotFound desc = could not find container \"a59eb6453c33e4bcf63e4df1ae18760da71547435931f229f637e404522c83a6\": container with ID starting with a59eb6453c33e4bcf63e4df1ae18760da71547435931f229f637e404522c83a6 not found: ID does not exist"
Jul 01 19:44:35.907525 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.907473    2574 scope.go:117] "RemoveContainer" containerID="8ebc228198e8e5cfc438c8ac2a8e1a8823c48c5aca49f3786be92f15dc5d8600"
Jul 01 19:44:35.907717 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:44:35.907698    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ebc228198e8e5cfc438c8ac2a8e1a8823c48c5aca49f3786be92f15dc5d8600\": container with ID starting with 8ebc228198e8e5cfc438c8ac2a8e1a8823c48c5aca49f3786be92f15dc5d8600 not found: ID does not exist" containerID="8ebc228198e8e5cfc438c8ac2a8e1a8823c48c5aca49f3786be92f15dc5d8600"
Jul 01 19:44:35.907782 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.907721    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ebc228198e8e5cfc438c8ac2a8e1a8823c48c5aca49f3786be92f15dc5d8600"} err="failed to get container status \"8ebc228198e8e5cfc438c8ac2a8e1a8823c48c5aca49f3786be92f15dc5d8600\": rpc error: code = NotFound desc = could not find container \"8ebc228198e8e5cfc438c8ac2a8e1a8823c48c5aca49f3786be92f15dc5d8600\": container with ID starting with 8ebc228198e8e5cfc438c8ac2a8e1a8823c48c5aca49f3786be92f15dc5d8600 not found: ID does not exist"
Jul 01 19:44:35.907782 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.907737    2574 scope.go:117] "RemoveContainer" containerID="2fc8e2500d003eef2737ec6a7d050224975d5480e8337d90b2082afaf5bac85f"
Jul 01 19:44:35.908004 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:44:35.907975    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fc8e2500d003eef2737ec6a7d050224975d5480e8337d90b2082afaf5bac85f\": container with ID starting with 2fc8e2500d003eef2737ec6a7d050224975d5480e8337d90b2082afaf5bac85f not found: ID does not exist" containerID="2fc8e2500d003eef2737ec6a7d050224975d5480e8337d90b2082afaf5bac85f"
Jul 01 19:44:35.908088 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.908008    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fc8e2500d003eef2737ec6a7d050224975d5480e8337d90b2082afaf5bac85f"} err="failed to get container status \"2fc8e2500d003eef2737ec6a7d050224975d5480e8337d90b2082afaf5bac85f\": rpc error: code = NotFound desc = could not find container \"2fc8e2500d003eef2737ec6a7d050224975d5480e8337d90b2082afaf5bac85f\": container with ID starting with 2fc8e2500d003eef2737ec6a7d050224975d5480e8337d90b2082afaf5bac85f not found: ID does not exist"
Jul 01 19:44:35.915334 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.915307    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-93e68-predictor-56757bf6ff-5sbjx"]
Jul 01 19:44:35.959913 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.959880    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/587adb44-937a-4d1a-8dbe-3113389fa36a-cabundle-cert\") pod \"raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf\" (UID: \"587adb44-937a-4d1a-8dbe-3113389fa36a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf"
Jul 01 19:44:35.960024 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.960007    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/587adb44-937a-4d1a-8dbe-3113389fa36a-proxy-tls\") pod \"raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf\" (UID: \"587adb44-937a-4d1a-8dbe-3113389fa36a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf"
Jul 01 19:44:35.960072 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.960035    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/587adb44-937a-4d1a-8dbe-3113389fa36a-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf\" (UID: \"587adb44-937a-4d1a-8dbe-3113389fa36a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf"
Jul 01 19:44:35.960072 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.960062    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqkzk\" (UniqueName: \"kubernetes.io/projected/587adb44-937a-4d1a-8dbe-3113389fa36a-kube-api-access-qqkzk\") pod \"raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf\" (UID: \"587adb44-937a-4d1a-8dbe-3113389fa36a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf"
Jul 01 19:44:35.960159 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:35.960081    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/587adb44-937a-4d1a-8dbe-3113389fa36a-kserve-provision-location\") pod \"raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf\" (UID: \"587adb44-937a-4d1a-8dbe-3113389fa36a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf"
Jul 01 19:44:36.061170 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:36.061128    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/587adb44-937a-4d1a-8dbe-3113389fa36a-cabundle-cert\") pod \"raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf\" (UID: \"587adb44-937a-4d1a-8dbe-3113389fa36a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf"
Jul 01 19:44:36.061321 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:36.061216    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/587adb44-937a-4d1a-8dbe-3113389fa36a-proxy-tls\") pod \"raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf\" (UID: \"587adb44-937a-4d1a-8dbe-3113389fa36a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf"
Jul 01 19:44:36.061321 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:36.061268    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/587adb44-937a-4d1a-8dbe-3113389fa36a-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf\" (UID: \"587adb44-937a-4d1a-8dbe-3113389fa36a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf"
Jul 01 19:44:36.061321 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:36.061302    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqkzk\" (UniqueName: \"kubernetes.io/projected/587adb44-937a-4d1a-8dbe-3113389fa36a-kube-api-access-qqkzk\") pod \"raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf\" (UID: \"587adb44-937a-4d1a-8dbe-3113389fa36a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf"
Jul 01 19:44:36.061492 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:36.061329    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/587adb44-937a-4d1a-8dbe-3113389fa36a-kserve-provision-location\") pod \"raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf\" (UID: \"587adb44-937a-4d1a-8dbe-3113389fa36a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf"
Jul 01 19:44:36.061492 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:44:36.061367    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-serving-cert: secret "raw-sklearn-runtime-4e2bb-predictor-serving-cert" not found
Jul 01 19:44:36.061492 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:44:36.061443    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/587adb44-937a-4d1a-8dbe-3113389fa36a-proxy-tls podName:587adb44-937a-4d1a-8dbe-3113389fa36a nodeName:}" failed. No retries permitted until 2026-07-01 19:44:36.561416552 +0000 UTC m=+1382.653415997 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/587adb44-937a-4d1a-8dbe-3113389fa36a-proxy-tls") pod "raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf" (UID: "587adb44-937a-4d1a-8dbe-3113389fa36a") : secret "raw-sklearn-runtime-4e2bb-predictor-serving-cert" not found
Jul 01 19:44:36.061730 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:36.061711    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/587adb44-937a-4d1a-8dbe-3113389fa36a-kserve-provision-location\") pod \"raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf\" (UID: \"587adb44-937a-4d1a-8dbe-3113389fa36a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf"
Jul 01 19:44:36.061867 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:36.061849    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/587adb44-937a-4d1a-8dbe-3113389fa36a-cabundle-cert\") pod \"raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf\" (UID: \"587adb44-937a-4d1a-8dbe-3113389fa36a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf"
Jul 01 19:44:36.061960 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:36.061944    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/587adb44-937a-4d1a-8dbe-3113389fa36a-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf\" (UID: \"587adb44-937a-4d1a-8dbe-3113389fa36a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf"
Jul 01 19:44:36.070040 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:36.070020    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqkzk\" (UniqueName: \"kubernetes.io/projected/587adb44-937a-4d1a-8dbe-3113389fa36a-kube-api-access-qqkzk\") pod \"raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf\" (UID: \"587adb44-937a-4d1a-8dbe-3113389fa36a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf"
Jul 01 19:44:36.356844 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:36.356813    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdf51e1f-4915-4a9c-acff-4d11d63530d0" path="/var/lib/kubelet/pods/bdf51e1f-4915-4a9c-acff-4d11d63530d0/volumes"
Jul 01 19:44:36.566366 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:36.566333    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/587adb44-937a-4d1a-8dbe-3113389fa36a-proxy-tls\") pod \"raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf\" (UID: \"587adb44-937a-4d1a-8dbe-3113389fa36a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf"
Jul 01 19:44:36.568714 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:36.568692    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/587adb44-937a-4d1a-8dbe-3113389fa36a-proxy-tls\") pod \"raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf\" (UID: \"587adb44-937a-4d1a-8dbe-3113389fa36a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf"
Jul 01 19:44:36.803734 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:36.803649    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf"
Jul 01 19:44:36.938241 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:36.938152    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf"]
Jul 01 19:44:36.940454 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:44:36.940419    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod587adb44_937a_4d1a_8dbe_3113389fa36a.slice/crio-c2589baccbcd91d56454557a616828626345ce3b16a7ef010c56f540c8dc18f2 WatchSource:0}: Error finding container c2589baccbcd91d56454557a616828626345ce3b16a7ef010c56f540c8dc18f2: Status 404 returned error can't find the container with id c2589baccbcd91d56454557a616828626345ce3b16a7ef010c56f540c8dc18f2
Jul 01 19:44:37.891436 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:37.891400    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf" event={"ID":"587adb44-937a-4d1a-8dbe-3113389fa36a","Type":"ContainerStarted","Data":"95c7519f7df54b16cd91a2c00c62adf9a363de65b00c7f577a6d495ca7a197f9"}
Jul 01 19:44:37.891824 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:37.891468    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf" event={"ID":"587adb44-937a-4d1a-8dbe-3113389fa36a","Type":"ContainerStarted","Data":"c2589baccbcd91d56454557a616828626345ce3b16a7ef010c56f540c8dc18f2"}
Jul 01 19:44:40.905943 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:40.905890    2574 generic.go:356] "Generic (PLEG): container finished" podID="587adb44-937a-4d1a-8dbe-3113389fa36a" containerID="95c7519f7df54b16cd91a2c00c62adf9a363de65b00c7f577a6d495ca7a197f9" exitCode=0
Jul 01 19:44:40.906405 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:40.905962    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf" event={"ID":"587adb44-937a-4d1a-8dbe-3113389fa36a","Type":"ContainerDied","Data":"95c7519f7df54b16cd91a2c00c62adf9a363de65b00c7f577a6d495ca7a197f9"}
Jul 01 19:44:41.912921 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:41.912862    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf" event={"ID":"587adb44-937a-4d1a-8dbe-3113389fa36a","Type":"ContainerStarted","Data":"0b3aab5829431ec1a7f4765ef381a7add322720de194bce140553c152bc34a5e"}
Jul 01 19:44:41.913301 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:41.912952    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf" event={"ID":"587adb44-937a-4d1a-8dbe-3113389fa36a","Type":"ContainerStarted","Data":"8ab89ed5b73d98480964c6857214babe384e127ef84b73a2e034900260159ef1"}
Jul 01 19:44:41.913301 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:41.913147    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf"
Jul 01 19:44:41.937405 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:41.937341    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf" podStartSLOduration=3.182654341 podStartE2EDuration="6.937320325s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.754665984s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:44:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:44:41.935837166 +0000 UTC m=+1388.027836614" watchObservedRunningTime="2026-07-01 19:44:41.937320325 +0000 UTC m=+1388.029319777"
Jul 01 19:44:42.918044 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:42.918011    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf"
Jul 01 19:44:42.919144 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:42.919117    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf" podUID="587adb44-937a-4d1a-8dbe-3113389fa36a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 01 19:44:43.922390 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:43.922349    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf" podUID="587adb44-937a-4d1a-8dbe-3113389fa36a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 01 19:44:48.927091 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:48.927056    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf"
Jul 01 19:44:48.927723 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:48.927693    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf" podUID="587adb44-937a-4d1a-8dbe-3113389fa36a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 01 19:44:58.928332 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:44:58.928287    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf" podUID="587adb44-937a-4d1a-8dbe-3113389fa36a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 01 19:45:08.927786 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:45:08.927743    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf" podUID="587adb44-937a-4d1a-8dbe-3113389fa36a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 01 19:45:18.928531 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:45:18.928491    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf" podUID="587adb44-937a-4d1a-8dbe-3113389fa36a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 01 19:45:28.928666 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:45:28.928582    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf" podUID="587adb44-937a-4d1a-8dbe-3113389fa36a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 01 19:45:38.928044 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:45:38.928000    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf" podUID="587adb44-937a-4d1a-8dbe-3113389fa36a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 01 19:45:48.927869 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:45:48.927822    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf" podUID="587adb44-937a-4d1a-8dbe-3113389fa36a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 01 19:45:54.358000 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:45:54.357970    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf"
Jul 01 19:45:55.982380 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:45:55.982341    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf"]
Jul 01 19:45:56.242610 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:45:56.242521    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf" podUID="587adb44-937a-4d1a-8dbe-3113389fa36a" containerName="kube-rbac-proxy" containerID="cri-o://0b3aab5829431ec1a7f4765ef381a7add322720de194bce140553c152bc34a5e" gracePeriod=30
Jul 01 19:45:56.242768 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:45:56.242644    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf" podUID="587adb44-937a-4d1a-8dbe-3113389fa36a" containerName="kserve-container" containerID="cri-o://8ab89ed5b73d98480964c6857214babe384e127ef84b73a2e034900260159ef1" gracePeriod=30
Jul 01 19:45:57.252043 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:45:57.252001    2574 generic.go:356] "Generic (PLEG): container finished" podID="587adb44-937a-4d1a-8dbe-3113389fa36a" containerID="0b3aab5829431ec1a7f4765ef381a7add322720de194bce140553c152bc34a5e" exitCode=2
Jul 01 19:45:57.252428 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:45:57.252045    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf" event={"ID":"587adb44-937a-4d1a-8dbe-3113389fa36a","Type":"ContainerDied","Data":"0b3aab5829431ec1a7f4765ef381a7add322720de194bce140553c152bc34a5e"}
Jul 01 19:45:58.923005 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:45:58.922964    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf" podUID="587adb44-937a-4d1a-8dbe-3113389fa36a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Jul 01 19:46:00.992222 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:00.992195    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf"
Jul 01 19:46:01.069164 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:01.069083    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/587adb44-937a-4d1a-8dbe-3113389fa36a-cabundle-cert\") pod \"587adb44-937a-4d1a-8dbe-3113389fa36a\" (UID: \"587adb44-937a-4d1a-8dbe-3113389fa36a\") "
Jul 01 19:46:01.069164 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:01.069123    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/587adb44-937a-4d1a-8dbe-3113389fa36a-proxy-tls\") pod \"587adb44-937a-4d1a-8dbe-3113389fa36a\" (UID: \"587adb44-937a-4d1a-8dbe-3113389fa36a\") "
Jul 01 19:46:01.069164 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:01.069162    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/587adb44-937a-4d1a-8dbe-3113389fa36a-kserve-provision-location\") pod \"587adb44-937a-4d1a-8dbe-3113389fa36a\" (UID: \"587adb44-937a-4d1a-8dbe-3113389fa36a\") "
Jul 01 19:46:01.069444 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:01.069181    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/587adb44-937a-4d1a-8dbe-3113389fa36a-kube-rbac-proxy-sar-config\") pod \"587adb44-937a-4d1a-8dbe-3113389fa36a\" (UID: \"587adb44-937a-4d1a-8dbe-3113389fa36a\") "
Jul 01 19:46:01.069444 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:01.069207    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqkzk\" (UniqueName: \"kubernetes.io/projected/587adb44-937a-4d1a-8dbe-3113389fa36a-kube-api-access-qqkzk\") pod \"587adb44-937a-4d1a-8dbe-3113389fa36a\" (UID: \"587adb44-937a-4d1a-8dbe-3113389fa36a\") "
Jul 01 19:46:01.069444 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:01.069423    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/587adb44-937a-4d1a-8dbe-3113389fa36a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "587adb44-937a-4d1a-8dbe-3113389fa36a" (UID: "587adb44-937a-4d1a-8dbe-3113389fa36a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:46:01.069619 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:01.069493    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/587adb44-937a-4d1a-8dbe-3113389fa36a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "587adb44-937a-4d1a-8dbe-3113389fa36a" (UID: "587adb44-937a-4d1a-8dbe-3113389fa36a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:46:01.069619 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:01.069549    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/587adb44-937a-4d1a-8dbe-3113389fa36a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "587adb44-937a-4d1a-8dbe-3113389fa36a" (UID: "587adb44-937a-4d1a-8dbe-3113389fa36a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:46:01.071365 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:01.071332    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/587adb44-937a-4d1a-8dbe-3113389fa36a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "587adb44-937a-4d1a-8dbe-3113389fa36a" (UID: "587adb44-937a-4d1a-8dbe-3113389fa36a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:46:01.071467 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:01.071364    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/587adb44-937a-4d1a-8dbe-3113389fa36a-kube-api-access-qqkzk" (OuterVolumeSpecName: "kube-api-access-qqkzk") pod "587adb44-937a-4d1a-8dbe-3113389fa36a" (UID: "587adb44-937a-4d1a-8dbe-3113389fa36a"). InnerVolumeSpecName "kube-api-access-qqkzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:46:01.170555 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:01.170511    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/587adb44-937a-4d1a-8dbe-3113389fa36a-cabundle-cert\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:46:01.170555 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:01.170550    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/587adb44-937a-4d1a-8dbe-3113389fa36a-proxy-tls\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:46:01.170816 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:01.170564    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/587adb44-937a-4d1a-8dbe-3113389fa36a-kserve-provision-location\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:46:01.170816 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:01.170579    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/587adb44-937a-4d1a-8dbe-3113389fa36a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:46:01.170816 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:01.170592    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqkzk\" (UniqueName: \"kubernetes.io/projected/587adb44-937a-4d1a-8dbe-3113389fa36a-kube-api-access-qqkzk\") on node \"ip-10-0-137-164.ec2.internal\" DevicePath \"\""
Jul 01 19:46:01.273303 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:01.273265    2574 generic.go:356] "Generic (PLEG): container finished" podID="587adb44-937a-4d1a-8dbe-3113389fa36a" containerID="8ab89ed5b73d98480964c6857214babe384e127ef84b73a2e034900260159ef1" exitCode=0
Jul 01 19:46:01.273472 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:01.273324    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf"
Jul 01 19:46:01.273472 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:01.273351    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf" event={"ID":"587adb44-937a-4d1a-8dbe-3113389fa36a","Type":"ContainerDied","Data":"8ab89ed5b73d98480964c6857214babe384e127ef84b73a2e034900260159ef1"}
Jul 01 19:46:01.273472 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:01.273410    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf" event={"ID":"587adb44-937a-4d1a-8dbe-3113389fa36a","Type":"ContainerDied","Data":"c2589baccbcd91d56454557a616828626345ce3b16a7ef010c56f540c8dc18f2"}
Jul 01 19:46:01.273472 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:01.273439    2574 scope.go:117] "RemoveContainer" containerID="0b3aab5829431ec1a7f4765ef381a7add322720de194bce140553c152bc34a5e"
Jul 01 19:46:01.282553 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:01.282533    2574 scope.go:117] "RemoveContainer" containerID="8ab89ed5b73d98480964c6857214babe384e127ef84b73a2e034900260159ef1"
Jul 01 19:46:01.289778 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:01.289761    2574 scope.go:117] "RemoveContainer" containerID="95c7519f7df54b16cd91a2c00c62adf9a363de65b00c7f577a6d495ca7a197f9"
Jul 01 19:46:01.297200 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:01.297183    2574 scope.go:117] "RemoveContainer" containerID="0b3aab5829431ec1a7f4765ef381a7add322720de194bce140553c152bc34a5e"
Jul 01 19:46:01.297436 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:46:01.297418    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b3aab5829431ec1a7f4765ef381a7add322720de194bce140553c152bc34a5e\": container with ID starting with 0b3aab5829431ec1a7f4765ef381a7add322720de194bce140553c152bc34a5e not found: ID does not exist" containerID="0b3aab5829431ec1a7f4765ef381a7add322720de194bce140553c152bc34a5e"
Jul 01 19:46:01.297493 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:01.297443    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b3aab5829431ec1a7f4765ef381a7add322720de194bce140553c152bc34a5e"} err="failed to get container status \"0b3aab5829431ec1a7f4765ef381a7add322720de194bce140553c152bc34a5e\": rpc error: code = NotFound desc = could not find container \"0b3aab5829431ec1a7f4765ef381a7add322720de194bce140553c152bc34a5e\": container with ID starting with 0b3aab5829431ec1a7f4765ef381a7add322720de194bce140553c152bc34a5e not found: ID does not exist"
Jul 01 19:46:01.297493 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:01.297461    2574 scope.go:117] "RemoveContainer" containerID="8ab89ed5b73d98480964c6857214babe384e127ef84b73a2e034900260159ef1"
Jul 01 19:46:01.297585 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:01.297507    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf"]
Jul 01 19:46:01.297663 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:46:01.297648    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ab89ed5b73d98480964c6857214babe384e127ef84b73a2e034900260159ef1\": container with ID starting with 8ab89ed5b73d98480964c6857214babe384e127ef84b73a2e034900260159ef1 not found: ID does not exist" containerID="8ab89ed5b73d98480964c6857214babe384e127ef84b73a2e034900260159ef1"
Jul 01 19:46:01.297702 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:01.297665    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ab89ed5b73d98480964c6857214babe384e127ef84b73a2e034900260159ef1"} err="failed to get container status \"8ab89ed5b73d98480964c6857214babe384e127ef84b73a2e034900260159ef1\": rpc error: code = NotFound desc = could not find container \"8ab89ed5b73d98480964c6857214babe384e127ef84b73a2e034900260159ef1\": container with ID starting with 8ab89ed5b73d98480964c6857214babe384e127ef84b73a2e034900260159ef1 not found: ID does not exist"
Jul 01 19:46:01.297702 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:01.297678    2574 scope.go:117] "RemoveContainer" containerID="95c7519f7df54b16cd91a2c00c62adf9a363de65b00c7f577a6d495ca7a197f9"
Jul 01 19:46:01.297940 ip-10-0-137-164 kubenswrapper[2574]: E0701 19:46:01.297922    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95c7519f7df54b16cd91a2c00c62adf9a363de65b00c7f577a6d495ca7a197f9\": container with ID starting with 95c7519f7df54b16cd91a2c00c62adf9a363de65b00c7f577a6d495ca7a197f9 not found: ID does not exist" containerID="95c7519f7df54b16cd91a2c00c62adf9a363de65b00c7f577a6d495ca7a197f9"
Jul 01 19:46:01.298006 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:01.297948    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95c7519f7df54b16cd91a2c00c62adf9a363de65b00c7f577a6d495ca7a197f9"} err="failed to get container status \"95c7519f7df54b16cd91a2c00c62adf9a363de65b00c7f577a6d495ca7a197f9\": rpc error: code = NotFound desc = could not find container \"95c7519f7df54b16cd91a2c00c62adf9a363de65b00c7f577a6d495ca7a197f9\": container with ID starting with 95c7519f7df54b16cd91a2c00c62adf9a363de65b00c7f577a6d495ca7a197f9 not found: ID does not exist"
Jul 01 19:46:01.301506 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:01.301485    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-4e2bb-predictor-784b69577f-txrtf"]
Jul 01 19:46:02.356707 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:02.356676    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="587adb44-937a-4d1a-8dbe-3113389fa36a" path="/var/lib/kubelet/pods/587adb44-937a-4d1a-8dbe-3113389fa36a/volumes"
Jul 01 19:46:32.296735 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:32.296701    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-f2cj7/must-gather-h5trn"]
Jul 01 19:46:32.297234 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:32.297046    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="587adb44-937a-4d1a-8dbe-3113389fa36a" containerName="storage-initializer"
Jul 01 19:46:32.297234 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:32.297058    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="587adb44-937a-4d1a-8dbe-3113389fa36a" containerName="storage-initializer"
Jul 01 19:46:32.297234 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:32.297070    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="587adb44-937a-4d1a-8dbe-3113389fa36a" containerName="kube-rbac-proxy"
Jul 01 19:46:32.297234 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:32.297076    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="587adb44-937a-4d1a-8dbe-3113389fa36a" containerName="kube-rbac-proxy"
Jul 01 19:46:32.297234 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:32.297085    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="587adb44-937a-4d1a-8dbe-3113389fa36a" containerName="kserve-container"
Jul 01 19:46:32.297234 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:32.297090    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="587adb44-937a-4d1a-8dbe-3113389fa36a" containerName="kserve-container"
Jul 01 19:46:32.297234 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:32.297144    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="587adb44-937a-4d1a-8dbe-3113389fa36a" containerName="kube-rbac-proxy"
Jul 01 19:46:32.297234 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:32.297154    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="587adb44-937a-4d1a-8dbe-3113389fa36a" containerName="kserve-container"
Jul 01 19:46:32.300416 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:32.300398    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f2cj7/must-gather-h5trn"
Jul 01 19:46:32.303239 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:32.303216    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f2cj7\"/\"kube-root-ca.crt\""
Jul 01 19:46:32.303353 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:32.303284    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f2cj7\"/\"openshift-service-ca.crt\""
Jul 01 19:46:32.303559 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:32.303541    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-f2cj7\"/\"default-dockercfg-fpbtf\""
Jul 01 19:46:32.310690 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:32.310668    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f2cj7/must-gather-h5trn"]
Jul 01 19:46:32.433756 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:32.433722    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9b6df6ee-efda-4179-a2bb-26c05c74245a-must-gather-output\") pod \"must-gather-h5trn\" (UID: \"9b6df6ee-efda-4179-a2bb-26c05c74245a\") " pod="openshift-must-gather-f2cj7/must-gather-h5trn"
Jul 01 19:46:32.433998 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:32.433861    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mts8c\" (UniqueName: \"kubernetes.io/projected/9b6df6ee-efda-4179-a2bb-26c05c74245a-kube-api-access-mts8c\") pod \"must-gather-h5trn\" (UID: \"9b6df6ee-efda-4179-a2bb-26c05c74245a\") " pod="openshift-must-gather-f2cj7/must-gather-h5trn"
Jul 01 19:46:32.535416 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:32.535386    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mts8c\" (UniqueName: \"kubernetes.io/projected/9b6df6ee-efda-4179-a2bb-26c05c74245a-kube-api-access-mts8c\") pod \"must-gather-h5trn\" (UID: \"9b6df6ee-efda-4179-a2bb-26c05c74245a\") " pod="openshift-must-gather-f2cj7/must-gather-h5trn"
Jul 01 19:46:32.535570 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:32.535456    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9b6df6ee-efda-4179-a2bb-26c05c74245a-must-gather-output\") pod \"must-gather-h5trn\" (UID: \"9b6df6ee-efda-4179-a2bb-26c05c74245a\") " pod="openshift-must-gather-f2cj7/must-gather-h5trn"
Jul 01 19:46:32.535782 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:32.535764    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9b6df6ee-efda-4179-a2bb-26c05c74245a-must-gather-output\") pod \"must-gather-h5trn\" (UID: \"9b6df6ee-efda-4179-a2bb-26c05c74245a\") " pod="openshift-must-gather-f2cj7/must-gather-h5trn"
Jul 01 19:46:32.545254 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:32.545231    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mts8c\" (UniqueName: \"kubernetes.io/projected/9b6df6ee-efda-4179-a2bb-26c05c74245a-kube-api-access-mts8c\") pod \"must-gather-h5trn\" (UID: \"9b6df6ee-efda-4179-a2bb-26c05c74245a\") " pod="openshift-must-gather-f2cj7/must-gather-h5trn"
Jul 01 19:46:32.610014 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:32.609988    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f2cj7/must-gather-h5trn"
Jul 01 19:46:32.735169 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:32.735142    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f2cj7/must-gather-h5trn"]
Jul 01 19:46:32.737458 ip-10-0-137-164 kubenswrapper[2574]: W0701 19:46:32.737423    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b6df6ee_efda_4179_a2bb_26c05c74245a.slice/crio-a65077b98d7acda8a93f745d814817e4ea14fc7cf0172c08300c738699ce1292 WatchSource:0}: Error finding container a65077b98d7acda8a93f745d814817e4ea14fc7cf0172c08300c738699ce1292: Status 404 returned error can't find the container with id a65077b98d7acda8a93f745d814817e4ea14fc7cf0172c08300c738699ce1292
Jul 01 19:46:32.739235 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:32.739217    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 19:46:33.422505 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:33.422455    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f2cj7/must-gather-h5trn" event={"ID":"9b6df6ee-efda-4179-a2bb-26c05c74245a","Type":"ContainerStarted","Data":"a65077b98d7acda8a93f745d814817e4ea14fc7cf0172c08300c738699ce1292"}
Jul 01 19:46:34.435102 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:34.435060    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f2cj7/must-gather-h5trn" event={"ID":"9b6df6ee-efda-4179-a2bb-26c05c74245a","Type":"ContainerStarted","Data":"07d7055b9f9d2727b5190f9b3484c8d469018122a32bc41c1bbd1eae813d62b5"}
Jul 01 19:46:34.435102 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:34.435116    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f2cj7/must-gather-h5trn" event={"ID":"9b6df6ee-efda-4179-a2bb-26c05c74245a","Type":"ContainerStarted","Data":"cdd10daee9ce066b44f85049637e207ce140829fdcba3e54fd5f6c878a8d188f"}
Jul 01 19:46:34.458588 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:34.458559    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bplsm_bfb845e6-341d-492e-91c9-9c698a0b8d29/ovn-acl-logging/0.log"
Jul 01 19:46:34.465134 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:34.465104    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bplsm_bfb845e6-341d-492e-91c9-9c698a0b8d29/ovn-acl-logging/0.log"
Jul 01 19:46:34.986179 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:34.986154    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mq22h_f9eaa621-a5fa-49a2-ad92-8dcd5729803a/global-pull-secret-syncer/0.log"
Jul 01 19:46:35.202240 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:35.202211    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mch4p_9b5d6712-c210-43d7-ae23-6fe6bb85ef6c/konnectivity-agent/0.log"
Jul 01 19:46:35.251407 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:35.251333    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-164.ec2.internal_f180425c803edc40a93bf63fc1f451d4/haproxy/0.log"
Jul 01 19:46:38.688853 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:38.688817    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3bf3ec80-44ad-4d65-a942-94c74f8aa1b3/alertmanager/0.log"
Jul 01 19:46:38.724806 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:38.724758    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3bf3ec80-44ad-4d65-a942-94c74f8aa1b3/config-reloader/0.log"
Jul 01 19:46:38.753245 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:38.753207    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3bf3ec80-44ad-4d65-a942-94c74f8aa1b3/kube-rbac-proxy-web/0.log"
Jul 01 19:46:38.783437 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:38.783381    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3bf3ec80-44ad-4d65-a942-94c74f8aa1b3/kube-rbac-proxy/0.log"
Jul 01 19:46:38.807646 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:38.807619    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3bf3ec80-44ad-4d65-a942-94c74f8aa1b3/kube-rbac-proxy-metric/0.log"
Jul 01 19:46:38.843939 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:38.843889    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3bf3ec80-44ad-4d65-a942-94c74f8aa1b3/prom-label-proxy/0.log"
Jul 01 19:46:38.876950 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:38.876920    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3bf3ec80-44ad-4d65-a942-94c74f8aa1b3/init-config-reloader/0.log"
Jul 01 19:46:38.968719 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:38.968595    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-gjdxq_074f32fe-7406-4e01-9868-da1ec706d589/kube-state-metrics/0.log"
Jul 01 19:46:38.999428 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:38.999386    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-gjdxq_074f32fe-7406-4e01-9868-da1ec706d589/kube-rbac-proxy-main/0.log"
Jul 01 19:46:39.036680 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:39.036645    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-gjdxq_074f32fe-7406-4e01-9868-da1ec706d589/kube-rbac-proxy-self/0.log"
Jul 01 19:46:39.369258 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:39.369198    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ntch4_8638dc64-df92-4461-b287-314a1fe5944e/node-exporter/0.log"
Jul 01 19:46:39.399406 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:39.399355    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ntch4_8638dc64-df92-4461-b287-314a1fe5944e/kube-rbac-proxy/0.log"
Jul 01 19:46:39.428127 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:39.428099    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ntch4_8638dc64-df92-4461-b287-314a1fe5944e/init-textfile/0.log"
Jul 01 19:46:39.456802 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:39.456776    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-cxsgw_2ceea4b5-bcc4-405c-9364-c2dc815b0c97/kube-rbac-proxy-main/0.log"
Jul 01 19:46:39.484407 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:39.484356    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-cxsgw_2ceea4b5-bcc4-405c-9364-c2dc815b0c97/kube-rbac-proxy-self/0.log"
Jul 01 19:46:39.511876 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:39.511836    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-cxsgw_2ceea4b5-bcc4-405c-9364-c2dc815b0c97/openshift-state-metrics/0.log"
Jul 01 19:46:39.981739 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:39.981652    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7b7d896497-8wxbh_c2063d45-dbf3-402d-8a70-ea43861e5380/prometheus-operator/0.log"
Jul 01 19:46:40.005393 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:40.005358    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7b7d896497-8wxbh_c2063d45-dbf3-402d-8a70-ea43861e5380/kube-rbac-proxy/0.log"
Jul 01 19:46:40.079434 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:40.079398    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7cdc9756fd-hntp7_79bc6c96-6740-4296-8fa4-a0da34a98370/telemeter-client/0.log"
Jul 01 19:46:40.104824 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:40.104798    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7cdc9756fd-hntp7_79bc6c96-6740-4296-8fa4-a0da34a98370/reload/0.log"
Jul 01 19:46:40.131305 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:40.131270    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7cdc9756fd-hntp7_79bc6c96-6740-4296-8fa4-a0da34a98370/kube-rbac-proxy/0.log"
Jul 01 19:46:40.166596 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:40.166567    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6df8f7d99d-rdvgb_95c3faaa-6932-4f96-bb46-24fdd489b653/thanos-query/0.log"
Jul 01 19:46:40.193581 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:40.193551    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6df8f7d99d-rdvgb_95c3faaa-6932-4f96-bb46-24fdd489b653/kube-rbac-proxy-web/0.log"
Jul 01 19:46:40.228652 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:40.228599    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6df8f7d99d-rdvgb_95c3faaa-6932-4f96-bb46-24fdd489b653/kube-rbac-proxy/0.log"
Jul 01 19:46:40.252944 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:40.252852    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6df8f7d99d-rdvgb_95c3faaa-6932-4f96-bb46-24fdd489b653/prom-label-proxy/0.log"
Jul 01 19:46:40.283977 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:40.283947    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6df8f7d99d-rdvgb_95c3faaa-6932-4f96-bb46-24fdd489b653/kube-rbac-proxy-rules/0.log"
Jul 01 19:46:40.309965 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:40.309931    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6df8f7d99d-rdvgb_95c3faaa-6932-4f96-bb46-24fdd489b653/kube-rbac-proxy-metrics/0.log"
Jul 01 19:46:42.705774 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:42.705695    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-85df97f456-m66xl_bf2583a0-c190-4f5f-8b48-bfef7a3316fa/console/0.log"
Jul 01 19:46:42.742294 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:42.742258    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5499b6fb8f-g2fv2_d20bf32b-d17e-47a2-a6d0-809d3e465a08/download-server/0.log"
Jul 01 19:46:43.336891 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:43.336822    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-f2cj7/must-gather-h5trn" podStartSLOduration=10.563827598 podStartE2EDuration="11.336800597s" totalImagesPullingTime="772.972999ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:46:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:46:34.452542736 +0000 UTC m=+1500.544542210" watchObservedRunningTime="2026-07-01 19:46:43.336800597 +0000 UTC m=+1509.428800046"
Jul 01 19:46:43.339265 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:43.339234    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-f2cj7/perf-node-gather-daemonset-zr6bf"]
Jul 01 19:46:43.345919 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:43.345882    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f2cj7/perf-node-gather-daemonset-zr6bf"
Jul 01 19:46:43.353190 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:43.353155    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f2cj7/perf-node-gather-daemonset-zr6bf"]
Jul 01 19:46:43.438602 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:43.438567    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/37fe1be9-d9c5-42f1-ab39-28d84d8ad237-podres\") pod \"perf-node-gather-daemonset-zr6bf\" (UID: \"37fe1be9-d9c5-42f1-ab39-28d84d8ad237\") " pod="openshift-must-gather-f2cj7/perf-node-gather-daemonset-zr6bf"
Jul 01 19:46:43.438780 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:43.438610    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/37fe1be9-d9c5-42f1-ab39-28d84d8ad237-sys\") pod \"perf-node-gather-daemonset-zr6bf\" (UID: \"37fe1be9-d9c5-42f1-ab39-28d84d8ad237\") " pod="openshift-must-gather-f2cj7/perf-node-gather-daemonset-zr6bf"
Jul 01 19:46:43.438780 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:43.438716    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/37fe1be9-d9c5-42f1-ab39-28d84d8ad237-proc\") pod \"perf-node-gather-daemonset-zr6bf\" (UID: \"37fe1be9-d9c5-42f1-ab39-28d84d8ad237\") " pod="openshift-must-gather-f2cj7/perf-node-gather-daemonset-zr6bf"
Jul 01 19:46:43.438931 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:43.438813    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bckl\" (UniqueName: \"kubernetes.io/projected/37fe1be9-d9c5-42f1-ab39-28d84d8ad237-kube-api-access-9bckl\") pod \"perf-node-gather-daemonset-zr6bf\" (UID: \"37fe1be9-d9c5-42f1-ab39-28d84d8ad237\") " pod="openshift-must-gather-f2cj7/perf-node-gather-daemonset-zr6bf"
Jul 01 19:46:43.438931 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:43.438861    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/37fe1be9-d9c5-42f1-ab39-28d84d8ad237-lib-modules\") pod \"perf-node-gather-daemonset-zr6bf\" (UID: \"37fe1be9-d9c5-42f1-ab39-28d84d8ad237\") " pod="openshift-must-gather-f2cj7/perf-node-gather-daemonset-zr6bf"
Jul 01 19:46:43.541011 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:43.540090    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/37fe1be9-d9c5-42f1-ab39-28d84d8ad237-podres\") pod \"perf-node-gather-daemonset-zr6bf\" (UID: \"37fe1be9-d9c5-42f1-ab39-28d84d8ad237\") " pod="openshift-must-gather-f2cj7/perf-node-gather-daemonset-zr6bf"
Jul 01 19:46:43.541011 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:43.540145    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/37fe1be9-d9c5-42f1-ab39-28d84d8ad237-sys\") pod \"perf-node-gather-daemonset-zr6bf\" (UID: \"37fe1be9-d9c5-42f1-ab39-28d84d8ad237\") " pod="openshift-must-gather-f2cj7/perf-node-gather-daemonset-zr6bf"
Jul 01 19:46:43.541011 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:43.540206    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/37fe1be9-d9c5-42f1-ab39-28d84d8ad237-proc\") pod \"perf-node-gather-daemonset-zr6bf\" (UID: \"37fe1be9-d9c5-42f1-ab39-28d84d8ad237\") " pod="openshift-must-gather-f2cj7/perf-node-gather-daemonset-zr6bf"
Jul 01 19:46:43.541011 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:43.540263    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bckl\" (UniqueName: \"kubernetes.io/projected/37fe1be9-d9c5-42f1-ab39-28d84d8ad237-kube-api-access-9bckl\") pod \"perf-node-gather-daemonset-zr6bf\" (UID: \"37fe1be9-d9c5-42f1-ab39-28d84d8ad237\") " pod="openshift-must-gather-f2cj7/perf-node-gather-daemonset-zr6bf"
Jul 01 19:46:43.541011 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:43.540306    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/37fe1be9-d9c5-42f1-ab39-28d84d8ad237-lib-modules\") pod \"perf-node-gather-daemonset-zr6bf\" (UID: \"37fe1be9-d9c5-42f1-ab39-28d84d8ad237\") " pod="openshift-must-gather-f2cj7/perf-node-gather-daemonset-zr6bf"
Jul 01 19:46:43.541011 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:43.540469    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/37fe1be9-d9c5-42f1-ab39-28d84d8ad237-lib-modules\") pod \"perf-node-gather-daemonset-zr6bf\" (UID: \"37fe1be9-d9c5-42f1-ab39-28d84d8ad237\") " pod="openshift-must-gather-f2cj7/perf-node-gather-daemonset-zr6bf"
Jul 01 19:46:43.541011 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:43.540557    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/37fe1be9-d9c5-42f1-ab39-28d84d8ad237-podres\") pod \"perf-node-gather-daemonset-zr6bf\" (UID: \"37fe1be9-d9c5-42f1-ab39-28d84d8ad237\") " pod="openshift-must-gather-f2cj7/perf-node-gather-daemonset-zr6bf"
Jul 01 19:46:43.541011 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:43.540604    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/37fe1be9-d9c5-42f1-ab39-28d84d8ad237-sys\") pod \"perf-node-gather-daemonset-zr6bf\" (UID: \"37fe1be9-d9c5-42f1-ab39-28d84d8ad237\") " pod="openshift-must-gather-f2cj7/perf-node-gather-daemonset-zr6bf"
Jul 01 19:46:43.541011 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:43.540648    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/37fe1be9-d9c5-42f1-ab39-28d84d8ad237-proc\") pod \"perf-node-gather-daemonset-zr6bf\" (UID: \"37fe1be9-d9c5-42f1-ab39-28d84d8ad237\") " pod="openshift-must-gather-f2cj7/perf-node-gather-daemonset-zr6bf"
Jul 01 19:46:43.550165 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:43.550103    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bckl\" (UniqueName: \"kubernetes.io/projected/37fe1be9-d9c5-42f1-ab39-28d84d8ad237-kube-api-access-9bckl\") pod \"perf-node-gather-daemonset-zr6bf\" (UID: \"37fe1be9-d9c5-42f1-ab39-28d84d8ad237\") " pod="openshift-must-gather-f2cj7/perf-node-gather-daemonset-zr6bf"
Jul 01 19:46:43.662595 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:43.662557    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f2cj7/perf-node-gather-daemonset-zr6bf"
Jul 01 19:46:43.838757 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:43.838419    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f2cj7/perf-node-gather-daemonset-zr6bf"]
Jul 01 19:46:44.379289 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:44.379260    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n29hh_58b6a949-f2a2-41a0-9f47-79c420dd33ef/dns/0.log"
Jul 01 19:46:44.408586 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:44.408555    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n29hh_58b6a949-f2a2-41a0-9f47-79c420dd33ef/kube-rbac-proxy/0.log"
Jul 01 19:46:44.499411 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:44.499370    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f2cj7/perf-node-gather-daemonset-zr6bf" event={"ID":"37fe1be9-d9c5-42f1-ab39-28d84d8ad237","Type":"ContainerStarted","Data":"2f1cdfb9a1ef9d899268b6ea36a58d3b08dfdacd0272dd8c5713916aed027cb7"}
Jul 01 19:46:44.499611 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:44.499437    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f2cj7/perf-node-gather-daemonset-zr6bf" event={"ID":"37fe1be9-d9c5-42f1-ab39-28d84d8ad237","Type":"ContainerStarted","Data":"61e7b9a932b0f7d86ecb1d1d23c951241e94b0c63f6a52e36089a09ec47df6eb"}
Jul 01 19:46:44.500236 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:44.500199    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-f2cj7/perf-node-gather-daemonset-zr6bf"
Jul 01 19:46:44.520458 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:44.520398    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-f2cj7/perf-node-gather-daemonset-zr6bf" podStartSLOduration=1.5203699560000001 podStartE2EDuration="1.520369956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:46:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:46:44.517240604 +0000 UTC m=+1510.609240052" watchObservedRunningTime="2026-07-01 19:46:44.520369956 +0000 UTC m=+1510.612369406"
Jul 01 19:46:44.522409 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:44.522389    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-p58v4_8d985f94-c35c-431b-bd74-89873c00e1e6/dns-node-resolver/0.log"
Jul 01 19:46:45.075408 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:45.075362    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kjtgf_846ea9a9-80a2-4628-a2da-e3224610bf57/node-ca/0.log"
Jul 01 19:46:46.391536 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:46.391501    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ndjvj_a0e57b1d-40fc-4eef-86f4-51970c68b93b/serve-healthcheck-canary/0.log"
Jul 01 19:46:46.901443 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:46.901421    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n8l46_3359c7fa-a51a-4dc7-b46e-c99a59154ba9/kube-rbac-proxy/0.log"
Jul 01 19:46:46.926056 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:46.926032    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n8l46_3359c7fa-a51a-4dc7-b46e-c99a59154ba9/exporter/0.log"
Jul 01 19:46:46.952538 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:46.952518    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n8l46_3359c7fa-a51a-4dc7-b46e-c99a59154ba9/extractor/0.log"
Jul 01 19:46:49.404212 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:49.404176    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-67d95fdc48-j4m6t_68177740-cc72-4d09-9c42-9d9146ba216d/manager/0.log"
Jul 01 19:46:49.523602 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:49.523574    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-t2bnp_f23ff531-b808-4adf-ae47-e3955c020b75/manager/0.log"
Jul 01 19:46:49.557913 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:49.557878    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-h8x54_afd6ed61-31e8-48a6-b2f3-74e5bf203c19/s3-init/0.log"
Jul 01 19:46:49.576670 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:49.576645    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-h8x54_afd6ed61-31e8-48a6-b2f3-74e5bf203c19/download-hf-model/0.log"
Jul 01 19:46:51.520078 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:51.520045    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-f2cj7/perf-node-gather-daemonset-zr6bf"
Jul 01 19:46:55.769495 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:55.769422    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hnpkl_1673e32e-d2ee-445b-95d6-144e138343df/kube-multus-additional-cni-plugins/0.log"
Jul 01 19:46:55.794999 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:55.794971    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hnpkl_1673e32e-d2ee-445b-95d6-144e138343df/egress-router-binary-copy/0.log"
Jul 01 19:46:55.822583 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:55.822548    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hnpkl_1673e32e-d2ee-445b-95d6-144e138343df/cni-plugins/0.log"
Jul 01 19:46:55.848054 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:55.848021    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hnpkl_1673e32e-d2ee-445b-95d6-144e138343df/bond-cni-plugin/0.log"
Jul 01 19:46:55.880829 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:55.880790    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hnpkl_1673e32e-d2ee-445b-95d6-144e138343df/routeoverride-cni/0.log"
Jul 01 19:46:55.911836 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:55.911812    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hnpkl_1673e32e-d2ee-445b-95d6-144e138343df/whereabouts-cni-bincopy/0.log"
Jul 01 19:46:55.938036 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:55.938008    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hnpkl_1673e32e-d2ee-445b-95d6-144e138343df/whereabouts-cni/0.log"
Jul 01 19:46:56.011109 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:56.011081    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-txnzp_bb05bf30-f19a-4b15-af14-30bf33f28fa9/kube-multus/0.log"
Jul 01 19:46:56.084526 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:56.084447    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-m6lm4_98d86579-8161-4185-8cf7-5f427d712b2e/network-metrics-daemon/0.log"
Jul 01 19:46:56.116295 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:56.116265    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-m6lm4_98d86579-8161-4185-8cf7-5f427d712b2e/kube-rbac-proxy/0.log"
Jul 01 19:46:56.934658 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:56.934628    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bplsm_bfb845e6-341d-492e-91c9-9c698a0b8d29/ovn-controller/0.log"
Jul 01 19:46:56.999890 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:56.999861    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bplsm_bfb845e6-341d-492e-91c9-9c698a0b8d29/ovn-acl-logging/1.log"
Jul 01 19:46:56.999890 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:56.999889    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bplsm_bfb845e6-341d-492e-91c9-9c698a0b8d29/ovn-acl-logging/0.log"
Jul 01 19:46:57.051867 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:57.051840    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bplsm_bfb845e6-341d-492e-91c9-9c698a0b8d29/kube-rbac-proxy-node/0.log"
Jul 01 19:46:57.083527 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:57.083502    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bplsm_bfb845e6-341d-492e-91c9-9c698a0b8d29/kube-rbac-proxy-ovn-metrics/0.log"
Jul 01 19:46:57.126274 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:57.126247    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bplsm_bfb845e6-341d-492e-91c9-9c698a0b8d29/northd/0.log"
Jul 01 19:46:57.170789 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:57.170766    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bplsm_bfb845e6-341d-492e-91c9-9c698a0b8d29/nbdb/0.log"
Jul 01 19:46:57.204870 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:57.204799    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bplsm_bfb845e6-341d-492e-91c9-9c698a0b8d29/sbdb/0.log"
Jul 01 19:46:57.257637 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:57.257604    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bplsm_bfb845e6-341d-492e-91c9-9c698a0b8d29/ovnkube-controller/0.log"
Jul 01 19:46:58.942706 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:58.942677    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-7g8pr_9e551421-5f11-40eb-8a22-e594b6f4eb61/network-check-target-container/0.log"
Jul 01 19:46:59.977214 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:46:59.977185    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-xtknv_190cde15-2eee-4b40-b4e1-c7cad1331778/iptables-alerter/0.log"
Jul 01 19:47:00.743177 ip-10-0-137-164 kubenswrapper[2574]: I0701 19:47:00.743150    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-nj5ft_e96a778c-46ff-4720-a550-499d2f39ff83/tuned/0.log"